SSD Nodes Learn Hosting plans →
How to do am Matt ConnorBy Matt Connor

Python venv vs pipx vs uv for Ubuntu server

Ubuntu 24.04 fit show externally-managed-environment when pip install fail. See when to use venv, pipx or uv, plus how to point systemd to the right one.

Why pip install no work for fresh Ubuntu server

To choose between Python venv, pipx and uv for server, you need answer one question: wetin you wan install? Application dependencies belong inside virtual environment for the application's own directory. Command-line tools wey you want type by name belong inside pipx. uv fit do both jobs and e add lockfile too. This one become important as soon as another machine need build the same environment. None of dem install inside system Python, because current Ubuntu server dey reject that directly.

Run sudo pip install requests for Ubuntu 24.04 and pip go stop before e download even one file.

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this behaviour by passing --break-system-packages.

Na PEP 668 (Python enhancement proposal 668, "externally managed environments") dey do im work. Debian and Ubuntu dey put marker file beside the interpreter for /usr/lib/python3.12/EXTERNALLY-MANAGED, and pip no gree write inside any interpreter wey get the marker.

The rule dey exist because of sys.path order. apt dey install libraries into /usr/lib/python3/dist-packages. pip, when root run am against system interpreter, dey write into /usr/local/lib/python3.12/dist-packages, and Debian packaging dey put that directory earlier for the search path. Print am yourself with python3 -c 'import sys; print(sys.path)' and read the order. So, the copy wey pip write go hide the copy wey apt install, for every program on the machine wey dey run under /usr/bin/python3, including the distribution own tools. cloud-init dey import requests, jinja2 and PyYAML from that interpreter. If you upgrade one of dem with pip and e land on incompatible release, something wey you never touch fit fail for the next boot with traceback wey name package you no know say e dey inside the chain. apt still dey record im own version as installed, so nothing go warn you. To repair am, remove pip copy with sudo pip uninstall --break-system-packages requests, because sudo apt reinstall python3-requests by itself only dey rewrite the apt copy wey the pip copy don already hide.

The rule wey follow short. The distribution own system Python belong to the distribution. No install inside am, no use pip upgrade im libraries, and no delete the EXTERNALLY-MANAGED file just to make the message disappear. The only work wey you suppose give /usr/bin/python3 na to build virtual environments.

venv vs pipx vs uv: decision rule

Choose based on wetin you dey install, no be based on which tool you read about most recently.

  • Application wey you deploy and run as service, like Django or Flask project: use one virtual environment (venv) inside that application directory.
  • Command-line tool wey you want for your PATH, like ansible or httpie: use pipx. E give each tool private environment and one link for PATH.
  • Project wey need lockfile, faster installs, or Python version wey the distribution no ship: use uv. E produce normal venv plus uv.lock file.
  • Library wey distribution tool need instead of your code: use sudo apt install python3-<name>. Na the only supported way to add anything to system interpreter.

pipx and uv tool install dey do the same work, so if box already get uv, e no need pipx too. The web framework wey you choose no change anything here: Django and Flask for VPS differ for wetin go enter requirements.txt, no be for how dem build the environment around am. Everything below dey use Ubuntu 24.04 and e Python 3.12, so adjust the version inside the paths if your own one different.

Build the per-application venv

Ubuntu dey separate the venv module from the base Python package. So for a minimal image, the first attempt go fail with message wey name exactly wetin dey miss.

The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.12-venv

Install am, then create the environment as the user wey go own the code.

sudo apt update
sudo apt install -y python3-venv
sudo install -d -o deploy -g deploy -m 755 /srv/myapp
sudo -u deploy python3 -m venv /srv/myapp/.venv
sudo -u deploy /srv/myapp/.venv/bin/pip install -r /srv/myapp/requirements.txt

Notice wetin no dey there: no source, no activate. /srv/myapp/.venv/bin/pip installs inside that environment because of where the binary dey, not because of anything wey you export into the shell. Confirm am before you continue.

/srv/myapp/.venv/bin/python -c 'import sys; print(sys.prefix)'

That one go print /srv/myapp/.venv. If e print /usr, you dey run the system interpreter and your packages enter place wey you no intend.

Two properties of a venv decide wetin you fit do with am afterwards. A venv no be relocatable, because every entry-point script for bin/ get absolute shebang line: head -1 /srv/myapp/.venv/bin/pip reads #!/srv/myapp/.venv/bin/python. If you rename the parent directory, those scripts go fail with bad interpreter: No such file or directory. A venv also pins the interpreter wey create am. This one dey recorded as the home line for /srv/myapp/.venv/pyvenv.cfg, and bin/python3 na symlink to /usr/bin/python3. If you upgrade the release so that path now lead to newer Python, the new interpreter go look for packages inside a lib/python3.13 directory wey the venv no get. So the service go die when e start with ModuleNotFoundError. Both cases get the same fix: delete the venv and build new one from requirements.txt. Rebuilding dey take seconds. Never copy venv between machines.

Wey venv dey, and who own am

Put am beside the code for /srv/myapp/.venv, and keep one venv for each application. Deployment go then be one directory, systemd unit go get path wey no dey change, and two applications no fit break each other through one shared dependency upgrade. No put venv for anywhere wey your web server dey publish files directly, because e hold your dependencies and often your configuration.

Ownership deserve half a minute. Make a deploy user own the code and the environment, and give the service account only read and execute access.

sudo adduser --system --group --no-create-home myapp
sudo chown -R deploy:myapp /srv/myapp
sudo chmod -R o-rwx /srv/myapp

The service fit now import its dependencies but e no fit rewrite dem. This mean say code execution bug for the web application no fit quietly replace library for disk and survive restart. The same reason apply to the rest of the machine, and running services as least-privilege users cover am.

pipx for command-line tools

pipx dey install applications, no be libraries. Every tool get im own environment under ~/.local/share/pipx/venvs/<name>, and dem dey link that tool executables into ~/.local/bin. So, two tools wey need different versions of the same library no go clash.

sudo apt update
sudo apt install -y pipx
pipx ensurepath
pipx install httpie

pipx ensurepath dey add ~/.local/bin to PATH by editing your shell startup file. E no fit change the shell wey you dey use already. So, if http: command not found show immediately after installation, e usually mean say you never log out and log in again. Ubuntu default ~/.profile dey add ~/.local/bin only when that directory don already exist when you log in. Na why this problem dey happen once for new account, then e no dey happen again.

If you point pipx to a library, e go refuse and show message wey start with:

No apps associated with package requests.

Na the tool dey tell you say you use the wrong instrument. Libraries belong inside an application's venv.

The important detail for server na location. Plain pipx install dey put everything under one user's home directory. A systemd unit wey dey run as myapp no go see am. Root cron job no go see am too, and sudo no go find am either, because secure_path inside /etc/sudoers dey replace PATH with fixed list. If the whole machine suppose use the tool, install am globally.

sudo pipx install --global ansible
sudo pipx ensurepath --global

The --global flag dey put environments inside /opt/pipx and link executables into /usr/local/bin. That directory dey for the default PATH and inside secure_path. Check your version first with pipx --version, because Ubuntu 24.04 packages pipx 1.4.3, wey old pass --global. Older pipx go answer with unrecognized arguments: --global. For that version, set the two documented directories yourself:

sudo env PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install ansible
command -v ansible

command -v ansible suppose print /usr/local/bin/ansible. If e print path under /home, the tool enter one user's account, and no service go find am.

uv wen you need lockfile

uv na one binary from Astral wey cover wetin pip, venv, and pip-tools dey do. E fit download interpreters too. E fast reach say you go notice the difference for small VPS, and e dey write real lockfile.

The official installer dey put uv and uvx for ~/.local/bin:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version

Piping script enter shell for server need small caution. Pin the version for the URL, then read the file before you run am:

curl -LsSf https://astral.sh/uv/0.12.3/install.sh -o uv-install.sh
less uv-install.sh
sh uv-install.sh

pipx install uv dey work too, when pipx don already dey there. uv na one self-contained binary wey no get Python dependency of its own, so copying am into /usr/local/bin na valid way to share am with every user for the box.

For project wey get pyproject.toml, the workflow na four commands, and na only the last one dey run for server.

uv init myapp
uv add flask gunicorn
uv lock
uv sync --frozen --no-dev

uv lock dey write uv.lock, wey be cross-platform lockfile wey hold exact resolved versions, and you commit am beside your code. uv sync dey build .venv for project root to match am. For server, --frozen na the flag wey matter: documentation define am as using the versions for lockfile as the source of truth instead of checking whether the lockfile dey up to date. Na this behaviour deployment need. --no-dev dey leave out the development dependency group.

Existing requirements.txt project no need conversion, because uv dey speak pip language:

uv venv /srv/myapp/.venv
uv pip install --python /srv/myapp/.venv/bin/python -r /srv/myapp/requirements.txt

Wetin come out na ordinary virtual environment with one difference: uv no dey install pip inside am. So .venv/bin/pip and .venv/bin/python -m pip no go work unless you create the environment with uv venv --seed. Install with uv pip instead. Nothing later for this guide dey run pip inside the environment, so nothing later go change.

One uv default worth knowing before you use am for server. The python-preference setting default to managed. Documentation describe am as choosing "those that are downloaded and installed by uv" instead of interpreters wey already dey present for the system. So uv venv --python 3.13 for box wey only ship 3.12 go quietly fetch 3.13 into ~/.local/share/uv/python instead of failing. This convenient for laptop but e fit surprise you for server, because your service don depend on interpreter wey dey inside home directory, and apt upgrade no go ever patch am. Set python-preference to only-system inside uv.toml if you want use the distribution interpreter. If you want the environment for place wey no be project root, UV_PROJECT_ENVIRONMENT specify the directory to use for project virtual environment.

Point systemd to the venv interpreter, no be to activate

Na here most Python deployment dey break, and na misunderstanding of wetin activate dey do cause am.

bin/activate na shell script. E dey put the venv `bin directory for front of PATH, set VIRTUAL_ENV, save the old values so deactivate fit restore dem, and change your prompt. Nothing dey inside am wey the interpreter itself dey read. Activation na just convenience for human wey dey type python` for prompt.

Na the interpreter file wey you execute dey actually select the environment. When /srv/myapp/.venv/bin/python start, Python `site module dey look for pyvenv.cfg file for the directory wey hold the executable and one level above am. If e find /srv/myapp/.venv/pyvenv.cfg, e set sys.prefix to the venv, and this one put that venv site-packages for sys.path`. Na the whole mechanism be this. E no need environment variable or shell.

So this unit no go ever start:

[Service]
ExecStart=source /srv/myapp/.venv/bin/activate && gunicorn app:app
myapp.service: Failed to locate executable source: No such file or directory
myapp.service: Failed at step EXEC spawning source: No such file or directory
myapp.service: Main process exited, code=exited, status=203/EXEC

ExecStart no be shell command line. systemd dey execute program directly, so no source builtin dey, && go enter as literal argument, and none of shell expansions go happen apart from the few wey systemd document by itself.

And this unit go start, then die:

[Service]
ExecStart=/usr/bin/python3 /srv/myapp/app.py
ModuleNotFoundError: No module named 'flask'

/usr/bin/python3 na the system interpreter, and its sys.path never contain your venv. The same command dey work for your SSH session only because you don activate the venv there. So the shell resolve python3 through PATH to .venv/bin/python3 instead.

If you wrap the command for /bin/bash -c 'source ... && gunicorn ...', e go work. But e still put shell between systemd and your process without any benefit. One absolute path solve the matter:

[Unit]
Description=myapp web service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/srv/myapp
Environment=PYTHONUNBUFFERED=1
Environment=PATH=/srv/myapp/.venv/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=/srv/myapp/.venv/bin/gunicorn --workers 3 --bind 127.0.0.1:8000 app:app
Restart=on-failure
RestartSec=5
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=full

[Install]
WantedBy=multi-user.target
sudo systemctl daemon-reload
sudo systemctl enable --now myapp
systemctl status myapp
journalctl -u myapp -n 50 --no-pager

systemctl status myapp suppose report active (running) with Main PID wey be your gunicorn process. If e show anything else, read the journal.

The Environment=PATH= line no dey there for ExecStart, because ExecStart already carry full path. E dey there for the processes wey your application start. Service dey inherit short default PATH from systemd. So Python code wey call subprocess.run(["ffmpeg", ...]), or management command wey start console script from the venv, no go find wetin e need. To put the venv `bin directory first na the only part of activate wey service really use. Check wetin the unit actually receive with systemctl show -p Environment myapp`.

The same rule cover scheduled work. For Ubuntu, cron no dey pass job the PATH from your shell or from the daemon. Unless the crontab set PATH by itself, every job go get the compiled-in default /usr/bin:/bin, and /etc/environment no go override am. That default no dey ever contain your venv. So crontab line wey read python3 /srv/myapp/cleanup.py go run the system interpreter and fail with ModuleNotFoundError by 3 in the morning. cron dey mail the error to the crontab owner, or to the MAILTO address if you set one. For most servers, this mean local mailbox wey nobody dey read. If no mail program dey installed, cron go throw the output away and write only No MTA installed, discarding output to the system log. Write the absolute venv path there too. If you want make the output enter the journal and keep record of the last run instead, systemd service and timer pair go use the same ExecStart line.

Docker replace this decision?

Container get filesystem wey belong to am, so the question change shape instead of disappearing. For official image like python:3.12-slim, Python dey already inside /usr/local and e no carry any EXTERNALLY-MANAGED marker. So using pip install as root na the intended way to add packages, and venv no add much value. Build FROM ubuntu:24.04 instead, and you go meet externally-managed-environment again inside the image, for the same reason as for host: na the distribution interpreter dey carry the distribution marker file.

Plenty images still dey use venv because e make multi-stage build simple. Builder stage installs packages into /opt/venv, then runtime stage copies that one directory and leaves compilers behind. The activate problem still follow am. A RUN source /opt/venv/bin/activate line affects only the shell for that build layer, so when container run, e starts with system interpreter and raises ModuleNotFoundError. Set ENV PATH="/opt/venv/bin:$PATH", or give CMD the absolute path /opt/venv/bin/gunicorn. Na the same bug as the systemd one, but e dey inside another file.

So container replace the interpreter question because the image pins the interpreter and everything under am. E no replace the pinning question. Image wey you build from unpinned requirements.txt go resolve different versions next month. This mean say the image tag dey reproducible, but the build wey produce am no be reproducible. Lockfile like uv.lock, or requirements file wey pin every version fully, na wetin close that gap, whether container dey involved or not. And when one application dey run for one VPS under systemd, container mostly move this same decision into a Dockerfile, because systemd already dey restart failed process and capture the output for journal. Running Docker for VPS dey worth am when you want the built image itself to be the thing wey you deploy.

FAQ

I fit just use pip install with --break-system-packages?

No, no be wise for server wey you need keep running. The flag do exactly wetin e talk: e remove the protection, and pip write inside /usr/local/lib/python3.12/dist-packages, wey dey come before the apt directory for sys.path. Your version go then override the distribution own for every system script wey dey run under /usr/bin/python3. apt still believe say na its own version dey installed, so nothing go detect the conflict until something break. For container image wey you rebuild from scratch every time, the damage stop for that image, so e fit make sense there. But for machine wey you dey maintain, create a venv. Na one command.

Where venv suppose dey for server?

Put am inside the application's own directory, as /srv/myapp/.venv. A deploy user suppose own am, while the service account get read and execute access only. Keep one venv for each application, because shared venv mean say upgrade for the first application fit break the second one. No move or copy venv after you create am: every entry-point script for its bin/ directory get that absolute path written inside its shebang line. So moved venv go fail with bad interpreter: No such file or directory. Delete am and rebuild from requirements.txt instead.

Why my systemd service dey fail with ModuleNotFoundError?

Because the unit dey execute interpreter wey no be the venv own. Run systemctl cat myapp and read ExecStart. E must name /srv/myapp/.venv/bin/python, or console script from that same bin/ directory, with absolute path. Sourcing activate directly inside ExecStart no fit work, because ExecStart no be shell, and systemd report Failed to locate executable source with status=203/EXEC. Add Environment=PATH=/srv/myapp/.venv/bin:/usr/local/bin:/usr/bin:/bin so any subprocess wey your code start fit find the venv tools too.

I suppose use uv instead of venv and pip?

Use uv when you want lockfile, when installation time slow reach make e disturb you, or when you need Python version wey your distribution no release. E create ordinary venv, so the systemd unit and file layout no change, and uv sync --frozen install exactly wetin the lockfile record. If one application dey deploy from git with pinned requirements.txt and installation finish within seconds, python3 -m venv don already enough, and na one less binary to keep updated for server.

Dis article help you?

#python#venv#pipx#uv#deployment