Python venv vs pipx vs uv for Ubuntu server

Why pip install dey fail for fresh Ubuntu server

To choose between Python venv, pipx and uv for server, ask one question: wetin you wan install? Application dependencies belong inside virtual environment for the application's own directory. Command-line tools wey you wan run by name belong for pipx. uv fit do both jobs and e add lockfile. This one become important once another machine need build the same environment. None of dem install into system Python, because current Ubuntu server go reject that directly.

Run sudo pip install requests for Ubuntu 24.04 and pip go stop before e download even one file.

error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
    python3-xyz, where xyz is the package you are trying to
    install.

    If you wish to install a non-Debian-packaged Python package,
    create a virtual environment using python3 -m venv path/to/venv.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.

    If you wish to install a non-Debian packaged Python application,
    it may be easiest to use pipx install xyz, which will manage a
    virtual environment for you.

note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this behaviour by passing --break-system-packages.

Na PEP 668 (Python enhancement proposal 668, "externally managed environments") dey do the correct work here. Debian and Ubuntu put marker file beside the interpreter for /usr/lib/python3.12/EXTERNALLY-MANAGED, and pip refuse to write into any interpreter wey get that file.

The rule dey exist because of sys.path order. apt install libraries into /usr/lib/python3/dist-packages. pip, when root run am against system interpreter, write into /usr/local/lib/python3.12/dist-packages, and Debian packaging put that directory earlier for the search path. Print am yourself with python3 -c 'import sys; print(sys.path)' and read the order. So the copy wey pip write go cover the copy wey apt install, for every program for the machine wey run under /usr/bin/python3, including the distribution own tools. cloud-init import requests, jinja2 and PyYAML from that interpreter. If you upgrade any of dem with pip and land on incompatible release, something wey you never touch go fail for the next boot with traceback wey name package you no know say dey inside the chain. apt still record its own version as installed, so nothing go warn you, and the repair na sudo apt reinstall python3-requests.

The rule wey follow short. The distribution get system Python. No install inside am, no upgrade its libraries with pip, and no delete EXTERNALLY-MANAGED file to make the message disappear. The only job to give /usr/bin/python3 na to build virtual environments.

venv vs pipx vs uv: decision rule

Choose based on wetin you dey install, no be based on which tool you read about most recently.

• Application wey you deploy and run as service, like Django or Flask project: use one virtual environment (venv) inside that application directory.
• Command-line tool wey you want for your PATH, like ansible or httpie: use pipx. E give each tool private environment and one link for PATH.
• Project wey need lockfile, faster installs, or Python version wey the distribution no ship: use uv. E produce normal venv plus uv.lock file.
• Library wey distribution tool need instead of your code: use sudo apt install python3-<name>. Na the only supported way to add anything to system interpreter.

pipx and uv tool install dey do the same work, so if box already get uv, e no need pipx too. The web framework wey you choose no change anything here: Django and Flask for VPS differ for wetin go enter requirements.txt, no be for how dem build the environment around am. Everything below dey use Ubuntu 24.04 and e Python 3.12, so adjust the version inside the paths if your own one different.

Build per-application venv

Ubuntu dey separate venv module from the base Python package. So for minimal image, the first attempt go fail with message wey name exactly wetin dey miss.

The virtual environment was not created successfully because ensurepip is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

    apt install python3.12-venv

Install am, then create the environment as the user wey go own the code.

sudo apt update
sudo apt install -y python3-venv
sudo install -d -o deploy -g deploy -m 755 /srv/myapp
sudo -u deploy python3 -m venv /srv/myapp/.venv
sudo -u deploy /srv/myapp/.venv/bin/pip install -r /srv/myapp/requirements.txt

Notice wetin no dey there: no source, no activate. /srv/myapp/.venv/bin/pip dey install inside that environment because of where the binary dey, no be because of anything wey you export into the shell. Confirm am before you continue.

/srv/myapp/.venv/bin/python -c 'import sys; print(sys.prefix)'

That one go print /srv/myapp/.venv. If e print /usr, you dey run the system interpreter and your packages go enter place wey you no intend.

Two properties of a venv determine wetin you fit do with am afterwards. A venv no be relocatable, because every script for bin/ get absolute shebang line: head -1 /srv/myapp/.venv/bin/pip reads #!/srv/myapp/.venv/bin/python. If you rename the parent directory, those scripts go fail with bad interpreter: No such file or directory. A venv also pin the interpreter wey create am. The interpreter dey recorded as the home line for /srv/myapp/.venv/pyvenv.cfg, and bin/python3 na symlink to that binary. Upgrade the release until python3.12 no dey again, the symlink no get target, and the service go die when e start with No such file or directory. Both cases get the same fix: delete the venv and build new one from requirements.txt. Rebuilding dey take seconds. Never copy venv between machines.

Wey venv dey, and who own am

Put am beside the code for /srv/myapp/.venv, and keep one venv for each application. Deployment go then be one directory, systemd unit go get path wey no dey change, and two applications no fit break each other through one shared dependency upgrade. No put venv for anywhere wey your web server dey publish files directly, because e hold your dependencies and often your configuration.

Ownership deserve half a minute. Make a deploy user own the code and the environment, and give the service account only read and execute access.

sudo adduser --system --group --no-create-home myapp
sudo chown -R deploy:myapp /srv/myapp
sudo chmod -R o-rwx /srv/myapp

The service fit now import its dependencies but e no fit rewrite dem. This mean say code execution bug for the web application no fit quietly replace library for disk and survive restart. The same reason apply to the rest of the machine, and running services as least-privilege users cover am.

pipx for command-line tools

pipx dey install applications, no be libraries. Each tool get im own environment under ~/.local/share/pipx/venvs/<name>, and dem link that tool executables go ~/.local/bin. So, two tools wey need different versions of the same library no go clash.

sudo apt update
sudo apt install -y pipx
pipx ensurepath
pipx install httpie

pipx ensurepath dey add ~/.local/bin to PATH by editing your shell startup file. E no fit change the shell wey you dey use already. So if http: command not found show immediately after installation, e usually mean say you never log out and log in again. Ubuntu default ~/.profile dey add ~/.local/bin only when that directory already dey exist when you log in. Na why this problem dey happen once for new account and no dey happen again.

If you point pipx to a library, e go refuse am with a message wey start like this:

No apps associated with package requests or its dependencies.

Na the tool dey tell you say you use the wrong instrument. Libraries belong inside an application's venv.

For server, the important detail na where pipx install the tool. Plain pipx install dey put everything under one user's home directory. A systemd unit wey dey run as myapp no fit see am. A root cron job no fit see am too, and sudo no go find am either, because secure_path for /etc/sudoers dey replace PATH with a fixed list. If the whole machine need the tool, install am globally.

sudo pipx install --global ansible
sudo pipx ensurepath --global

The --global flag dey put environments inside /opt/pipx and link executables into /usr/local/bin. That path dey inside the default PATH and also inside secure_path. First check your version with pipx --version. Ubuntu 24.04 packages pipx 1.4.3, wey old pass --global, and older pipx go answer with unrecognized arguments: --global. For that version, set the two documented directories yourself:

sudo env PIPX_HOME=/opt/pipx PIPX_BIN_DIR=/usr/local/bin pipx install ansible
command -v ansible

command -v ansible suppose print /usr/local/bin/ansible. If e print path under /home, the tool enter one user's account, and no service go find am.

uv wen you need lockfile

uv na one binary from Astral wey cover wetin pip, venv and pip-tools dey do, and e fit download interpreters too. E fast reach wey you go notice the difference even for small VPS, and e dey write proper lockfile.

The official installer dey put uv and uvx inside ~/.local/bin:

curl -LsSf https://astral.sh/uv/install.sh | sh
uv --version

Piping script enter shell for server need make you take small care. Pin the version for the URL and read the file before you run am:

curl -LsSf https://astral.sh/uv/0.12.3/install.sh -o uv-install.sh
less uv-install.sh
sh uv-install.sh

pipx install uv dey work too, when pipx don already dey there. uv na one self-contained binary wey no get im own Python dependency, so copying am enter /usr/local/bin na valid way to share am with every user for the box.

For project wey get pyproject.toml, the workflow na four commands, and na only the last one dey run for server.

uv init myapp
uv add flask gunicorn
uv lock
uv sync --frozen --no-dev

uv lock dey write uv.lock, wey be cross-platform lockfile wey hold exact resolved versions, and you commit am beside your code. uv sync dey build .venv for the project root to match am. For server, --frozen na the flag wey matter: the documentation define am as using the versions for the lockfile as the source of truth instead of checking whether the lockfile dey up to date. Na this behaviour deployment need. --no-dev dey leave out the development dependency group.

Existing requirements.txt project no need conversion, because uv dey speak pip language:

uv venv /srv/myapp/.venv
uv pip install --python /srv/myapp/.venv/bin/python -r /srv/myapp/requirements.txt

Wetin come out na ordinary virtual environment. .venv/bin/python dey behave exactly as e go behave if python3 -m venv build am, so nothing later for this guide go change.

One uv default good make you know before you use am for server. Im python-preference setting default to managed, and documentation talk say e dey choose "those that are downloaded and installed by uv" instead of interpreters wey already dey for system. So uv venv --python 3.13 for box wey only ship 3.12 go quietly fetch 3.13 enter ~/.local/share/uv/python instead of failing. This convenient for laptop but e fit surprise you for server, because your service don depend on interpreter wey dey inside home directory and apt upgrade no go ever patch am. Set python-preference to only-system inside uv.toml if you want use the interpreter from the distribution. If you want the environment for place wey no be project root, UV_PROJECT_ENVIRONMENT specify the directory to use for project virtual environment.

Point systemd to the venv interpreter, no be to activate

Na here most Python deployment dey break, and the reason na say people no understand wetin activate dey do.

bin/activate na shell script. E dey put the venv bin directory for front of PATH, set VIRTUAL_ENV, save the old values so deactivate fit restore dem, and change your prompt. E no get anything wey the interpreter itself dey read. Activation na convenience for person wey dey type python for prompt.

Na the interpreter file wey you execute dey select the environment. When /srv/myapp/.venv/bin/python start, Python's site module dey look for pyvenv.cfg file for the directory wey hold the executable and one level above am. If e find /srv/myapp/.venv/pyvenv.cfg, e set sys.prefix to the venv. This one put the venv site-packages for sys.path. Na the complete mechanism be that. E no need environment variable or shell.

So this unit no go ever start:

[Service]
ExecStart=source /srv/myapp/.venv/bin/activate && gunicorn app:app

myapp.service: Failed to locate executable source: No such file or directory
myapp.service: Failed at step EXEC spawning source: No such file or directory
myapp.service: Main process exited, code=exited, status=203/EXEC

ExecStart no be shell command line. systemd dey execute program directly, so no source builtin dey available, && go enter as literal argument, and nothing go expand.

And this unit go start, then die:

[Service]
ExecStart=/usr/bin/python3 /srv/myapp/app.py

ModuleNotFoundError: No module named 'flask'

/usr/bin/python3 na the system interpreter, and your venv never dey inside its sys.path. The same command dey work for your SSH session only because you don activate the venv there. Na that make the shell resolve python3 through PATH to .venv/bin/python3 instead.

If you wrap the command inside /bin/bash -c 'source ... && gunicorn ...', e go work. But e still put shell between systemd and your process without any benefit. One absolute path don settle the matter:

[Unit]
Description=myapp web service
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/srv/myapp
Environment=PYTHONUNBUFFERED=1
Environment=PATH=/srv/myapp/.venv/bin:/usr/local/bin:/usr/bin:/bin
ExecStart=/srv/myapp/.venv/bin/gunicorn --workers 3 --bind 127.0.0.1:8000 app:app
Restart=on-failure
RestartSec=5
NoNewPrivileges=yes
PrivateTmp=yes
ProtectSystem=full

[Install]
WantedBy=multi-user.target

sudo systemctl daemon-reload
sudo systemctl enable --now myapp
systemctl status myapp
journalctl -u myapp -n 50 --no-pager

systemctl status myapp suppose report active (running) with Main PID wey be your gunicorn process. If e show anything else, read the journal.

The Environment=PATH= line no dey there for ExecStart, because ExecStart already carry complete path. E dey there for processes wey your application start. Service dey inherit short default PATH from systemd. So Python code wey call subprocess.run(["ffmpeg", ...]), or management command wey run console script from the venv, no go find wetin e need. Put the venv bin directory first. Na the only part of activate wey service truly dey use. Check wetin the unit really receive with systemctl show -p Environment myapp.

The same rule apply to scheduled work. cron dey run jobs with PATH of /usr/bin:/bin. So crontab line wey read python3 /srv/myapp/cleanup.py go run the system interpreter and fail with ModuleNotFoundError by three in the morning. The error go enter local mail spool wey nobody dey read. Write the absolute venv path there too. If you want that output for the journal and record of the last run instead, a systemd service and timer pair go use the same ExecStart line.

Docker replace this decision?

Container get filesystem wey belong to am, so the question change shape instead of disappearing. For official image like python:3.12-slim, Python dey already inside /usr/local and e no carry any EXTERNALLY-MANAGED marker. So using pip install as root na the intended way to add packages, and venv no add much value. Build FROM ubuntu:24.04 instead, and you go meet externally-managed-environment again inside the image, for the same reason as for host: na the distribution interpreter dey carry the distribution marker file.

Plenty images still dey use venv because e make multi-stage build simple. Builder stage installs packages into /opt/venv, then runtime stage copies that one directory and leaves compilers behind. The activate problem still follow am. A RUN source /opt/venv/bin/activate line affects only the shell for that build layer, so when container run, e starts with system interpreter and raises ModuleNotFoundError. Set ENV PATH="/opt/venv/bin:$PATH", or give CMD the absolute path /opt/venv/bin/gunicorn. Na the same bug as the systemd one, but e dey inside another file.

So container replace the interpreter question because the image pins the interpreter and everything under am. E no replace the pinning question. Image wey you build from unpinned requirements.txt go resolve different versions next month. This mean say the image tag dey reproducible, but the build wey produce am no be reproducible. Lockfile like uv.lock, or requirements file wey pin every version fully, na wetin close that gap, whether container dey involved or not. And when one application dey run for one VPS under systemd, container mostly move this same decision into a Dockerfile, because systemd already dey restart failed process and capture the output for journal. Running Docker for VPS dey worth am when you want the built image itself to be the thing wey you deploy.

FAQ

I fit just use pip install with --break-system-packages?

No, no be wise for server wey you must keep running. The flag dey do exactly wetin e talk: e remove the guard, and pip write inside /usr/local/lib/python3.12/dist-packages, wey dey before apt directory for sys.path. Your version go then shadow the distribution version for every system script wey dey run under /usr/bin/python3. apt still believe say na im own version dey installed, so nothing go detect the conflict until something break. Inside container image wey you rebuild from scratch every time, the damage stop for that image, so e fit make sense there. For machine wey you dey maintain, create a venv. Na one command.

Where virtual environment suppose dey for server?

Put am inside the application's own directory, as /srv/myapp/.venv. A deploy user suppose own am, while the service account get only read and execute access. Keep one venv for each application, because shared venv mean say upgrade for the first application fit break the second one. No move or copy venv after you create am: every script inside im bin/ directory get that absolute path written for im shebang line, so moved venv go fail with bad interpreter: No such file or directory. Delete am and rebuild from requirements.txt instead.

Why my systemd service dey fail with ModuleNotFoundError?

Because the unit dey execute interpreter wey no be the venv interpreter. Run systemctl cat myapp and read ExecStart. E must name /srv/myapp/.venv/bin/python, or console script from that same bin/ directory, with absolute path. Sourcing activate inside unit file no fit work, because ExecStart no be shell, and systemd report Failed to locate executable source with status=203/EXEC. Add Environment=PATH=/srv/myapp/.venv/bin:/usr/local/bin:/usr/bin:/bin so any subprocess wey your code start fit find the venv tools too.

I suppose use uv instead of venv and pip?

Use uv when you want lockfile, when install time slow reach the point wey e dey disturb you, or when you need Python version wey your distribution no release. E create ordinary venv, so systemd unit and file layout no change, and uv sync --frozen install exactly wetin the lockfile record. If one application dey deploy from git with pinned requirements.txt and the install finish within seconds, python3 -m venv don already enough, and na one less binary to keep updated for server.