Simplifying Python 3 setup using venv

Dependency hell

Without virtual environments, working with Python is a nightmare! In case of Ubuntu, the base operating system comes with Python preinstalled, and you might think that that\'s a nice thing. Except, that version of Python is old and pip is not even installed. So you install pip only to find out that it is detecting and trying to update Python programs that are part of the base operating system and are supposed to be managed by your operating system's package manager.

This same pattern of unavailability and conflicts gets worse over time as you start working on multiple Python projects. Dependencies of one might conflict with the other. You may require a specific version of request package for one project and a different version for another, and on and on. This frustration is well-depicted in this XKCD Comic:

This post tries to address all these problems, and then some more. Obviously, most of this is aimmed at Linux users, and is tested on a Ubuntu 20.04 LTS server, but the principles can be generalized for macOS and Windows 10 too.

Goals

The goal is to achieve the following with regards to our Python 3 environment:

• Isolating it from the broader Operating system.
• Making it easy to configure and backup
• Making it easily upgradable
• Using methods recommended by the Official Documentation

We will be using venv module which is the recommended way to create virtual environments

Python venv to the rescue

In Python 3.6 the problem of managing multiple dependencies has received the cleanest resolution -- Virtual Environments or venv. In my opinion, this has rendered a lot of third party options like conda, pyenv, etc less relevant, if not completely redundant.

venv is an official Python module that is meant for running an isolated Python environment. This isolated environment can include Python interpreter, pip, and setuptools. All these dependencies are packaged into a user-specified directory.

Creating a virtual environment or venv

To get started, let\'s install venv on our system first:

$ sudo apt install python3-venv

Let's create a virtual environment next. I will name this virtual environment, my-venv and it will be created in the user\'s home directory for this specific case.

$ python3 -m venv $HOME/my-venv

For Ubuntu 20.04, we had to use python3 in the command instead of just python because python word is reserved for Python version 2. Same is true for macOS, but if you\'ve installed Python on Windows then you can just use python to invoke Python 3 interpreter.

Activating and using a virtual environment or venv

To start using the virtual environment, we need to import a script located at $HOME/my-venv/bin into our SHELL environment. This script will be add the virtual environment\'s directory to your shell\'s PATH environment such that when python or pip or other packages are searched for the SHELL will search the virtual environment\'s directory first search the my-venv directory to look for appropriate python program.

Depending on your SHELL the script you need to invoke will be different. For bash, and zsh interpreters the script is named activate:

root@node-00:~# source ~/my-venv/bin/activate
(my-venv) root@node-00:~#

For fish shell the script is called activate.fish and there is a similar script for tsch in the same directory. For Windows 10, the scripts will have a .bat and .ps1 extensions for command prompt and powershell respectively. Use the command echo $SHELL to know what shell your system is using.

Notice in the above shell, once the venv is activated the prompt changed. It also has a copy of pip install by default. You can verify this and even update it:

$ pip --version
pip 20.0.2 from /root/my-venv/lib/python3.8/site-packages/pip (python 3.8)

To list all the packages installed, and to update them just do the following:

$ pip list
Package       Version
------------- -------
pip           20.0.2
pkg-resources 0.0.0
setuptools    44.0.0

$ pip install --upgrade pip pkg-resources setuptools
$ pip list
Package       Version
------------- -------
pip           21.0.1
pkg-resources 0.0.0
setuptools    56.0.0

This won\'t update your system\'s copy of pip or any other pip package you have installed system-wide. The version of Python being used is the same as the Python command used to create your virtual environment.

You can install any number of pip packages inside this environment and use it for your Python projects. As long as the environment is active, the Python interpreter will prioritize these packages when your scripts call for a specific module, regardless of where you are in your filesystem. Once you are done working on the project, and want to switch gears simply type in deactivate and the virtual environment will be disabled:

(my-venv) root@node-00:~# deactivate
root@node-00:~#

Notice that the prompt is back to being normal. You can always reactivate the venv by running the source command along with the activate script. If your environment is broken simply remove the directory to get rid of it. Make sure that your Python project's file don't live in the same directory as the virtual environment.

Note on Updating Packages and Portability

Once you have installed necessary packages, it is always important to keep a snapshot of all the packages and their version numbers. This allows you to maintain and upgrade things gradually without the risk of things breaking down.

$ pip freeze > requirements.txt

By convention the requirements.txt file is used to store all the packages. Here's a typical output:

click==7.1.2
cycler==0.10.0
joblib==1.0.1
kiwisolver==1.3.1
matplotlib==3.4.1
nltk==3.6.1
numpy==1.20.2
Pillow==8.2.0
pyparsing==2.4.7
python-dateutil==2.8.1
regex==2021.4.4
six==1.15.0
textblob==0.15.3
tqdm==4.60.0

It also allows you to:

• Recreate the environment quickly and exactly, even if the underlying packages update in the future. Reinstall the same packages using the following command:

  $ pip install -r requirements.txt

• Keep track of the dependencies with your version control like git or GitHub.
• Reuse the file for other parts of your CI/CD like when creating a Docker image for your file.

That\'s it for Python environments! Happy computing!