One of the biggest hurdles for those just beginning to code is the set up and management of their coding environment. It can get confusing quickly when you start working on projects that have different requirements or dependencies, especially when manipulation of your project PATH is required. Today, I’ll be covering some of the tools I’ve found helpful in managing packages and environments when working with Python 2 and 3. Please note that I use an M1 macbook and will be making references to commands based on a bash shell on macOS.
Starting with package management, python has a built-in package management system called pip. You can use it to install packages from the Python Package Index or other indexes. pip comes pre-installed with Python version 3.4 and later. If you are working with an earlier version of python, you can use the following command line to run a built-in python module to install pip
$ python -m ensurepip –upgrade
Some useful pip commands include the following:
| $ pip install <package name> | Installs specified python package |
| $ pip install -r requirements.txt | Installs all python packages specified in requirements.txt |
| $ pip uninstall <package name> | Uninstalls specified python package |
| $ pip list | Lists all installed python packages in current environment |
| $ pip freeze | Outputs all installed packages in requirements format |
For more information on the uses of pip, you can use pip help or check out the documentation at https://pip.pypa.io/en/stable/.
Python also comes with a built-in lightweight virtual environment manager called venv. This module allows you to create your own isolated virtual environments. These virtual environments have their own Python binary and can have it’s own independent set of installed Python packages. Using venv is very simple..
Creating a new virtual environment:
$ python -m venv /path/to/new/virtual/environment
For example..
$ python -m venv env
The above will create a new environment and python binary under the folder ‘env’ within the local directory you used to call venv. This environment will copy the python binary you were previously using in your local directory. Once you’ve created a new virtual environment, you will then need to activate it.
Activating a virtual environment:
$ source env/bin/activate
Note for windows machines, the bin folder and activate binary may be named differently.
Once activated, Python will treat this environment and python binary as it’s primary PYTHONPATH. You can double check this by using the $ which python and $ which pip commands. It should show you the filepath to your env folder. From here, when you use pip to install packages it will be saved to your env folder, leaving your main python directory uncluttered and independent from the packages you may be using for a specific project. Once you’re done with your project or if you want to switch back to your normal python instance, you can deactivate your virtual environment.
Deactivating a virtual environment:
$ deactivate
It’s a fairly straight forward command. Once you’re completely done with a specific environment and want to remove it, all you need to do is to delete the env folder from your hard drive. Again a fairly straight forward process.
For those who are working with a lot of data science related libraries, and want a simplified way to managing their packages and environments together, Anaconda is a viable solution. Anaconda is a distribution of packages built for data science. It comes pre-installed with conda, a package and environment manager. Conda can be used to create new virtual environments that run specific versions of python, packages, and comes pre-installed with over 150 scientific packages including numpy, matplotlib, and pandas just to name a few. Conda also works with R, Ruby, Lua, and more, providing a similar experience across many popular data science languages. Conda also comes with built-in commands for sharing environments with team members making it a highly productive tool for collaboration.
Some useful conda commands include the following:
| $ conda create –-name <ENVNAME> python=3.6 | Creates a new virtual environment with the specified ENVNAME and python version |
| $ conda base | Returns to the ‘base’ python environment |
| $ conda activate ENVNAME | Activates virtual environment |
| $ conda deactivate | Deactivates current virtual environment |
| $ conda env list | Lists all available conda environments |
| $ conda list –-name ENVNAME | Lists all installed packages for a specified environment. ENVNAME can be omitted to print list of packages for current environment |
For those who don’t need the whole suite of data science packages available with full fledge Anaconda, miniconda provides just a basic installation of Python, pip, and conda. It allows you the flexibility to choose which packages you want to install while providing you with the same package and environment management as Anaconda with conda.
Miniforge is an additional community-led spinoff of Anaconda and miniconda, but provides additional support for ARM based processors, which is not present for Anaconda and miniconda. You can read more here
- https://docs.conda.io/projects/conda/en/latest/
- https://docs.conda.io/en/latest/miniconda.html
- https://github.com/conda-forge/miniforge
The ability to isolate your environment path and dependencies is great for managing numerous projects. All the tools above have provided command line utilities for activating virtual environments. When you’re using an IDE, there are also ways to specify which Python environment you would like to run your current code in. Looking specifically at VS code, you can specify this by selecting which python interpreter you want to use. To do this, open up ‘Command Pallette’ under the ‘view’ menu. From here, type in ‘Python: Select Interpreter’. This will give you a drop down of all available python binaries to use. Select the corresponding (and active) virtual environment python binary you would like to use and VScode will run your current project using that binary.
I hope this helps get you started on leveraging the power of virtual environments for your python projects!
