Python + Conda Container

The Why, What & How of Conda

If you've spent much time with Python, you're probably familiar with pip. It's a package installer, and it'll download pretty much any Python package you can imagine, along with any Python dependencies. This is the key, though: pip only installs Python packages.

An example of where this might fall through is when you want to communicate with an external database - perhaps on another server, or even just in another container. The major Python clients for database communication (such as pymysql or psycopg) have some system dependencies. Usually on an Ubuntu-based systems, you can install these dependencies with apt, and be done with it. Cloud Containers are built in a way that the system is immutable, though, and as a result these dependencies can't be installed to the system. This is where Conda comes in.

Conda (in our case Miniconda) can create isolated environments, with a critical feature for Cloud Containers: Python packages with system dependencies can be installed along with those dependencies. This means, instead of needing to run pip install psycopg and also apt-get install pgsql-common to use Postgres from your Python app, you can activate your environment, run conda install psycopg and it'll get everything you need.

Another nice feature of Conda is that you can create environments with a range of Python versions. Inside the container, you'll be able to see the full list of options with conda search python, but officially, all stable releases that are at least in security or bugfix phases should be supported (some older, unsupported versions might also be available). This gives you great flexibility, allowing you to update your applications to later Python releases at your convenience.

Creating a Python + Conda Container

The Python + Conda Cloud Container is a service container. The creation process is much the same as other service containers - pick a name, add any environment variables you want, and expose (or publish) the ports you want open, if any.

Conda Setup

The first thing you'll need to do in this container is set up the Conda environment you want to use for your Python application. Run conda create -n env_name python=version, replacing env_name with whatever you'd like to call the environment, and version with a Python version (e.g. 3.11, 2.7, 3.7.3). You'll probably be prompted to confirm some setup options. Once the process is complete, you'll be notified and the environment will be ready to start using.

When you create an environment, you can specify multiple packages (and optionally their versions) that you want to include, not just python (in fact, python itself is completely optional but Conda will install it automatically if it's required). You can also install more packages into an existing environment once you activate it, using conda install instead.

You can also create an environment with pre-defined characteristics using an environment.yml file: read up on this over at the conda docs.

Using your Conda Environment

You can activate your Conda environment using conda activate env_name, where env_name matches the name you initially created the environment with. From there, you can check that Python is alive and well with python --version (this should display the version you've created the container with). While it is not generally recommended, you can optionally install dependencies using pip install as you normally would with a pure Python env - but keep in mind that Conda won't keep track of these, and any dependency that uses system packages to work (like our SQL examples above) will need to be installed via conda install instead.

Conda provides a number of scientific packages, and we have additionally enabled the conda-forge source to allow a (much) wider range of community-maintained packages. If you don't want conda-forge enabled, or you'd like to add additional sources, see the Conda Configuration section.

Freezing Requirements

Conda, similar to pip, has the capability to list and export currently installed dependencies using the command conda list --export > requirements.txt. This will only check Conda dependencies, though, so if you've installed any dependencies via pip or another source, you'll have to manage them separately.

Alternatively, you can export the entire environment's configuration, including pip dependencies, using conda env export > environment.yml. Using this, you'll be able to build another container with identical packages. You may want to manually update the file with version ranges (instead of fixed versions).

Conda Configuration

You can override the Conda environment defaults provided by using conda config within the active environment, or via a .condarc file. You can read about .condarc and configuration over on the Conda documentation..

To see what the given defaults are, you can use conda config --show-sources - this should show you the container defaults, plus any user configs being sourced. We have locked down a few configuration options in order to maintain the integrity of the container, and to ensure that your Conda environments continue to operate as expected across container restarts. However, anything else (including channels) can be overridden.

Directory Structure

The Python + Conda container follows our standard Cloud Container directory structure, with one addition: the /container/system/ directory, which is new to Cloud Containers. This location was added to support two features:

• Allowing SSH users to control Supervisor with supervisorctl.
• Provide a separate read/write directory where container-specific files (such as Conda environments) can live, without cluttering up the /container/application/ directory.

Adding a Supervisor Process

More likely than not, you're going to want Supervisor to take control of your process to ensure it's always running. A default template has been provided (commented out in /container/config/supervisor.conf), but there are a couple things to keep in mind:

• conda activate fully integrates an environment with your prompt. You don't need (or want) this for your supervisor process! Instead, use source activate, which will activate the environment and make binaries, libraries and other modules available to the process.
• You will need to ensure that your application is run by the www-data user, and not your SSH user. This is because Cloud Containers are built with directory and system permissions catered to www-data - running your application as any other user will likely result in permission issues and your application will not work as expected.

For more information about using Supervisor, check out Supervisor in Cloud Containers.

Cronjobs

Many applications need to run scheduled jobs - updating data, running mailers, etc. While you can incorporate these into Python apps using modules like APScheduler, most Linux systems (our Cloud Container images included) come with Cron as a scheduler which is usually preferred. System schedulers tend to be more forgiving on resources, and if you can run your scheduled jobs with the Cron scheduler instead of within the application you can avoid cluttering your application threads with additional work. You can read more about using Cloud Container crontabs in our KB.

Web Applications

One of the use cases we see for the Python + Conda image is running Python-based web applications, such as those running on popular frameworks like Flask, Django or FastAPI. If you're looking to run one of these, we also provide some information and a (basic) example in our KB.