If you've spent much time with Python, you're probably familiar with
pip. It's a package installer, and it'll download pretty much any Python package you can imagine, along with any Python dependencies. This is the key, though:
pip only installs Python packages.
An example of where this might fall through is when you want to communicate with an external database - perhaps on another server, or even just in another container. The major Python clients for database communication (such as
psycopg) have some system dependencies. Usually on an Ubuntu-based systems, you can install these dependencies with
apt, and be done with it. Cloud Containers are built in a way that the system is immutable, though, and as a result these dependencies can't be installed to the system. This is where Conda comes in.
Conda (in our case Miniconda) can create isolated environments, with a critical feature for Cloud Containers: Python packages with system dependencies can be installed along with those dependencies. This means, instead of needing to run
pip install psycopg and also
apt-get install pgsql-common to use Postgres from your Python app, you can activate your environment, run
conda install psycopg and it'll get everything you need.
Another nice feature of Conda is that you can create environments with a range of Python versions. Inside the container, you'll be able to see the full list of options with
conda search python, but officially, all stable releases that are at least in security or bugfix phases should be supported (some older, unsupported versions might also be available). This gives you great flexibility, allowing you to update your applications to later Python releases at your convenience.
The Python + Conda Cloud Container is a service container. The creation process is much the same as other service containers - pick a name, add any environment variables you want, and expose (or publish) the ports you want open, if any.
The first thing you'll need to do in this container is set up the Conda environment you want to use for your Python application. Run
conda create -n env_name python=version, replacing
env_name with whatever you'd like to call the environment, and
version with a Python version (e.g.
3.7.3). You'll probably be prompted to confirm some setup options. Once the process is complete, you'll be notified and the environment will be ready to start using.
When you create an environment, you can specify multiple packages (and optionally their versions) that you want to include, not just
python (in fact,
python itself is completely optional but Conda will install it automatically if it's required). You can also install more packages into an existing environment once you activate it, using
conda install instead.
You can also create an environment with pre-defined characteristics using an
environment.yml file: read up on this over at the conda docs.
You can activate your Conda environment using
conda activate env_name, where
env_name matches the name you initially created the environment with. From there, you can check that Python is alive and well with
python --version (this should display the version you've created the container with). While it is not generally recommended, you can optionally install dependencies using
pip install as you normally would with a pure Python env - but keep in mind that Conda won't keep track of these, and any dependency that uses system packages to work (like our SQL examples above) will need to be installed via
conda install instead.
Conda provides a number of scientific packages, and we have additionally enabled the
conda-forgesource to allow a (much) wider range of community-maintained packages. If you don't want
conda-forgeenabled, or you'd like to add additional sources, see the Conda Configuration section.
Conda, similar to
pip, has the capability to list and export currently installed dependencies using the command
conda list --export > requirements.txt. This will only check Conda dependencies, though, so if you've installed any dependencies via
pip or another source, you'll have to manage them separately.
Alternatively, you can export the entire environment's configuration, including
pip dependencies, using
conda env export > environment.yml. Using this, you'll be able to build another container with identical packages. You may want to manually update the file with version ranges (instead of fixed versions).
You can override the Conda environment defaults provided by using
conda config within the active environment, or via a
.condarc file. You can read about
.condarc and configuration over on the Conda documentation..
To see what the given defaults are, you can use
conda config --show-sources - this should show you the container defaults, plus any user configs being sourced. We have locked down a few configuration options in order to maintain the integrity of the container, and to ensure that your Conda environments continue to operate as expected across container restarts. However, anything else (including channels) can be overridden.
The Python + Conda container follows our standard Cloud Container directory structure, with one addition: the
/container/system/ directory, which is new to Cloud Containers. This location was added to support two features:
More likely than not, you're going to want Supervisor to take control of your process to ensure it's always running. A default template has been provided (commented out in
/container/config/supervisor.conf), but there are a couple things to keep in mind:
conda activate fully integrates an environment with your prompt. You don't need (or want) this for your supervisor process! Instead, use
source activate, which will activate the environment and make binaries, libraries and other modules available to the process.
www-data user, and not your SSH user. This is because Cloud Containers are built with directory and system permissions catered to
www-data - running your application as any other user will likely result in permission issues and your application will not work as expected.
For more information about using Supervisor, check out Supervisor in Cloud Containers.
Many applications need to run scheduled jobs - updating data, running mailers, etc. While you can incorporate these into Python apps using modules like
APScheduler, most Linux systems (our Cloud Container images included) come with Cron as a scheduler which is usually preferred. System schedulers tend to be more forgiving on resources, and if you can run your scheduled jobs with the Cron scheduler instead of within the application you can avoid cluttering your application threads with additional work. You can read more about using Cloud Container crontabs in our KB.
One of the use cases we see for the Python + Conda image is running Python-based web applications, such as those running on popular frameworks like Flask, Django or FastAPI. If you're looking to run one of these, we also provide some information and a (basic) example in our KB.