Python Dependency Management (pip/pipenv/poetry)

Steven Kessler -

Pip

Python dependency management has long been a bit of a pain, while some languages package managers have seen rapid development in very short amounts of time such as npm/gradle/cargo, Pip has very much trundled along being just good enough but always a bit frustrating.

I personally had used pip for many years without having too many issue's until I began work on a project which required maintaining a fairly out of date Django app with a very large number of packages and working on getting it up to date.

Here is where the madness begins!

First of all as everyone knows pip is not deterministic but in addition to that it lacks the ability to come up with anything more than a barebones and simple dependency graph.

It cannot retry various versions of libraries to come up with a working set, this can be easily reproduced when trying to upgrade from boto2 -> boto3 for the aws libs while serving both, often you will find your awscli or boto versions being installed while expressly not being able to work together!

pip freeze

This is a command you will become very accustomed too when trying to puzzle why a feature suddenly stopped working on a co-workers machine despite working fine on your local dev machine. The reason for this is simple there is no guarantee you pull the same version of the sub dependencies if they are not explicitly pinned inside the requirements.txt.

A common exercise is that once you have a working virtual environment is to basically take the results of the pip freeze as a pseudo lock file like so

pip freeze > requirements.txt

These issue's in my experience tend not to affect smaller projects as much because there is less wiggle room but in an older and less well constructed application where the req.txt has grown to a truly ponderous size of 100+ lines it quickly becomes a huge headache to upgrade or change any of the dependencies.

The example below is a particular issue I have run into in the past and that is attempting to manage packages that rely on a particular version of sub dependency. Very few people are going to be using botocore and it will be easy to miss pinning it entirely and end up with whatever package is installed first winning out on declaring the sub dependency. This can cause you to end up with either too new a sub dependency or too old and broken functionality with minimal feedback.

awscli
boto3==1.4.0
boto==2.42.0
botocore==1.8.33

produces a pip freeze of:

awscli==1.16.117
boto==2.42.0
boto3==1.4.0
botocore==1.8.33
colorama==0.3.9
docutils==0.14
jmespath==0.9.4
pyasn1==0.4.5
python-dateutil==2.8.0
PyYAML==3.13
rsa==3.4.2
s3transfer==0.2.0
six==1.12.0

with 2 warnings/errors

awscli 1.16.117 has requirement botocore==1.12.107, but you'll have botocore 1.8.33 which is incompatible.
boto3 1.4.0 has requirement botocore<1.5.0,>=1.4.1, but you'll have botocore 1.8.33 which is incompatible.
boto3 1.4.0 has requirement s3transfer<0.2.0,>=0.1.0, but you'll have s3transfer 0.2.0 which is incompatible.

On the surface everything seems fine, seems to install correctly the libraries you specified are all present but under the hood you have incompatible versions sitting next to each other, this right here to me is kinda the killer with PIP unless you are extremely knowledgeable about python and dependencies you just introduced a regression that will silently pass all CI checks and deploy out to prod and depending on the level of mocking you do your tests will probably all pass.

This is very obviously not ideal and I have seen it get a fair few jr. dev's into trouble or be absolutely terrified of installing anything, even their virtual env's for fear of introducing a regression.

So what are the alternatives if PIP is not up for the task of protecting and handling reproducible and stable installs? There are two competing tools that to me stand out ahead of the pack those are Poetry https://github.com/sdispater/poetry and Pipenv https://github.com/pypa/pipenv

Lets start with the above case what happens when we try to install

Pipenv

Pipfile

[[source]]
name = "pypi"
url = "https://pypi.org/simple"
verify_ssl = true

[dev-packages]

[packages]
awscli = "*"
boto3 = "==1.4.0"
boto = "==2.42.0"
botocore = "==1.8.33"

[requires]
python_version = "3.7"

pipenv install

will yield

Pipfile.lock (d04302) out of date, updating to (6d0532)…
Locking [dev-packages] dependencies…
Locking [packages] dependencies…
✘ Locking Failed!
[pipenv.exceptions.ResolutionFailure]:       req_dir=requirements_dir
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 726, in resolve_deps
[pipenv.exceptions.ResolutionFailure]:       req_dir=req_dir,
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 480, in actually_resolve_deps
[pipenv.exceptions.ResolutionFailure]:       resolved_tree = resolver.resolve()
[pipenv.exceptions.ResolutionFailure]:   File "/usr/local/Cellar/pipenv/2018.11.26_2/libexec/lib/python3.7/site-packages/pipenv/utils.py", line 395, in resolve
[pipenv.exceptions.ResolutionFailure]:       raise ResolutionFailure(message=str(e))
[pipenv.exceptions.ResolutionFailure]:       pipenv.exceptions.ResolutionFailure: ERROR: ERROR: Could not find a version that matches botocore<1.5.0,==1.12.107,==1.8.33,>=1.4.1

Excellent! the regression was prevented! Our installation failed as it should noting that you are trying to install packages with conflicting sub dependencies.

one advantage of pipenv is also the ability to immediately port over the dependencies of a requirements.txt so to run the previous requirements.txt you can also do pipenv install -r requirements.txt, not really setting the wold on fire but its an appreciated time saver.

The problem with pipenv is kinda two fold but one of those is not really a big deal anymore

  1. It was pushed by pypa before it was ready, and I mean that as a huge fan of pipenv it was not ready for wide use at all. The resolver was slow and unable to find working combinations within a set of dependencies if they existed as a range and it picked something outside the range before getting to that file. This was actually a huge and frustrating issue that drove alot of people away from the project.

It did however get fixed and alot of blood and sweat has been poured into the project the team has also listened to the feedback of its users and adjusted. The current release candidate is one I would consider ready for production and one im mostly happy with.

2. Pipenv does nothing for the existing ecosystem outside of providing a lockfile for consumers of packages. A better way to say this is pipenv is useless as a library maintainer it exists solely for application developers to consume packages.

This is not a huge deal breaker to me but it has to be said, its stated goal is not to cater to library creators and I think its a great example of a tool that sets out to do 1 or 2 things and do them well. However as a tool Pipenv now exists kinda outside the python ecosystem as a helpful addon rather than proper solution.

Overall pipenv works very well for the task it sets out to do and it is the tool that I am currently using in my day to day. The abstraction of virtualenv's and installation are great features but they are shared amongst the two tool's so I wont be talking too much about them but I am a very big fan of streamlining the python workflow.

Poetry

pyproject.toml

[tool.poetry]
name = "python_blog"
version = "0.1.0"
description = ""
authors = ["Steven Kessler <stvnksslr@gmail.com>"]
license = "MIT"

[tool.poetry.dependencies]
python = "3.7.1"
awscli = "^1.16"
boto3 = "1.4.0"
boto = "2.42.0"
botocore = "1.8.33"

[tool.poetry.dev-dependencies]

a simple poetry install and we get our error message!

Updating dependencies
Resolving dependencies... (0.3s)

[SolverProblemError]
Because python-blog depends on boto3 (1.4.0) which depends on botocore (>=1.4.1,<1.5.0), botocore is required.
So, because python-blog depends on botocore (1.8.33), version solving failed.

install [--no-dev] [--dry-run] [-E|--extras EXTRAS] [--develop DEVELOP]

Poetry spits out a similar error message to that of pipenv, less verbose but that aspect is one of personal preference. But the core feature locking the dependencies and guarding against accidental changes that would introduce a regression is there and it's a rather fantastic addition just as in pipenv.

The killer feature of poetry right now is that it implements a flavor of the PEP approved pyproject.toml of which theres a great write up on the PEP itself https://snarky.ca/clarifying-pep-518/ by Brett Cannon but basically this is the first big step on creating a unified way to specify packages and package dependencies for those consuming packages and those writing packages and here is where poetry and pipenv diverge in intent pretty heavily.

Your personal preferences will dictate whether this divergence is a plus or a minus but poetry is very much trying to be the next step in python packaging and attempting to get ahead of the curve a bit and implement some much needed features. They have a hell of a job ahead of them in my opinion as pleasing both those crowds is not going to be easy for the young project.