Python Packaging Course

The Python Environment is old. Python development started before the internet. Naturally, such a grown environment is messy:

The Python environmental protection agency wants to seal it in a cement chamber, with pictorial messages to future civilizations warning them about the danger of using sudo to install random Python packages.
Python Environment

In this course, you will learn the details about Python packaging and how all of the tools related.

Application Types

  • Library: Should be included in other code. It will never be executed directly. It does NOT have an entry point.
  • CLI Application: Command line applications are supposed to be executed in their own environment. It receives input via the CLI, might access local storage or make web requests. It could read environment variables. It should be started, execute the tasks, and finish. It has an entry point.
  • GUI Applications: Similar to command line applications, but with a GUI instead of a CLI.
  • Service: Similar to a command line application, but it never finishes. It just runs in the background.
  • Notebooks: Ad-hoc scripts which are primarily used for data analysis.

Development Environment

I like the ZSH shell with the plugin Oh My ZSH and Sublime Text as an editor with many different plugins; I've written down some of my Sublime Text plugins.

A common alternatives to ZSH are Fish. Common alternatives to Sublime Text are Atom and VS Code. If you want more, PyCharm.

Make sure you have git, pipenv, cookiecutter and pre-commit installed.

Starting a Project

Suppose you want to develop a new awesome_project. Then you create a folder and make it a git repository:

$ cookiecutter https://github.com/MartinThoma/cookiecutter-python-package
full_name [Martin Thoma]:
email [[email protected]]:
github_username [MartinThoma]:
project_name [Awesome Project]:
project_slug [awesome_project]:
project_short_description [Awesome Project lets you feel the pure awesomeness of awesome.]:
version [0.1.0]:
Select open_source_license:
1 - MIT license
2 - BSD license
3 - Not open source
Choose from 1, 2, 3 (1, 2, 3) [1]:

$ cd awesome_project
$ git init

To make sure that a failing hard drive does cause only little loss of work and to allow collaboration, we add a remote. Github and Gitlab are excellent choices. Once you created an empty repository there, add it as a remote locally:

$ git remote add origin [email protected]:MartinThoma/awesome_project.git

Before starting development, it's a good idea to make sure that you don't get into trouble for having different versions of Python / packages. pipenv is my tool of choice. At this point, using Python 3.8 is a good idea (current support status of Python versions).

# Initialize the virtual environment
$ pipenv --python 3.8

# Import the requirements
$ pipenv install -r requirements.txt

# Import the dev requirements
$ pipenv install --dev --pre -r requirements-dev.txt

# Make sure the repository stays that nice
$ pre-commit install

Formatting

By the mentioned cookiecutter template, formatting is to a big extend already handled:

  • black: An opinionated formatter which respects PEP8 and implements a lot of Flake8
  • isort: Sort imports

The only missing thing is a docstring style formatter. I like the numpydoc docstring format a lot.

If you want to know more about formatting, I recommend to read my Python style guide.

Unit Testing

Before you start testing your application, you should decide which Python version you want to support. An orientation should be which CPython versions currently receive security updates (see SO Question).

Python has multiple testing frameworks. Use pytest. It's extremely widespread, stable and super simple to use.

The cookiecutter template installs a couple of useful plugins:

  • pytest-cov: Generate a test coverage report. This helps you to identify sections where bugs cannot possibly be catched by a unittest.
  • pytest-black: Check if black was applied.
  • pytest-flake8: Another formatting test.
  • pytest-mccabe: Check if a section of your code might be too complicated.

You might also want to give mutmut a try. It could help you to discover which lines have been checked, but maybe not thoroughly enough.

To run all of your tests, execute tox.

Documentation

If you develop a library, you need documentation. For the other application types not so much.

Sphinx as a documentation generator and readthedocs.org as a hosting platforms are the tools of choice.

Security

If you develop an application, you need to be sure to update your dependencies if vulnerabilities occur. The package bandit and the services snyk.io / pyup.io can help you to detect those cases.

Versions

It is good practice to make [module].__version__ available. Of course, it should be the same as the version you see via pip freeze. And then it would be nice if the git commit whould have a git tag.

Of course, you can all of that manually. If you want a tool, bumpversion is pretty widespread. However, it is not maintained. So some people use bump2version. I'm not too sure if I would use that.

Project Structure

awesome-project  # (git root)
├── awesome_project  # This is the package
│   ├── cli.py
│   ├── __init__.py  # Required until Python 3.3; I'd still add it
│   └── _version.py
├── README.md
├── requirements-dev.txt
├── requirements.txt
├── setup.cfg
├── setup.py
├── tests
│   └── test_awesome_project.py
└── tox.ini

Setuptools Files

If you have the described project structure, then packaging is not a big deal.

setup.py

Make sure it has at least

from setuptools import setup

setup()

setup.cfg

The setup.cfg is read by setuptools.setup(). It can contain a lot of things, but a minimal one would look like this:

[metadata]
name = awesome_project

author = Martin Thoma
author_email = [email protected]

# keep in sync with awesome_project/_version.py
version = 0.1.0

description = Awesome Project lets you feel the pure awesomeness of awesome.
long_description = file: README.md
long_description_content_type = text/markdown

license = MIT license

[options]
packages = find:
python_requires = >= 3.0

Creating a distribution

Create a source distribution file:

$ python setup.py sdist

Create a wheel distribution:

$ python setup.py bdist_wheel

Share the Package

After creating it, upload it to PyPI with twine.

To do so, first setup your ~/.pypirc file:

[distutils]
index-servers =
  pypi
  pypitest

[pypi]
username: YourUsername
password: plaintext whatever you had

[pypitest]
repository: https://test.pypi.org/legacy/
username: YourUsername
password: plaintext whatever you had

Now you can upload the distributions you've built before:

$ twine upload --repository pypitest dist/*

# Alternatively, if you want to sign it with GPG:
$ twine upload --repository pypitest -s dist/*
There is python setup.py upload as well. It didn't use https for quite a while. While this changed, the de-facto standard is still twine. For reasons.

Package Management Modules

distutils

Deprecated. Use setuptools.

distribute

Was a fork of setuptools which got merged back. Use setuptools.

setuptools

Setuptools is used in the setup.py and gives you a lot of commands:

$ python setup.py --help-commands
Standard commands:
  build             build everything needed to install
  build_py          "build" pure Python modules (copy to build directory)
  build_ext         build C/C++ extensions (compile/link to build directory)
  build_clib        build C/C++ libraries used by Python extensions
  build_scripts     "build" scripts (copy and fixup #! line)
  clean             clean up temporary files from 'build' command
  install           install everything from build directory
  install_lib       install all Python modules (extensions and pure Python)
  install_headers   install C/C++ header files
  install_scripts   install scripts (Python or otherwise)
  install_data      install data files
  sdist             create a source distribution (tarball, zip file, etc.)
  register          register the distribution with the Python package index
  bdist             create a built (binary) distribution
  bdist_dumb        create a "dumb" built distribution
  bdist_rpm         create an RPM distribution
  bdist_wininst     create an executable installer for MS Windows
  check             perform some checks on the package
  upload            upload binary package to PyPI

Extra commands:
  bdist_wheel       create a wheel distribution
  alias             define a shortcut to invoke one or more commands
  bdist_egg         create an "egg" distribution
  develop           install package in 'development mode'
  dist_info         create a .dist-info directory
  easy_install      Find/get/install Python packages
  egg_info          create a distribution's .egg-info directory
  install_egg_info  Install an .egg-info directory for the package
  rotate            delete older distributions, keeping N newest files
  saveopts          save supplied options to setup.cfg or other config file
  setopt            set an option in setup.cfg or another config file
  test              run unit tests after in-place build (deprecated)
  upload_docs       Upload documentation to PyPI
  flake8            Run Flake8 on modules registered in setup.py

usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
   or: setup.py --help [cmd1 cmd2 ...]
   or: setup.py --help-commands
   or: setup.py cmd --help

Package Distribution Formats

There are 3 common formats:

  • Source distributions
  • Egg
  • Wheel

Egg is outdated and can be replaced by either source distributions or wheel (source)

Package manager

PIP is short for 'PIP installs Python'. Use it. Don't use easy_install.

PIP commands are

  install       Install packages.
  download      Download packages.
  uninstall     Uninstall packages.
  freeze        Output installed packages in requirements format.
  list          List installed packages.
  show          Show information about installed packages.
  check         Verify installed packages have compatible dependencies.
  config        Manage local and global configuration.
  search        Search PyPI for packages.
  wheel         Build wheels from your requirements.
  hash          Compute hashes of package archives.
  completion    A helper command used for command completion.
  debug         Show information useful for debugging.
  help          Show help for commands.

See also

Other Packaging stuff:

Published


by

Category

Code

Tags

Contact