The Python Environment is old. Python development started before the internet. Naturally, such a grown environment is messy:
In this course, you will learn the details about Python packaging and how all of the tools related.
- Library: Should be included in other code. It will never be executed directly. It does NOT have an entry point.
- CLI Application: Command line applications are supposed to be executed in their own environment. It receives input via the CLI, might access local storage or make web requests. It could read environment variables. It should be started, execute the tasks, and finish. It has an entry point.
- GUI Applications: Similar to command line applications, but with a GUI instead of a CLI.
- Service: Similar to a command line application, but it never finishes. It just runs in the background.
- Notebooks: Ad-hoc scripts which are primarily used for data analysis.
I like the ZSH shell with the plugin Oh My ZSH and Sublime Text as an editor with many different plugins; I've written down some of my Sublime Text plugins.
A common alternatives to ZSH are Fish. Common alternatives to Sublime Text are Atom and VS Code. If you want more, PyCharm.
Make sure you have git, pipenv, cookiecutter and pre-commit installed.
Starting a Project
Suppose you want to develop a new
awesome_project. Then you create a folder
and make it a git repository:
$ cookiecutter https://github.com/MartinThoma/cookiecutter-python-package full_name [Martin Thoma]: email [[email protected]]: github_username [MartinThoma]: project_name [Awesome Project]: project_slug [awesome_project]: project_short_description [Awesome Project lets you feel the pure awesomeness of awesome.]: version [0.1.0]: Select open_source_license: 1 - MIT license 2 - BSD license 3 - Not open source Choose from 1, 2, 3 (1, 2, 3) : $ cd awesome_project $ git init
To make sure that a failing hard drive does cause only little loss of work and to allow collaboration, we add a remote. Github and Gitlab are excellent choices. Once you created an empty repository there, add it as a remote locally:
$ git remote add origin [email protected]:MartinThoma/awesome_project.git
Before starting development, it's a good idea to make sure that you don't get
into trouble for having different versions of Python / packages.
pipenv is my tool of choice. At this point,
using Python 3.8 is a good idea (current support status of Python versions).
# Initialize the virtual environment $ pipenv --python 3.8 # Import the requirements $ pipenv install -r requirements.txt # Import the dev requirements $ pipenv install --dev --pre -r requirements-dev.txt # Make sure the repository stays that nice $ pre-commit install
By the mentioned cookiecutter template, formatting is to a big extend already handled:
- black: An opinionated formatter which respects PEP8 and implements a lot of Flake8
- isort: Sort imports
The only missing thing is a docstring style formatter. I like the numpydoc docstring format a lot.
If you want to know more about formatting, I recommend to read my Python style guide.
Before you start testing your application, you should decide which Python version you want to support. An orientation should be which CPython versions currently receive security updates (see SO Question).
Python has multiple testing frameworks. Use pytest. It's extremely widespread, stable and super simple to use.
The cookiecutter template installs a couple of useful plugins:
pytest-cov: Generate a test coverage report. This helps you to identify sections where bugs cannot possibly be catched by a unittest.
pytest-black: Check if black was applied.
pytest-flake8: Another formatting test.
pytest-mccabe: Check if a section of your code might be too complicated.
You might also want to give
mutmut a try.
It could help you to discover which lines have been checked, but maybe not
To run all of your tests, execute
If you develop a library, you need documentation. For the other application types not so much.
Sphinx as a documentation generator and readthedocs.org as a hosting platforms are the tools of choice.
If you develop an application, you need to be sure to update your dependencies
if vulnerabilities occur. The package
and the services snyk.io / pyup.io
can help you to detect those cases.
It is good practice to make
[module].__version__ available. Of course, it
should be the same as the version you see via
pip freeze. And then it would
be nice if the git commit whould have a git tag.
Of course, you can all of that manually. If you want a tool, bumpversion is pretty widespread. However, it is not maintained. So some people use bump2version. I'm not too sure if I would use that.
awesome-project # (git root) ├── awesome_project # This is the package │ ├── cli.py │ ├── __init__.py # Required until Python 3.3; I'd still add it │ └── _version.py ├── README.md ├── requirements-dev.txt ├── requirements.txt ├── setup.cfg ├── setup.py ├── tests │ └── test_awesome_project.py └── tox.ini
If you have the described project structure, then packaging is not a big deal.
Make sure it has at least
from setuptools import setup setup()
setup.cfg is read by
setuptools.setup(). It can contain a lot of things,
but a minimal one would look like this:
[metadata] name = awesome_project author = Martin Thoma author_email = [email protected] # keep in sync with awesome_project/_version.py version = 0.1.0 description = Awesome Project lets you feel the pure awesomeness of awesome. long_description = file: README.md long_description_content_type = text/markdown license = MIT license [options] packages = find: python_requires = >= 3.0
Creating a distribution
Create a source distribution file:
$ python setup.py sdist
Create a wheel distribution:
$ python setup.py bdist_wheel
Share the Package
After creating it, upload it to PyPI with twine.
To do so, first setup your
[distutils] index-servers = pypi pypitest [pypi] username: YourUsername password: plaintext whatever you had [pypitest] repository: https://test.pypi.org/legacy/ username: YourUsername password: plaintext whatever you had
Now you can upload the distributions you've built before:
$ twine upload --repository pypitest dist/* # Alternatively, if you want to sign it with GPG: $ twine upload --repository pypitest -s dist/*
python setup.py uploadas well. It didn't use https for quite a while. While this changed, the de-facto standard is still twine. For reasons.
Package Management Modules
Deprecated. Use setuptools.
Was a fork of setuptools which got merged back. Use setuptools.
Setuptools is used in the
setup.py and gives you a lot of commands:
$ python setup.py --help-commands Standard commands: build build everything needed to install build_py "build" pure Python modules (copy to build directory) build_ext build C/C++ extensions (compile/link to build directory) build_clib build C/C++ libraries used by Python extensions build_scripts "build" scripts (copy and fixup #! line) clean clean up temporary files from 'build' command install install everything from build directory install_lib install all Python modules (extensions and pure Python) install_headers install C/C++ header files install_scripts install scripts (Python or otherwise) install_data install data files sdist create a source distribution (tarball, zip file, etc.) register register the distribution with the Python package index bdist create a built (binary) distribution bdist_dumb create a "dumb" built distribution bdist_rpm create an RPM distribution bdist_wininst create an executable installer for MS Windows check perform some checks on the package upload upload binary package to PyPI Extra commands: bdist_wheel create a wheel distribution alias define a shortcut to invoke one or more commands bdist_egg create an "egg" distribution develop install package in 'development mode' dist_info create a .dist-info directory easy_install Find/get/install Python packages egg_info create a distribution's .egg-info directory install_egg_info Install an .egg-info directory for the package rotate delete older distributions, keeping N newest files saveopts save supplied options to setup.cfg or other config file setopt set an option in setup.cfg or another config file test run unit tests after in-place build (deprecated) upload_docs Upload documentation to PyPI flake8 Run Flake8 on modules registered in setup.py usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...] or: setup.py --help [cmd1 cmd2 ...] or: setup.py --help-commands or: setup.py cmd --help
Package Distribution Formats
There are 3 common formats:
- Source distributions
Egg is outdated and can be replaced by either source distributions or wheel (source)
PIP is short for 'PIP installs Python'. Use it. Don't use
PIP commands are
install Install packages. download Download packages. uninstall Uninstall packages. freeze Output installed packages in requirements format. list List installed packages. show Show information about installed packages. check Verify installed packages have compatible dependencies. config Manage local and global configuration. search Search PyPI for packages. wheel Build wheels from your requirements. hash Compute hashes of package archives. completion A helper command used for command completion. debug Show information useful for debugging. help Show help for commands.
- Alexander VanTol: Pipenv: A Guide to the New Python Packaging Tool,
- Tutorial: Packaging Python Projects
- Knewton: The Nine Circles of Python Dependency Hell, 2015
- Martin Thoma: How does Python / pip handle conflicting transitive dependencies?, 2020
- Building and Distributing Packages with Setuptools
- Tl;DR Legal: Compare licenses
Other Packaging stuff:
- How to create Windows executable (.exe) from Python script: You need to work on a Windows system for this.
- py2exe - Tutorial
- py2exe - generate single executable file
- compiling .py into windows AND mac executables on Ubuntu
- Windows .exe*cutable from Python developing in Ubuntu
- Cross-compiling a Python script on Linux into a Windows executable