Type Annotations in Python 3.8

One reason why Python is so easy to get started with is that it has dynamic types. You don’t have to specify the type of a variable, you just use variables as labels for containers of data. But in bigger projects, having types is helpful. If you have an undocumented function without types and maybe crappy variable naming, new developers will have a hard time. Luckily, variable annotations were added in Python 3.6 with PEP 526 🎉

This article is written in such a way that you can easily stop after the “mypy” section and then only look at individual sections.

Hello, Type-Annotated World!

def fib(n: int = 0) -> int:
    a, b = 0, 1
    for _ in range(n):
        a, b = b, a + b
    return a


# Print the first 10 Fibonacci numbers
for i in range(10):
    print(f"fib({i}) = {fib(i)}")

So you can simply use the pattern:

def some_function(param_name: typename) -> return_type_name:
    ...  # whatever the function does

Having type annotations is nice, but you need to check them! The Python runtimes do not do that, no matter if you use CPython, PyPy, or something more exotic.

mypy logo

Type Checking with mypy

Install mypy via pip install mypy and run it:

$ mypy . --ignore-missing-imports
Success: no issues found in 1 source file

The --ignore-missing-imports flag is necessary because otherwise you will get a lot of messages like this:

error: Skipping analyzing 'setuptools': found module but no type hints or library stubs

In order to make it more convenient, I usually add a setup.cfg file in which I specify that I always want this flag to be applied:

[mypy]
ignore_missing_imports=true

Then you can pip install pytest-mypy and make sure mypy is always executed when you run pytest by adding this section to your setup.cfg:

[tool:pytest]
addopts = --mypy

It is important to note that the Python community and also mypy assume that you come from a non-type-annotated code base. They want to make it easy for you to switch to annotated code and thus support gradual typing. However, this means that you might miss errors if you don’t annotate your code! Mypy has a lot of flags to help you to make the move. You don’t need to annotate everything.

typing: List, Dict, Tuple, Any

The typing module adds support for type hints. It contains some of the types you will use most often: List, Dict, and Tuple.

from typing import List


def fib_list(n: int = 0) -> List[int]:
    fib_numbers: List[int] = [0, 1]
    for _ in range(n):
        fib_numbers.append(fib_numbers[-1] + fib_numbers[-2])
    return fib_numbers


print(f"fib_list(10) = {fib_list(10)}")

Similarly, you can annotate that a dictionary maps strings to integers by Dict[str, int]. So List, Dict, and Tuple are generics. Any is just a way to specify that you could have arbitrary data in those containers. It is reasonable to use Any in the beginning when you start to add type annotations to a bigger code base.

Stop Type Checking

As mentioned before, mypy and Python support gradual typing. And sometimes you need to silence the type checker to be able to continue (and hopefully fix it later 🤞). There are a couple of ways to do this with typing:

typing.Any: Every type is compatible with Any.
typing.cast(SomeClass, variable): Sometimes mypy is not smart enough, so you can tell it which type you have. I did that a couple of times before I knew about typing.overload. Alternatively, you can also add assert isinstance(variable, SomeClass).
# type: ignore: Explicitly tell the type checker to ignore that line.

typing: Union and Optional

Pretty often, you want to accept multiple types. Then you use Union:

from typing import Union


def upcase(s: Union[str, bytes]) -> Union[str, bytes]:
    if isinstance(s, str):
        return s.upper()
    elif isinstance(s, bytes):
        return bytes(x - 0x20 if 0x61 <= x <= 0x7A else x for x in s)
    else:
        raise TypeError("need str or bytes")

As it happens pretty often that you need to accept some type and None, there is also typing.Optional. Optional[SomeType] is the same as Union[SomeType, None].

typing: List vs Sequence

The type typing.List actually represents list. A typing.Sequence is “an iterable with random access” as Jochen Ritzel put it so nicely. For example, a string is a Sequence[Any], but not a List[Any].

typing: Dict vs Mapping

Similarly to the example List vs Sequence, typing.Dict is meant mainly to represent a dict, whereas typing.Mapping is more general. Stacksonstacks gives a good answer.

Custom Types: Not all Strings are Created Equal

Not all strings contain the same type of content. They can represent a user_id, a user_name, a password_hash, …

Especially for IDs, I have seen this become messy. I think it’s pretty ridiculous to create a separate class for those different string types as creating a class is usually development and maintenance overhead. So, what do you do?

Don’t worry, typing.NewType has got you covered!

from typing import NewType

UserId = NewType("UserId", str)

typing.overload

typing.Union is actually an anti-pattern sometimes, because you can also overload a function as Josh Reed shows:

from typing import overload


@overload
def upcase(s: str) -> str:
    ...


@overload
def upcase(s: bytes) -> bytes:
    ...


def upcase(s):
    if isinstance(s, str):
        return s.upper()
    elif isinstance(s, bytes):
        return bytes(x - 0x20 if 0x61 <= x <= 0x7A else x for x in s)
    else:
        raise TypeError("need str or bytes")

Type checking only imports

I recently found myself in the position that I made a pretty heavy import on module level, just because of type checking. This felt wrong, so I asked for help. The solution was simple: typing.TYPE_CHECKING. This is True when running a type checker, but False during normal runs ❤️

Protocols

PEP 544 introduced structural subtyping in Python 3.8. It feels like Interfaces in Java and works like this:

from typing import Protocol


class SupportsClose(Protocol):
    def close(self) -> None:
        ...


def finish_it(obj: SupportsClose):
    obj.close()


class Foo:
    def close(self):
        pass


foo = Foo()
finish_it(foo)

Note that there is no function body. After that definition, you can then use SupportsClose like any type.

The cool part is that the class Foo has no explicit relationship to SupportsClose! It is only related by its structure!

Type comments

Type hints which are given as comments like this are outdated since Python 3.6:

from typing import List, Any, Sequence


def fib_list(n=0):
    # type: (int) -> Sequence[str]
    fib_numbers: List[int] = [0, 1]
    for _ in range(n):
        fib_numbers.append(fib_numbers[-1] + fib_numbers[-2])
    return "adf"


print(f"fib_list(10) = {fib_list(10)}")

However, you might want to disable type checking for single lines:

# type: ignore

Stub files

Stub files end in .pyi. If mypy finds a .py file and a .pyi file, it only loads the .pyi file. They are like header files in C++, but for Python. Instead of a function body, you use an Ellipsis ...:

def fib_list(n: int) -> List[int]:
    ...

pyright, pyre, pytype

pyright is a Python static type checker written by Microsoft, pyre is one by Facebook, and pytype is one by Google. All of them claim to be faster than mypy, all of them have lower adoption than mypy. I haven’t used them so far.

pyre logo

Install them:

$ pip install pyre-check pytype

# Yes, pyright is written in TypeScript...
$ npm install -g pyright

Run them:

$ pyright .
$ pytype .

pyright was complaining a lot about stuff that is actually correct.

pydantic

Variable annotations can also be used to remove a lot of boilerplate code. For example, pydantic can help you with serialization / deserialization:

# Core Library modules
import json
from typing import List

# Third party modules
import pydantic.json
from pydantic import BaseModel, parse_obj_as


class User(BaseModel):
    name: str
    age: int


# Deserialize a JSON string
users_str = '[{"name": "user1", "age": 15}, {"name": "user2", "age": 28}]'
users = parse_obj_as(List[User], json.loads(users_str))

# Proof it!
print(users)

# Serialize
print(json.dumps([user.dict() for user in users]))

Which gives:

[User(name='user1', age=15), User(name='user2', age=28)]
[{"name": "user1", "age": 15}, {"name": "user2", "age": 28}]

FastAPI uses pydantic directly.

A cool thing about pydantic is the constrained types: PositiveFloat, NegativeInt, constr, …