python

📜 Python typing tips & tricks

I used to be in the 'static types only' camp and kind of hated Python. But mypy (and gradual typing) changed my mind about dynamic languages in general.
At some point it 'clicked' and I learnt to sense better when I'm comfortable with some dynamic madness, and when it makes sense to set it in stone by typing.

Here I'm collecting some guides to get started, cool not well known features I find useful, and some common annoying issues.
Mostly it's going to be about mypy, but not limited to.

Table of Contents

[A] * 'philosophy' of mypy pythonmypy

Aka when and how it's best to use or not to use it

[B] [2019-12-08] similar approach to mine: Statically-typed error handling in Python using Mypy | Hacker News pythonmypy

My approach is to dial up strictness gradually as code proves its value. I'll start out building a project and not validating on I/O, but as the requirements get locked down and the code has proven itself, I'll clean up all the edge cases - which will often mean adding in progressively stricter validation on border code.
The advantage of this is that if you end up not wasting too much time "building the wrong thing". Let's say that you took one form of I/O and built massively strict validation in and then realized later that you should have taken an entirely different form of I/O for your subsystem. All that time building in validation on that useless part of code was a pointless waste.
I don't have any stats, but my gut feel is that on average 40% of code can end up being tossed in this way (in some projects it's 100% =).
Prototyping speed is, additionally, not just useful in reducing the cost of building the right kind of code, it's useful in reducing the cost of building the right kind of test (a really underappreciated facet of building mission critical systems).
In my younger years I used to believe that for mission critical systems "building the wrong thing" was somehow less of a problem in code because you could fix requirements and do architecture upfront with some sort of genius architect. Turns out this was wrong.

[B] [2020-06-22] We've been running mypy on our project for about a year now and it's one of the best decisions we've made | Hacker News pythonmypy

We implemented it progressively. At first I added it as a make target but didn't make it mandatory in CI so I could learn how to use it.
Then I made it mandatory for a few files that I was the only active contributor to.
Then I slowly added more and more files across the project, sometimes as I touched them for other reason and other times as independent changes.
Eventually as mypy caught more and more bugs in other contributor's changes they started getting on board and adding type hints as well, until the vast majority of the project was hinted (we'll be getting to 100% within a few weeks).

This is super reasonable, especially for existing projects which are hard to type in one go.

[C] [2021-02-19] you can use --exclude to exclude (duh) individual files you haven't yet managed to type properly pythonmypy

[B] * typing tutorials/feature overviews pythontypes

[B] [2019-12-30] good mypy intro, demonstrates many important features: Python Type Hints – NP-Incompleteness pythontypesmypy

[B] [2020-02-05] Applying mypy to real world projects: very good intro on configuration & fine tuning pythontypesmypy

[B] [2021-03-08] mypy-assisted error handling pythontypesmypy

[C] [2020-12-08] Exhaustiveness Checking with Mypy | Haki Benita pythontypesmypy

[B] * various useful mypy tricks/features pythonmypy

[B] get_origin/get_args is nice for metaprogramming/automatic discovery pythonmypy

CREATED: [2019-12-06]
get_origin(Union[T, int]) is Union
get_origin(List[Tuple[T, T]][int]) == list
get_args(Dict[str, int]) == (str, int)
get_args(Union[int, Tuple[T, int]][str]) == (int, Tuple[str, int])
  • [2021-03-08] started using it in HPI for automatic output detection

[B] [2019-10-26] revealtype can be used during the type checking pythonmypy

reveal_type(a) # N: Revealed type is 'builtins.int'

https://github.com/python/mypy/blob/09c243dcc12935b989367f31d1d25d7fd0ec634c/test-data/unit/check-python38.test

https://github.com/python/mypy/blob/master/test-data/unit/README.md
apparently there is also reveal_locals()!

[B] flags I like pythonmypy

pretty = True
show_error_context = True
show_error_codes = True
check_untyped_defs = True
namespace_packages = True

they probably make sense everywhere, you might even want to add them to ~/.config/mypy/config

TODO [C] disallow_generic_any could be good… pythonmypy

CREATED: [2020-05-25]

STRT [C] [2020-10-05] from __future__ import annotations to be able to write something like list[int] pythonmypy

PEP 585 – Type Hinting Generics In Standard Collections | Python.org

Starting with Python 3.7, when from __future__ import annotations is used, function and variable annotations can parameterize standard collections directly.
haystack: dict[str, list[int]]

[C] [2019-12-08] mypy is aware of sys.version_info, it can help for writing backwards compatible code pythonmypy

[C] [2020-05-06] Protocols and structural subtyping pythonmypy

You can use a protocol class with isinstance() if you decorate it with the @runtime_checkable class decorator. The decorator adds support for basic runtime structural checks:

[C] [2019-10-12] Protocols and structural subtyping — Mypy pythonmypy

isinstance() also works with the predefined protocols in typing such as Iterable.

[D] [2020-08-19] Reading a list of files from a file pythonmypy

any command-line argument starting with @ reads additional command-line arguments from the file following the @ character.
This is primarily useful if you have a file containing a list of files that you want to be type-checked

[C] * mypy gotchas/bugs pythonmypy

[A] [2019-12-08] Statically-typed error handling in Python using Mypy | Hacker News pythonmypy

you need 'py.typed' file and also explicitly mentioning it in setup.py

Third party libraries: I sometimes see them annotated, but what people don't suspect, is that you need to include 'py.typed' file with your package in order for it to be discoverable.

TODO [B] [2021-03-08] I think it's a really annoying obstacle to typing adoption. Perhaps that's something setuptools could warn the developer about? pythonmypy

[B] [2019-03-12] looks like mypy doesn't like the lack of init file, and struggles to discover submodules if it's missing? pythonmypy

My current workaround is using __init__.pyi

[C] [2020-10-31] mypy + multiple python versions pythonmypy

this is mypy friendly

if sys.version_info[:2] >= (3, 8):

this isn't

if sys.version_info.minor >= 7:

TODO [D] hmm, aliases aren't working with new annotations? Rows = list[list[str]] pythonmypy

CREATED: [2021-02-02]
TypeError: 'type' object is not subscriptable

TODO [D] that's odd, behaviour when checking package vs files is different? pythonmypy

CREATED: [2019-06-02]

mypy wereyouhere/extractors/custom.py

wereyouhere/extractors/custom.py:56: error: Unexpected keyword argument "line" for "make" of "Loc"
wereyouhere/extractors/custom.py:144: error: Too few arguments for "make" of "Loc"

mypy wereyouhere

no errors?

TODO [D] shit. I don't get why there is a difference between checking source and directly??? pythonmypy

CREATED: [2020-09-12]
$ mypy my/endomondo.py
my/endomondo.py:8: error: Skipping analyzing '.core.common': found module but no type hints or library stubs  [import]
    from .core.common import Paths, get_files
    ^
my/endomondo.py:8: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
my/endomondo.py:39: error: Skipping analyzing '.core.error': found module but no type hints or library stubs  [import]
    from .core.error import Res
    ^
my/endomondo.py:54: error: Skipping analyzing '.core': found module but no type hints or library stubs  [import]
        from .core import stat
    ^
Found 3 errors in 1 file (checked 1 source file)
$ mypy -p my.endomondo
my/endomondo.py: note: In function "workouts":
my/endomondo.py:48: error: Incompatible types in "yield" (actual type "endoapi.endomondo.Workout", expected type "Union[my.endomondo.Workout, Exception]")  [misc]
                yield w
                ^
Found 1 error in 1 file (checked 1 source file)

TODO [D] hmm, apparently function parameter/argument names are taken into the account? pythonmypy

CREATED: [2019-12-07]
def testx(x: int) -> None:
    pass

def testy(y: int) -> None:
    pass

test = testx
test = testy

[D] [2018-06-20] all and pycharm and mypy pythonmypy

TLDR: messing with all results in problems, don't do that..
I guess I have to be careful on module levels then if I don't want unwanted crap…

WAIT [D] error with reusing exception variable: https://github.com/python/mypy/issues/5080 pythonmypy

CREATED: [2018-08-22]
try:
  pass
except Exception as e:
  pass

for e in []:
  pass

[C] * typing complicated code pythonmypy

TODO [B] [2019-11-02] Type hints for naive/aware datetime objects? : Python pythonmypydatetime

[2019-12-20] use NewType for that? pythondatetimemypy

ndt = NewType('ndt', datetime.datetime)
dt = NewType('dt', datetime.datetime)

def dt_to_ndt(dt: dt) -> ndt:
    pass

[B] [2019-10-27] how to define type of lru_cache (also works on similar decorators) pythonmypy

from typing import TYPE_CHECKING, TypeVar
if TYPE_CHECKING:
    F = TypeVar('F', Callable)
    def lru_cache(f: F) -> F: pass
else:
    from functools import lru_cache

[B] [2020-05-25] Making a decorator which preserves function signature · Issue 1927 · python/mypy pythonmypy

from typing import Any, Callable, TypeVar
FuncT = TypeVar('FuncT', bound=Callable[..., Any])

ok, this + overrides allowed me to type it properly

[B] [2020-05-03] Support function decorators excellently · Issue 3157 · python/mypy pythonmypy

Just for the record: if someone needs to change the return type of the function inside the decorator and still have typed parameters, you can use a custom mypy plugin that literally takes 15 LoC: https://github.com/dry-python/returns/blob/92eda5574a8e41f4f5af4dd29887337886301ee3/returns/contrib/mypy/decorator_plugin.py

Saved me a lot of time!

[C] [2019-12-07] hack for preventing unnecessary module imports pythonmypy

Type annotation will require importing modules that you wouldn't need to import without it. However there is now a workaround (quote the name and import in a dead if):

if False:
    from bar_module import bar

def foo(a: 'bar'):
     pass

[C] [2019-08-31] Define a JSON type · Issue 182 · python/typing pythonmypy

With that and the idea of string self referencing, JSONType can be defined as:

from typing import Recursive, Union, List, Dict

JSONType = Recursive(
    "JSONType",
    Union[int, float, str, bool, None, List["JSONType"], Dict[str, "JSONType"]]
)

[C] [2019-08-31] . pythonmypy

Unless I'm mistaken, recursive types with ForwardRefs can already be checked correctly at runtime by using ForwardRef._evaluate(globals(), locals())

[C] [2020-11-25] Decorated property not supported · Issue 1362 · python/mypy pythonmypy

As a workaround, you could try defining your own alias to the property decorator that is annotated to return Any. If you use your custom property decorator (that at runtime behaves exactly like property) for all decorated properties, mypy will silently give all such properties the Any type.

[C] [2018-12-11] use Protocol for classes which I can't control, they are very neat! pythonmypyhabit

[C] [2021-02-11] TypeVar to represent a Callable's arguments (for decorators/wrappers) pythonmypy

PEP 612 (https://www.python.org/dev/peps/pep-0612/) ended up providing a similar feature. I think we can close this issue now.

TODO [D] [2019-04-16] mypy overloads pythonmypy

@overload
def fetch_data(raw: Literal[True]) -> bytes: ...
@overload
def fetch_data(raw: Literal[False]) -> str: ...
# Fallback overload if the user provides a regular bool

[D] [2019-12-07] Still alive? · Issue 25 · machinalis/mypy-data https://github.com/machinalis/mypy-data/issues/25 pythonmypynumpy

[D] * other type checking tools pythontypes

Overall everything I've seen (pyre/pydantic) seems to target large codebases.
They might make sense for huge monorepos, but if you need to cooperate between many libraries (my usecase) they aren't useful. So I don't know much about them.

[C] [2020-03-12] pydantic pythontypes

If validation fails pydantic will raise an error with a breakdown of what was wrong:
from pydantic import ValidationError

try:
    User(signup_ts='broken', friends=[1, 2, 'not number'])
except ValidationError as e:
    print(e.json())

[C] [2020-03-22] MonkeyType: A system for Python that automatically generates type annotations | Hacker News https://news.ycombinator.com/item?id=22624845 pythontypes

Check out mypy with "--no-any-expression" flag and Pydantic.

[B] [2019-09-16] typeddjango/awesome-python-typing: Collection of awesome Python types, stubs, plugins, and tools to work with them. python

[C] [2019-09-27] Python Negatypes • Hillel Wayne pythontypes

[C] [2020-10-03] tried this https://github.com/predictive-analytics-lab/data-science-types seems to give many false positives… pythonpandasmypy

TODO [C] https://github.com/wemake-services/wemake-python-styleguide/blob/master/styles/mypy.toml pythonmypy

CREATED: [2019-11-09]

STRT [C] [2019-12-30] Show HN: FastAPI: build Python APIs with Go-like speed and automatic UI docs | Hacker News python

To use Python type hints, and get automatic data validation, serialization, and documentation for your API (including interactive UI docs for your API). All that, even for deeply nested JSON bodies. And by using type hints, you get autocomplete everywhere, type error checks, etc.
Your API gets documented with standards: OpenAPI and JSON Schema.
Or to be able to have WebSockets.
Or for its dependency injection system, that saves you a lot of code and plugins.
You can check the features here: https://fastapi.tiangolo.com/features/
And you can see alternatives and comparisons here: https://fastapi.tiangolo.com/alternatives/

[2020-03-01] ok, nice, so it uses mypy types, not custom types (like hug?) python

STRT [C] stricter mypy flags python

CREATED: [2019-07-25]
--strict                  Strict mode; enables the following flags:

--warn-unused-ignores
--warn-redundant-casts

--warn-unused-configs
--disallow-subclassing-any
--disallow-any-generics
--disallow-untyped-calls
--disallow-untyped-defs
--disallow-incomplete-defs
--check-untyped-defs
--disallow-untyped-decorators
--no-implicit-optional
--warn-return-any
  • [2021-03-08] frankly it never stuck to me to use any of these except –check-untyped-defs

TODO [C] hmm, seems like fastapi always has to be wrapped in pydantic. ugh! python

CREATED: [2020-03-12]

run with uvicorn fa:app –reload –port 9090

from typing import NamedTuple
from fastapi import FastAPI
app = FastAPI()
from dataclasses import dataclass
@dataclass
class X:
    a: int
    x: str

@app.get("/", response_model=X)
async def root() -> X:
    return X(a=123, x="fwef")

CNCL [C] PathIsh thing can be generalized to Path constructible? pythonmypy

CREATED: [2019-05-02]

[2019-06-28] not sure if init would do well? https://mypy.readthedocs.io/en/latest/protocols.html pythonmypy

[2019-08-24] eh, so os.PathLike[str] kinda solves it, but using .pyi file for that is not really worth it pythonmypy

TODO [D] seems like a bug in fbmessenger export (on a branch) pythonmypy

CREATED: [2020-01-13]

------------------------------------------------- python

[B] [2021-04-05] crazy how useful it is to have gradual typing when you're gradually hardening the program, adding error hadnling etc pythontypes

allows to still test happy scenarious even if type checking fails

Jump to search, settings & sitemap