📜 Python typing tips & tricks

My approach is to dial up strictness gradually as code proves its value. I'll start out building a project and not validating on I/O, but as the requirements get locked down and the code has proven itself, I'll clean up all the edge cases - which will often mean adding in progressively stricter validation on border code.
The advantage of this is that if you end up not wasting too much time "building the wrong thing". Let's say that you took one form of I/O and built massively strict validation in and then realized later that you should have taken an entirely different form of I/O for your subsystem. All that time building in validation on that useless part of code was a pointless waste.
I don't have any stats, but my gut feel is that on average 40% of code can end up being tossed in this way (in some projects it's 100% =).
Prototyping speed is, additionally, not just useful in reducing the cost of building the right kind of code, it's useful in reducing the cost of building the right kind of test (a really underappreciated facet of building mission critical systems).
In my younger years I used to believe that for mission critical systems "building the wrong thing" was somehow less of a problem in code because you could fix requirements and do architecture upfront with some sort of genius architect. Turns out this was wrong.

We implemented it progressively. At first I added it as a make target but didn't make it mandatory in CI so I could learn how to use it.
Then I made it mandatory for a few files that I was the only active contributor to.
Then I slowly added more and more files across the project, sometimes as I touched them for other reason and other times as independent changes.
Eventually as mypy caught more and more bugs in other contributor's changes they started getting on board and adding type hints as well, until the vast majority of the project was hinted (we'll be getting to 100% within a few weeks).

This is super reasonable, especially for existing projects which are hard to type in one go.

get_origin(Union[T, int]) is Union
get_origin(List[Tuple[T, T]][int]) == list
get_args(Dict[str, int]) == (str, int)
get_args(Union[int, Tuple[T, int]][str]) == (int, Tuple[str, int])

• [2021-03-08] started using it in HPI for automatic output detection
  

reveal_type(a) # N: Revealed type is 'builtins.int'

https://github.com/python/mypy/blob/09c243dcc12935b989367f31d1d25d7fd0ec634c/test-data/unit/check-python38.test

https://github.com/python/mypy/blob/master/test-data/unit/README.md
apparently there is also reveal_locals()!

pretty = True
show_error_context = True
show_error_codes = True
check_untyped_defs = True
namespace_packages = True

they probably make sense everywhere, you might even want to add them to ~/.config/mypy/config

PEP 585 – Type Hinting Generics In Standard Collections | Python.org

Starting with Python 3.7, when from __future__ import annotations is used, function and variable annotations can parameterize standard collections directly.
haystack: dict[str, list[int]]

You can use a protocol class with isinstance() if you decorate it with the @runtime_checkable class decorator. The decorator adds support for basic runtime structural checks:

isinstance() also works with the predefined protocols in typing such as Iterable.

any command-line argument starting with @ reads additional command-line arguments from the file following the @ character.
This is primarily useful if you have a file containing a list of files that you want to be type-checked

you need 'py.typed' file and also explicitly mentioning it in setup.py

Third party libraries: I sometimes see them annotated, but what people don't suspect, is that you need to include 'py.typed' file with your package in order for it to be discoverable.

My current workaround is using __init__.pyi

this is mypy friendly

if sys.version_info[:2] >= (3, 8):

this isn't

if sys.version_info.minor >= 7:

TypeError: 'type' object is not subscriptable

mypy wereyouhere/extractors/custom.py

wereyouhere/extractors/custom.py:56: error: Unexpected keyword argument "line" for "make" of "Loc"
wereyouhere/extractors/custom.py:144: error: Too few arguments for "make" of "Loc"

mypy wereyouhere

no errors?

$ mypy my/endomondo.py
my/endomondo.py:8: error: Skipping analyzing '.core.common': found module but no type hints or library stubs  [import]
    from .core.common import Paths, get_files
    ^
my/endomondo.py:8: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
my/endomondo.py:39: error: Skipping analyzing '.core.error': found module but no type hints or library stubs  [import]
    from .core.error import Res
    ^
my/endomondo.py:54: error: Skipping analyzing '.core': found module but no type hints or library stubs  [import]
        from .core import stat
    ^
Found 3 errors in 1 file (checked 1 source file)
$ mypy -p my.endomondo
my/endomondo.py: note: In function "workouts":
my/endomondo.py:48: error: Incompatible types in "yield" (actual type "endoapi.endomondo.Workout", expected type "Union[my.endomondo.Workout, Exception]")  [misc]
                yield w
                ^
Found 1 error in 1 file (checked 1 source file)

def testx(x: int) -> None:
    pass

def testy(y: int) -> None:
    pass

test = testx
test = testy

TLDR: messing with all results in problems, don't do that..
I guess I have to be careful on module levels then if I don't want unwanted crap…

try:
  pass
except Exception as e:
  pass

for e in []:
  pass

from typing import TYPE_CHECKING, TypeVar
if TYPE_CHECKING:
    F = TypeVar('F', Callable)
    def lru_cache(f: F) -> F: pass
else:
    from functools import lru_cache

from typing import Any, Callable, TypeVar
FuncT = TypeVar('FuncT', bound=Callable[..., Any])

ok, this + overrides allowed me to type it properly

Just for the record: if someone needs to change the return type of the function inside the decorator and still have typed parameters, you can use a custom mypy plugin that literally takes 15 LoC: https://github.com/dry-python/returns/blob/92eda5574a8e41f4f5af4dd29887337886301ee3/returns/contrib/mypy/decorator_plugin.py

Saved me a lot of time!

Type annotation will require importing modules that you wouldn't need to import without it. However there is now a workaround (quote the name and import in a dead if):

if False:
    from bar_module import bar

def foo(a: 'bar'):
     pass

With that and the idea of string self referencing, JSONType can be defined as:

from typing import Recursive, Union, List, Dict

JSONType = Recursive(
    "JSONType",
    Union[int, float, str, bool, None, List["JSONType"], Dict[str, "JSONType"]]
)

As a workaround, you could try defining your own alias to the property decorator that is annotated to return Any. If you use your custom property decorator (that at runtime behaves exactly like property) for all decorated properties, mypy will silently give all such properties the Any type.

PEP 612 (https://www.python.org/dev/peps/pep-0612/) ended up providing a similar feature. I think we can close this issue now.

@overload
def fetch_data(raw: Literal[True]) -> bytes: ...
@overload
def fetch_data(raw: Literal[False]) -> str: ...
# Fallback overload if the user provides a regular bool

If validation fails pydantic will raise an error with a breakdown of what was wrong:
from pydantic import ValidationError

try:
    User(signup_ts='broken', friends=[1, 2, 'not number'])
except ValidationError as e:
    print(e.json())

Check out mypy with "--no-any-expression" flag and Pydantic.