Python etc

PEP-604 (landed in Python 3.10) introduced a new short syntax for typing.Union (as I predicted, but I messed up union with intersection, shame on me):

def greet(name: str) -> str | None:
  if not name:
    return None
  return f"Hello, {name}"

You already can use it in older Python versions by adding from __future__ import annotations, type checkers will understand you.

Often, your type annotations will have circular dependencies. For example, Article has an attribute category: Category, and Category has attribute articles: list[Article]. If both classes are in the same file, adding from __future__ import annotations would solve the issue. But what if they are in different modules? Then you can hide imports that you need only for type annotations inside of the if TYPE_CHECKING block:

from __future__ import annotations
from dataclasses import dataclass
from typing import TYPE_CHECKING

if TYPE_CHECKING:
  from .category import Category

@dataclass
class Article:
  category: Category

Fun fact: this constant is defined as TYPE_CHECKING = False. It won't be executed at runtime, but the type checker is a static analyzer, it doesn't care.

PEP-563 (landed in Python 3.7) introduced postponed evaluation of type annotations. That means, all your type annotations aren't executed at runtime but rather considered strings. The initial idea was to make it the default behavior in Python 3.10 but it was postponed after a negative reaction from the community. In short, it would be in some cases impossible to get type information at runtime which is crucial for some tools like pydantic or typeguard. For example, see pydantic#2678. Either way, starting from Python 3.7, you can activate this behavior by adding from __future__ import annotations at the beginning of a file. It will improve the import time and allow you to use in annotations objects that aren't defined yet. For example:

class A:
  @classmethod
  def create(cls) -> A:
    return cls()

This code will fail at import time:

Traceback (most recent call last):
  File "tmp.py", line 1, in <module>
    class A:
  File "tmp.py", line 3, in A
    def create(cls) -> A:
NameError: name 'A' is not defined

Now add the magic import, and it will work:

from __future__ import annotations

class A:
  @classmethod
  def create(cls) -> A:
    return cls()

Another solution is to manually make annotations strings. So, instead of -> A: you could write -> 'A':.

Now, let's see how to dump stack trace when a specific signal is received. We will use SIGUSR1 but you can do the same for any signal.

import faulthandler
from signal import SIGUSR1
from time import sleep

faulthandler.register(SIGUSR1)
sleep(60)

Now, in a new terminal, find out the PID of the interpreter. If the file is named tmp.py, this is how you can do it (we add [] in grep to exclude the grep itself from the output):

ps -ax | grep '[t]mp.py'

The first number in the output is the PID. Now, use it to send the signal for PID 12345:

kill -SIGUSR1 12345

And back in the terminal with the running script. You will see the stack trace:

Current thread 0x00007f22edb29740 (most recent call first):
  File "tmp.py", line 6 in <module>

This trick can help you to see where your program has frozen without adding logs to every line. However, a better alternative can be something like py-spy which allows you to dump the current stack trace without any changes in the code.

The module faulthandler allows registering a handler that will dump the current stack trace in a specific file (stderr by default) upon receiving a specific signal or every N seconds. For example, dump stack trace every 2 seconds: import faulthandler from time import sleep faulthandler.dump_traceback_later( timeout=2, repeat=True, ) for i in range(5): print(f"iteration {i}") sleep(1) Output: iteration 0 iteration 1 Timeout (0:00:02)! Thread 0x00007f8289147740 (most recent call first): File "tmp.py", line 10 in <module> iteration 2 iteration 3 Timeout (0:00:02)! Thread 0x00007f8289147740 (most recent call first): File "tmp.py", line 10 in <module> iteration 4

The module atexit allows registering hooks that will be executed when the program terminates. There are only a few cases when it is NOT executed: + When os._exit (don't confuse with sys.exit) is called. + When the interpreter failed with a fatal error. + When the process is hard-killed. For example, someone executed kill -9 or the system is ran out of memory. In all other cases, like an unhandled exception or sys.exit, the registered hooks will be executed. A few use cases: + Finish pending jobs + Send pending log messages into the log system + Save interactive interpreter history However, keep in mind that there is no way to handle unhandled exceptions using atexit because it is executed after the exception is printed and discarded. import atexit atexit.register(print, 'FINISHED') 1/0 Output: Traceback (most recent call last): File "example.py", line 4, in <module> 1/0 ZeroDivisionError: division by zero FINISHED

channel = '@pythonetc' print(f'Happy new Year, {channel}!') # there are our top posts from 2021 by_likes = { 'join-lists': 236, 'dev-mode': 181, 'is-warning': 170, 'str-concat': 166, 'class-scope': 149, } by_forwards = { 'class-scope': 111, 'dev-mode': 53, 'join-lists': 50, 'str-concat': 44, 'eval-strategy': 36, } by_views = { '__path__': 7_736, 'dev-mode': 7_113, 'immutable': 6_757, 'class-scope': 6_739, 'sre-parse': 6_661, } from datetime import date from textwrap import dedent if date.today().year == 2022: print(dedent(""" The season 2.6 is coming! This is what awaits: """)) print( 'native telegram reactions instead of buttons', 'deep dive into garbage collection, generators, and coroutines', 'the season is still ran by @orsinium', 'as always, guest posts and donations are welcome', sep='\n', ) print('See you next year \N{Sparkling Heart}!')

$donate

Finally, you can express your gratitude towards the authors by donating via the new telegram donation service. Thanks and hope to see you in the third season one day.

You are also welcome to consider joining my team. We still develop the voice assistant and beautiful devices for her to live in:

Hi there! As you’ve probably already noticed there is no activity here at the moment. I hereby declare the second season to be officially over. I currently have no specific plan nor ideas for the third season. DM @pushtaev if you do. For the time being, you can enjoy the archive. Posts of the channel are mostly relevant for this day.

Internally, the module re uses 2 undocumented libraries: + sre_parse to parse regular expressions into an abstract syntax tree. + sre_compile to compile parsed expression. The first one can be used to see how a regexp was parsed by Python. There are many better tools and services (like regex101.com) to debug regular expressions but this one is already in the stdlib.

>>> import sre_parse
>>> sre_parse.parse(r'([Pp]ython)\s?etc').dump()
SUBPATTERN 1 0 0
  IN
    LITERAL 80
    LITERAL 112
  LITERAL 121
  LITERAL 116
  LITERAL 104
  LITERAL 111
  LITERAL 110
MAX_REPEAT 0 1
  IN
    CATEGORY CATEGORY_SPACE
LITERAL 101
LITERAL 116
LITERAL 99

JSON states for "JavaScript Object Notation". It's a subset of JavaScript and representation of values is based on how they are represented in JavaScript:

import json
json.dumps(1)     # '1'
json.dumps(1.2)   # '1.2'
json.dumps('hi')  # '"hi"'
json.dumps({})    # '{}'
json.dumps([])    # '[]'
json.dumps(None)  # 'null'
json.dumps(float('inf'))  # 'Infinity'
json.dumps(float('nan'))  # 'NaN'

The last two examples are valid JavaScript but explicitly forbidden by RFC 4627 "The application/json Media Type for JSON": > Numeric values that cannot be represented as sequences of digits (such as Infinity and NaN) are not permitted. And so, the inf / nan values, successfully serialized in Python, can fail deserialization in another language. For example, in Go:

import "encoding/json"

func main() {
    var v float64
    err := json.Unmarshal(`Infinity`, &v)
    println(err)
    // Output: invalid character 'I' looking for beginning of value
}

To prevent producing invalid JSON, pass allow_nan=False argument:

json.dumps(float('nan'), allow_nan=False)
# ValueError: Out of range float values are not JSON compliant

Python 3.7 introduced Development Mode. The mode can be activated with the -X dev argument and it makes the interpreter produce some helpful warnings. For instance: + Unclosed files. + Unawaited coroutines. + Unknown encoding for str.encode (by default, it is unchecked for empty strings). + Memory allocation issues.

$ echo 'open("/dev/null")' > tmp.py
$ python3 -X dev tmp.py
tmp.py:1: ResourceWarning: unclosed file <_io.TextIOWrapper name='/dev/null' mode='r' encoding='UTF-8'>
  open("/dev/null")
ResourceWarning: Enable tracemalloc to get the object allocation traceback

by @PavelDurmanov

Modules have a magic attribute called __path__. Whenever you're doing subpackage imports, __path__ is being searched for that submodule. __path__ looks like a list of path strings, e.g ["foo/bar", "/path/to/location"]. So if you do from foo import bar, or import foo.bar, foo's __path__ is being searched for bar. And if found - loaded. You can play around with __path__ to test it out. Create simple Python module anywhere on your system: $ tree . └── foo.py $ cat foo.py def hello(): return "hello world" Then, run the interpreter there and do the following: python >>> import os >>> os.__path__ = ["."] >>> from os.foo import hello >>> hello() 'hello world' As you can see, foo is now available under os: python >>> os.foo <module 'os.foo' from './foo.py'>

by @PavelDurmanov

Have you ever wondered how do relative imports work? Im pretty sure that you've done something like that at some point:

from . import bar
from .bar import foo

It's using a special magic attribute on the module called __package__. Lets say you have the following structure:

foo/
    __init__.py
    bar/
        __init__.py
main.py

The value of __package__ for foo/__init__.py is set to "foo", and for foo/bar/__init__.py its "foo.bar". Note that for main.py __package__ isn't set, that's because main.py is not in a package. So when you're doing from .bar import buz within foo/__init__.py, it simply appends "bar" to foo/__init__.py's __package__ attribute, esentially it gets translated to from foo.bar import buz. You can actually hack __package__, e.g:

>>> __package__ = "re"
>>> from . import compile
>>> compile
<function compile at 0x10e0ee550>