Python etc

Module fnmatch provides a few functions to work with Unix-like patterns:

from fnmatch import fnmatch

fnmatch('example.py', '*.py')
# True

fnmatch('example.py', '*.cpp')
# False

Internally, it parses the given pattern and compiles it into a regular expression. So, don't expect it to be faster than re. Also, if you want to match actual files in the filesystem, use pathlib.Path.glob instead.

Python provides 2 useless but interesting built-in functions: copyright and license. copyright gives a short overview who owned Python in different moments of history:

>>> copyright()
Copyright (c) 2001-2020 Python Software Foundation.
All Rights Reserved.

Copyright (c) 2000 BeOpen.com.
All Rights Reserved.

Copyright (c) 1995-2001 Corporation for National Research Initiatives.
All Rights Reserved.

Copyright (c) 1991-1995 Stichting Mathematisch Centrum, Amsterdam.
All Rights Reserved.

An license gives not only all Python licenses but also an interesting reading about Python history:

>>> license()
A. HISTORY OF THE SOFTWARE
==========================

Python was created in the early 1990s by Guido van Rossum at Stichting
Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands
as a successor of a language called ABC.  Guido remains Python's
principal author, although it includes many contributions from others.

...

When you have a dict with a deep inheritance (like a configuration file) and path in it specified as a string, there is a fun and short way how to get a value from the dict by the given path:

from functools import reduce
from operator import getitem

d = {'a': {'b': {'c': 13}}}
path = 'a.b.c'
reduce(getitem, path.split('.'), d)
# 13

Some languages, like Java, allow you to mark a class as final that means you can't inherit from it. There is how it can be implemented in a few lines (thanks to Nikita Sobolev for the implementation!):

def _init_subclass(cls, *args, **kwargs) -> None:
    raise TypeError('no subclassing!')

def final(cls):
    setattr(cls, '__init_subclass__', classmethod(_init_subclass))
    return cls

@final
class A:
    pass

class B(A):
    pass
# TypeError: no subclassing!

In python 3.8, PEP-591 introduced typing.final. It doesn't make a runtime check but is processed by mypy at static type checking instead.

The colorsys module converts colors between different representations: RGB, YIQ, HLS, and HSV. Yes, it is in the stdlib!

import colorsys
colorsys.rgb_to_hsv(0.2, 0.4, 0.4)
# (0.5, 0.5, 0.4)

Basically, assert could be a function:

def assert_(test, *args):
    if not test:
        raise AssertionError(*args)

assert_(2 + 2 == 4, 'the world is broken')

However, there are few advantages of assert as directive over assert as a function: 1. All asserts removed on the bytecode compilation step if optimization is enabled. 2. The message is lazy and executed only when needed:

assert False, print("executed")
# executed
# AssertionError: None

assert True, print("not executed")
# (prints nothing)

Try to avoid getting dunder attributes directly. Python provides helper functions for getting of of standard dunder attributes: - type(self) instead of self.__class__ - inspect.getdoc(cls) instead of cls.__doc__ - vars(obj) instead of obj.__dict__ - cls.mro() instead of cls.__mro__

How do you read __init__ word? "underscore underscore init underscore underscore"? Just "init"? There is a convention to read it "dunder init". it was proposed by Mark Jackson in 2002 and popularized by Ned Batchelder in 2006.

If you're going to store data in the descriptor, the reasonable question is "where". 1. If data stored in the descriptor's attribute, it will be shared between all instances of the class where the descriptor is assigned. 2. If data is stored in a dict inside of the descriptor, where the key is hash of class and value is data, it will lead to a memory leak. So, the best solution is to store data in the class itself. But how to name the attribute? @cached_property that we implemented above, relies on the passed function name and it is wrong:

class C:
    @cached_property
    def a(self):
        print('called')
        return 1
    b = a
c = C()

# `a` is cached:
c.a
# called
# 1
c.a
# 1

# but `b` isn't:
c.b
# called
# 1
c.b
# called
# 1

PEP-487 introduced __set_name__ hook. It is called on descriptor assignment to a class attribute and accepts the class itself at the name of the attribute. Let's use it and fix the implementation:

class cached_property:
  def __init__(self, func):
    self.func = func

  def __set_name__(self, owner, name):
    self.name = name

  def __get__(self, obj, cls):
    if obj is None:
      return self
    # we've replaced `self.func.__name__` by `self.name` here
    value = obj.__dict__[self.name] = self.func(obj)
    return value

Decorator @cached_property is an amazing way to simplify your code. It's like the regular @property but remembers the value after the first call:

class C:
  @cached_property
  def p(self):
    print('computing...')
    return 1

c = C()
c.p
# computing...
# 1

c.p
# 1

The implementation is short and relatively simple:

class cached_property:
  def __init__(self, func):
    self.func = func

  def __get__(self, obj, cls):
    if obj is None:
        return self
    value = obj.__dict__[self.func.__name__] = self.func(obj)
    return value

However, there are a few corner-cases, like async functions and threads. Luckily, from Python 3.8 it's a part of standard library (functools.cached_property) and for older versions cached-propery library can be used.

Descriptors are special class attributes with a custom behavior on attribute get, set, or delete. If an object defines __set__ or __delete__, it is considered a data descriptor. Descriptors that only define __get__ are called non-data descriptors. The difference is that non-data descriptors are called only if the attribute isn't presented in __dict__ of the instance. Non-data descriptor:

class D:
  def __get__(self, obj, owner):
    print('get', obj, owner)

class C:
    d = D()

c = C()
c.d
# get <C object at ...> <class 'C'>

# updating __dict__ shadows the descriptor
c.__dict__['d'] = 1
c.d
# 1

Data descriptor:

class D:
  def __get__(self, obj, owner):
    print('get', obj, owner)

  def __set__(self, obj, owner):
    print('set', obj, owner)

class C:
    d = D()

c = C()
c.d
# get <C object at ...> <class 'C'>

# updating __dict__ doesn't shadow the descriptor
c.__dict__['d'] = 1
c.d
# get <C object at ...> <class 'C'>

Magic method __prepare__ on metaclass is called on class creation. It must return a dict instance that then will be used as __dict__ of the class. For example, it can be used to inject variables into the function scope:

class Meta(type):    
    def __prepare__(_name, _bases, **kwargs):
        d = {}
        for k, v in kwargs.items():
            d[k] = __import__(v)
        return d

class Base(metaclass=Meta):
    def __init_subclass__(cls, **kwargs):
      pass

class C(Base, m='math'):
    mypi = m.pi

C.mypi
# 3.141592653589793

C.m.pi
# 3.141592653589793

Python 3.6 introduced a few hooks to simplify things that could be done before only with metaclasses. Thanks to PEP-487. The most useful such hook is __init_subclass__. It is called on subclass creation and accepts the class and keyword arguments passed next to base classes. Let's see an example:

speakers = {}

class Speaker:
  # `name` is a custom argument
  def __init_subclass__(cls, name=None):
    if name is None:
      name = cls.__name__
    speakers[name] = cls

class Guido(Speaker): pass
class Beazley(Speaker, name='David Beazley'): pass

speakers
# {'Guido': __main__.Guido, 'David Beazley': __main__.Beazley}

The class body is the same as, let's say, the function body, with only a few limitations. You can put any statements inside, reuse previous results and so on:

class A:
    print('hello')
    a = 1
    if a:
      b = a + 1
# hello

A.b
# 2

Let's have more fun with frames and recursion. There is sum function that adds 2 natural small numbers by getting down into recursive calls and then counting back the stack size:

import inspect

def _sum(a, b):
    print(a, b)
    if a != 0:
        return _sum(a-1, b)
    if b != 0:
        return _sum(a, b-1)
    return len(inspect.stack())

  def sum(a, b):
    return _sum(a, b) - len(inspect.stack()) - 1

On this Tuesday, a team of 5 authors (including Guido van Rossum) published PEP-622. This is a huge draft in terms of size, complexity, and impact. It is a proposal to extend Python syntax to support structural pattern matching. Think about it as if statement on steroids. A small example using match as switch statement:

def http_error(status: int) -> str:
  match status:
    case 400:
        return 'Bad request'
    case 401:
        return 'Unauthorized'
    case _:
        return 'Something else'

Impractical but reach example:

def inspect(obj) -> None:
  match obj:
    case 0 | 1 | 2:   # matching 2 or more exact values
      print('small number')
    case x if x > 2:  # guards
      print('big positive number')
    case [] | [_]:    # matching sequence
      print('zero or one element sequence')
    case [x, _, *_]:  # unpacking to match rest
      print('2 or more elements sequence')
      print(f'the first element is {x}')
    case {'route': route}:  # matching dicts
      print(f'dict with ONLY `route` key which is {route}')
    case {'route': _, **_}:  # matching rest for dicts
      print(f'dict with `route` key')
    case str() | bytes():  # matching types
      print('something string-like')
    case [x := [_, *_]]:  # walrus and sub-patterns
      print('non-empty list inside a list')
    case _:  # default case
      print('something else')

For objects, the check is implemented via __match__ magic method. For object it does isinstance check. This is why case str() works:

class object:
  @classmethod
  def __match__(cls, obj):
    if isinstance(obj, cls):
      return obj

Also, it is possible to match objects' attributes:

class Point:
  def __init__(self, x, y):
    self.x = x
    self.y = y

match obj:
  case Point(x=0, y=0):
    print('both x and y are zero')
  case Point():
    print('it is a point')
  case _:
    print('something else')

Also, if a class has __match_args__, the given arguments can be positional in the pattern:

class Point:
  __match_args__ = ['x', 'y']

  def __init__(self, x, y):
    self.x = x
    self.y = y

match obj:
  case Point(0, 0):  # here args are positional now
    print('both x and y are zero')

You already can try it using patma. It is a fork of CPython with the reference implementation of the draft. It is expected to land into Python in 3.10 release.

You can implement f-strings in older Python versions by accessing globals and locals from the caller function. It is possible through getting the parent frame from the call stack:

import inspect
from collections import ChainMap
def f(s):
    frame = inspect.stack()[1][0]
    vrs = ChainMap(frame.f_locals, frame.f_globals)
    return s.format(**vrs)

name = '@pythonetc'
f('Hello, {name}')
# 'Hello, @pythonetc'

ChainMap merges locals and globals into one mapping without need to create a new dict. This implementation is a bit more limited, though. While the original f-strings can have any expression inside, our implementation can't:

f'{2-1}'
# '1'
f('{2-1}')
# KeyError: '2-1'

Python doesn't support tail recursion. Hence, it's easy to face RecursionError when implementing recursive algorithms. You can get and change maximum recursion depth with sys.getrecursionlimit and sys.setrecursionlimit functions:

sys.getrecursionlimit()
# 3000

sys.setrecursionlimit(4000)
sys.getrecursionlimit()
# 4000

However, it's a dangerous practice, especially because every new frame on the call stack is quite expensive. Luckily, any recursive algorithm can be rewritten with iterations.

Syntax for decorators is limited by getting attributes and calling objects:

decos = {
  'id': lambda x: x,
}

@decos['id']
def f(): pass
# SyntaxError: invalid syntax

Python 3.9 (via PEP-614) relaxes with restriction allowing to have any expression as a decorator:

decos = {
  'id': lambda x: x,
}

@decos['id']
def f(): pass

f
# <function f at ...>

You can use matrix multiplication to make it confusing (don't try it at home!):

class D:
  f = None
  def __init__(self, name):
    self.name = name

  def __call__(self, *args, **kwargs):
    # on the first call save the function
    if self.f is None:
      self.f = args[0]
      return self
    # on all the next calls call the function
    print(f'hello from {self.name}!')
    return self.f(*args, **kwargs)

  # matrix multiplication logic
  def __matmul__(self, other):
    return lambda f: self(other(f))

# the second `@` is actually the matrix multiplication
@D('a') @D('b')
def f(): pass

f()
# hello from a!
# hello from b!

You can use a simple wrapper function to have any expression in older python versions:

_ = lambda x: x

@_(D('a') @ D('b'))
def f(): pass

Python 3.9 introduces a new function ast.unparse. It accepts a parsed AST and produces a Python code. This code if parsed will produce the same AST:

import ast
tree = ast.parse('a=(1+2)+3 # example')
ast.unparse(tree)
# '\na = 1 + 2 + 3'

It knows nothing about the initial formatting and comments. So, it's not a code formatter but a tool to simplify visual AST analysis. Also, it can be used for code generation.