Python etc

If you want a context manager to suspend coroutine on entering or exiting context, you should use asynchronous context managers. Instead of calling m.__enter__() and m.__exit__() Python does await m.__aenter__() and await m.__aexit__() respectively. Asynchronous context managers should be used with async with syntax:

import asyncio

class Slow:
    def __init__(self, delay):
        self._delay = delay

    async def __aenter__(self):
        await asyncio.sleep(self._delay / 2)

    async def __aexit__(self, *exception):
        await asyncio.sleep(self._delay / 2)

async def main():
    async with Slow(1):
        print('slow')

loop = asyncio.get_event_loop()
loop.run_until_complete(main())

You can add unicode characters in a string literal not only by its number, but by also by its name.

>>> '\N{EM DASH}'
'—'
>>> '\u2014'
'—'

It’s also compatible with f-strings:

>>> width = 800
>>> f'Width \N{EM DASH} {width}'
'Width — 800'

Vacancies time! I’m looking for an experienced Python developer to work with me and my team at Mail.Ru Group in Moscow, Russia. Tons of fun and unique experience guaranteed. Contact @pushtaev. The friend of mine is also hiring. He would be happy to find a Python developer for his team. Vacancy description (rus). Contact @karbachinsky.

If you want objects of a class to have an auto-incremented ID, you can make it happen by tracking current ID in the class attribute:

class Task:
    _task_id = 0

    def __init__(self):
        self._id = self._task_id
        type(self)._task_id += 1

Mind, that you can't do self._task_id += 1. That creates the _task_id attribute within the instance, not the class. You should consider using a factory method instead of __init__ to make it look prettier:

class Task:
    _task_id = 0

    def __init__(self, task_id):
        self._id = task_id

    @classmethod
    def create(cls):
        obj = cls(cls._task_id)
        cls._task_id += 1
        return obj

This version is also easier to test since any custom ID can be easily provided.

break statement suppresses exception if used in the finally clause even when the except block is not presented:

for i in range(10):
    try:
        1 / i
    finally:
        print('finally')
        break
    print('after try')

print('after while')

Output:

finally
after while

The same is true for continue, however it can’t be used in finally until Python 3.8:

SyntaxError: 'continue' not supported inside 'finally' clause

In asyncio, the common practice to schedule execution of some code at a later time is to spawn a task that does await asyncio.sleep(x):

import asyncio

async def do(n=0):
    print(n)
    await asyncio.sleep(1)
    loop.create_task(do(n + 1))
    loop.create_task(do(n + 1))

loop = asyncio.get_event_loop()
loop.create_task(do())
loop.run_forever()

However, creating a new task may be expensive and is not necessary if you aren't planning to any asynchronous operations (like the do function in the example). Another way to do this is to use loop.call_later and loop.call_at functions that schedule an asynchronous callback to be called:

import asyncio                     
                                   
def do(n=0):                       
    print(n)                       
    loop = asyncio.get_event_loop()
    loop.call_later(1, do, n+1)    
    loop.call_later(1, do, n+1)    
                                   
loop = asyncio.get_event_loop()    
do()                               
loop.run_forever()

Instead of modifying the decorated function you can create another callable class to return its instances instead of a function:

class CallableWithOrig:
    def __init__(self, to_call, orig):
        self._to_call = to_call
        self._orig = orig
    
    def __call__(self, *args, **kwargs):
        return self._to_call(*args, **kwargs)

    @property
    def orig(self):
        if isinstance(self._orig, type(self)):
            return self._orig.orig
        else:
            return self._orig

class SavingOrig:
    def __init__(self, another_decorator):
        self._another = another_decorator
  
    def __call__(self, f):
        return CallableWithOrig(self._another(f), f)

saving_orig = SavingOrig

This is the last chapter of saving_orig’s adventures, view full code here.

If a decorator you are writing becomes too complicated, it may be reasonable to transform it from a function to a class with the __call__ method

class SavingOrig:
    def __init__(self, another_decorator):
        self._another = another_decorator
  
    def __call__(self, f):
        decorated = self._another(f)
        if hasattr(f, 'orig'):
            decorated.orig = f.orig
        else:
            decorated.orig = f
        return decorated

saving_orig = SavingOrig

The last line allows you both to name class with camel case and keep the decorator name in snake case.

The @saving_orig decorator doesn’t really do what we want if there more than one decorator applied to a function. We have to call orig for each such decorator:

import json
from functools import wraps

def saving_orig(another_decorator):
    def decorator(f):
        decorated = another_decorator(f)
        decorated.orig = f
        return decorated

    return decorator

def ensure_list(f):
    ...

def ensure_ints(*, default=None):
    ...

@saving_orig(ensure_ints(default=42))
@saving_orig(ensure_list)
def load_data(string):
    return json.loads(string)

for f in (
    load_data,
    load_data.orig,
    load_data.orig.orig,
):
    print(repr(f('"X"')))

Output:

[42]
['X']
'X'

We can fix it by supporting arbitrary number of decorators as saving_orig arguments:

def saving_orig(*decorators):
    def decorator(f):
        decorated = f
        for d in reversed(decorators):
            decorated = d(decorated)
        decorated.orig = f
        return decorated

    return decorator

...

@saving_orig(
  ensure_ints(default=42),
  ensure_list,
)
def load_data(string):
    return json.loads(string)

for f in (
    load_data,
    load_data.orig,
):
    print(repr(f('"X"')))

Output:

[42]
'X'

Another solution is to make saving_orig smart enough to pass orig from one decorated function to another:

def saving_orig(another_decorator):
    def decorator(f):
        decorated = another_decorator(f)
        if hasattr(f, 'orig'):
            decorated.orig = f.orig
        else:
            decorated.orig = f
        return decorated

    return decorator

@saving_orig(ensure_ints(default=42))
@saving_orig(ensure_list)
def load_data(string):
    return json.loads(string)

The @saving_orig mentioned above accepts another decorator as an argument. What if that decorator can be parametrized? Well, since parameterized decorator is a function that returns an actual decorator, this case is handled automatically:

import json
from functools import wraps

def saving_orig(another_decorator):
    def decorator(f):
        decorated = another_decorator(f)
        decorated.orig = f
        return decorated

    return decorator

def ensure_ints(*, default=None):
    def decorator(f):
        @wraps(f)
        def decorated(*args, **kwargs):
            result = f(*args, **kwargs)
            ints = []
            for x in result:
                try:
                    x_int = int(x)
                except ValueError:
                    if default is None:
                        raise
                    else:
                        x_int = default
                ints.append(x_int)
            return ints
        return decorated
    return decorator

@saving_orig(ensure_ints(default=0))
def load_data(string):
    return json.loads(string)

print(repr(load_data('["2", "3", "A"]')))
print(repr(load_data.orig('["2", "3", "A"]')))

If all decorators you are working with are created via functools.wraps you can use the __wrapped__ attribute to access the undecorated function:

import json
from functools import wraps

def ensure_list(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        result = f(*args, **kwargs)

        if isinstance(result, list):
            return result
        else:
            return [result]

    return decorated

@ensure_list
def load_data(string):
    return json.loads(string)

print(load_data('3'))      # [3]
print(load_data.__wrapped__('4')) # 4

Mind, however, that it doesn’t work for functions that are decorated by more than one decorator: you have to access __wrapped__ for each decorator applied:

def ensure_list(f):
    ...

def ensure_ints(f):
    @wraps(f)
    def decorated(*args, **kwargs):
        result = f(*args, **kwargs)
        return [int(x) for x in result]

    return decorated

@ensure_ints
@ensure_list
def load_data(string):
    return json.loads(string)

for f in (
    load_data,
    load_data.__wrapped__,
    load_data.__wrapped__.__wrapped__,
):
    print(repr(f('"4"')))

Output:

[4]
['4']
'4'

Sometimes you want to use both decorated and undecorated versions of a function. The easiest way to achieve that is to forgo the special decorator syntax (the one with @) and create the decorated function manually:

import json

def ensure_list(f):
    def decorated(*args, **kwargs):
        result = f(*args, **kwargs)

        if isinstance(result, list):
            return result
        else:
            return [result]

    return decorated

def load_data_orig(string):
    return json.loads(string)
  
load_data = ensure_list(load_data_orig)

print(load_data('3'))     # [3]
print(load_data_orig('4')) 4

Alternatively, you can write another decorator, that decorate a function while preserving its original version in the orig attribute of the new one:

import json

def saving_orig(another_decorator):
    def decorator(f):
        decorated = another_decorator(f)
        decorated.orig = f
        return decorated

    return decorator

def ensure_list(f):
    ...

@saving_orig(ensure_list)
def load_data(string):
    return json.loads(string)

print(load_data('3'))      # [3]
print(load_data.orig('4')) # 4

You can use any object as a dictionary key in Python as long as it implements the __hash__ method. This method can return any integer as long as the only requirement is met: equal objects should have equal hashes (not vice versa). You also should avoid using mutable objects as keys, because once the object becomes not equal to the old self, it can't be found in a dictionary anymore. There is also one bizarre thing that might surprise you during debugging or unit testing:

...: class A:
...:     def __init__(self, x):
...:         self.x = x
...:
...:     def __hash__(self):
...:         return self.x
...:
In : hash(A(2))
Out: 2
In : hash(A(1))
Out: 1
In : hash(A(0))
Out: 0
In : hash(A(-1))  # sic!
Out: -2
In : hash(A(-2))
Out: -2

In CPython -1 is internally reserved for error states, so it's implicitly converted to -2.

pip -e allows you install a package in editable mode: py-files are no actually copied and local changes count:

(main) $ find zzz
zzz
zzz/setup.py
zzz/zzz.py

(main) $ cat zzz/setup.py
#!/usr/bin/env python3.6

from setuptools import find_packages, setup

setup(
    name='zzz',
    packages=find_packages(include=['zzz']),
)

(main) $ cat zzz/zzz.py
zzz = 'zzz'

(main) $ pip install -e zzz
Obtaining file:///home/v.pushtaev/zzz
Installing collected packages: zzz
  Found existing installation: zzz 0.0.0
    Uninstalling zzz-0.0.0:
      Successfully uninstalled zzz-0.0.0
  Running setup.py develop for zzz
Successfully installed zzz
You are using pip version 10.0.1, however version 19.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

(main) $ python -c 'import zzz; print(zzz.zzz)'
zzz

(main) $ echo 'zzz = "yyy"' > zzz/zzz.py

(main) $ python -c 'import zzz; print(zzz.zzz)'
yyy

pip does it by creating egg-link instead of copying files:

(main) $ cat /home/v.pushtaev/.ve/main/lib/python3.6/site-packages/zzz.egg-link
/home/v.pushtaev/zzz

Probably the most common newbie mistake with Python is providing a mutable object as a default function argument. That object is shared between all function calls that can lead to bizarre results:

def append_length(lst=[]):
   lst.append(len(lst))
   return lst

print(append_length([1, 2])) # [1, 2, 2]
print(append_length())       # [0]
print(append_length())       # [0, 1]

However, for various caches sharing may be a good thing:

def fact(x, cache={0: 1}):
   if x not in cache:
       cache[x] = x * fact(x - 1)

   return cache[x]

print(fact(5))

In this example, we store calculated factorial values inside the default function value. It can even be extracted:

>>> fact.__defaults__
({0: 1, 1: 1, 2: 2, 3: 6, 4: 24, 5: 120},)

Storing and sending object via network as bytes is a huge topic.Let’s discuss some tools that are usually used for that in Python and their advantages and disadvantages. As an example I’ll try to serialize the Cities object which contains some City objects as well as their order. Here is four method you can use: 1. JSON. It’s human readable, easy to use, but consumes a lot of memory. The same is true for other like YAML or XML.

class City:
    def to_dict(self):
        return dict(
            name=self._name,
            country=self._country,
            lon=self._lon,
            lat=self._lat,
        )
class Cities:
    def __init__(self, cities):
        self._cities = cities

    def to_json(self):
        return json.dumps([
            c.to_dict() for c in self._cities
        ]).encode('utf8')

2. Pickle. Pickle is native for Python, can be customized and consumes less memory than JSON. The downside is you have to use Python to unpickle the data.

class Cities:
    def pickle(self):
        return pickle.dumps(self)

3. Protobuf (and other binary serializers such as msgpack). Consumes even less memory, can be used from any other programming languages, but but require custom schema:

syntax = "proto2";


message City {
    required string name = 1;
    required string country = 2;
    required float lon = 3;
    required float lat = 4;
}

message Cities {
    repeated City cities = 1;
}

class City:
    def to_protobuf(self):
        result = city_pb2.City()
        result.name = self._name
        result.country = self._country
        result.lon = self._lon
        result.lat = self._lat

        return result

class Cities:
    def to_protobuf(self):
        result = city_pb2.Cities()
        result.cities.extend([
            c.to_protobuf() for c in self._cities
        ])

        return result

4. Manual. You can manually pack and unpack data with the struct module. I allows you to consume the absolute minimum amount of memory, but protobuf still can be a better choice since it supports versioning and explicit schemas.

class City:
    def to_bytes(self):
        name_encoded = self._name.encode('utf8')
        name_length = len(name_encoded)

        country_encoded = self._country.encode('utf8')
        country_length = len(country_encoded)

        return struct.pack(
            'BsBsff',
            name_length, name_encoded,
            country_length, country_encoded,
            self._lon, self._lat,

class Cities:
    def to_bytes(self):
        return b''.join(
            c.to_bytes() for c in self._cities
        )

Any JSON is syntaxicaly correct Python code. However, true, false and null are not defined by default. Defining them makes it possible to use eval as a JSON parser (which isn’t a good idea anyway):

$ cat json
{"$id":"1","currentDateTime":"2019-04-25T14:16Z","utcOffset":"00:00:00","isDayLightSavingsTime":false,"dayOfTheWeek":"Thursday","timeZoneName":"UTC","currentFileTime":132006753872039629,"ordinalDate":"2019-115","serviceResponse":null}

>>> null = None
>>> true = True
>>> false = False
>>> with open('json') as f:
...     j = eval(f.read())
...
>>> j
{'currentFileTime': 132006753872039629, 'isDayLightSavingsTime': False, 'dayOfTheWeek': 'Thursday', 'utcOffset': '00:00:00', 'serviceResponse': None, '$id': '1', 'timeZoneName': 'UTC', 'ordinalDate': '2019-115', 'currentDateTime': '2019-04-25T14:16Z'}

You may wonder why we have to pass locals() as a parameter in the previous example. Is it possible to get the locals of the caller in the callee? It is indeed, but you have to mess with the inpsect module:

import inspect

def shorthand_dict(names):
    lcls = inspect.currentframe().f_back.f_locals
    return {k: lcls[k] for k in names}

context = dict(user_id=42, user_ip='1.2.3.4')
mode = 'force'
action_type = 7

shorthand_dict([
    'context',
    'mode',
    'action_type',
])

You can go even further and use something like this — https://github.com/alexmojaki/sorcery:

from sorcery import dict_of
dict_of(context, mode, action_type)