Python etc

collections.defaultdict allows you to create a dictionary that returns the default value if the requested key is missing (instead of raising KeyError). To create a defaultdict you should provide not a default value but a factory of such values. That allows you to create a dictionary that virtually contains infinite levels of nested dicts, allowing you to do something like d[a][b][c]...[z].

>>> def infinite_dict():
...     return defaultdict(infinite_dict)
...
>>> d = infinite_dict()
>>> d[1][2][3][4] = 10
>>> dict(d[1][2][3][5])
{}

Such behavior is called “autovivification”, the term came from the Perl language.

We often say that a coroutine may be interrupted at the point of any await. Strictly speaking, this is not true. A coroutine indeed may be interrupted on some awaits, but not at any of them. To be interruptable, await should await a future, or a coroutine that awaits a future, and so on. Usually, that await-chain of coroutines ends with awaiting future except for the async functions that have no await in their body:

import asyncio

async def p(x):
    print(x)

async def task(x):
    while True:
        await p(x)

loop = asyncio.get_event_loop()
loop.create_task(task(1))
loop.create_task(task(2))
loop.run_forever()

This code prints 1 infinitely and never prints 2. You may wonder, why to make function async if it has no await int the body. It may happen if you override a parent method that is async but don't need to await in the child implementation. Also, an author of public API may make functions async in case it will have to use await in future releases.

The popular method to declare an abstract method in Python is to use NotImplentedError exception:

def human_name(self):
    raise NotImplementedError

Though it's pretty popular and even has IDE support (PyCharm considers such method to be abstract), this approach has a downside. You get the error only upon method call, not upon class instantiation. Use abc to avoid this problem:

from abc import ABCMeta, abstractmethod
class Service(metaclass=ABCMeta):
    @abstractmethod
    def human_name(self):
        pass

Also be aware that NotImplemented is not the same that NotImplementedError. It's not even an exception. It's a special value (like True and False) that has an absolutely different meaning. Some special methods may return it (e.g., __eq__(), __add__(), etc.) so Python tries to reflect operation. If a.__add__(b) returns NotImplemented, Python tries to call b.__radd__.

To sort a dictionary by its values you use sorted with the custom key function:

>>> d = dict(a=1, c=3, b=2)
>>> sorted(d.items(), key=lambda item: item[1])
[('a', 1), ('b', 2), ('c', 3)]

However, such function already exists in the operator module:

>>> sorted(d.items(), key=itemgetter(1))
[('a', 1), ('b', 2), ('c', 3)]

You can also sort keys instead of items:

>>> sorted(d, key=lambda k: d[k])
['a', 'b', 'c']

Again, this lambda can be replaced with the already existing method:

>>> sorted(d, key=d.get)
['a', 'b', 'c']

All objects in Python are created via the call to the __new__ method. Even if you provide custom __new__ for your class, you have to call super().__new__(...). You might think that object.__new__ is a root implementation that is responsible for the creation of all objects. That is not entirely true. There are several such implementations, and they are incompatible. For example, dict has its own low-level __new__ and objects of types derived from dict can't be created with object.__new__:

In : class D(dict):
...:     pass
...:

In : class A:
...:     pass
...:

In : object.__new__(A)
Out: <__main__.A at 0x7f200c8902e8>

In : object.__new__(D)
...
TypeError: object.__new__(D) is not safe,
use D.__new__()

The default list slicing in Python creates copies. It may be undesirable if a sliced part is too big to be copied, you want it to reflect changes in the list, or even want to modify the slice to affect the original object. To solve the problem with copying a lot of data, one can use itertools.islice. It lets you iterate over the part of the list, but doesn't support indexing or modification. To achieve more than this, we have to write a custom class. Luckily Python provides the suitable abstract base class: collections.abc.MutableSequence. You only need to override __getitem__, __setitem__, __delitem__, __len__ and insert. This is the example of how you do it. It doesn't support deletion and inserting, but supports slicing slices and modifications.

Every call to next(x) returns the new value from the x iterator unless an exception is raised. If this is StopIteration, it means the iterator is exhausted and can supply no more values. If a generator is iterated, it automatically raises StopIteration upon the end of the body:

>>> def one_two():
...     yield 1
...     yield 2
...
>>> i = one_two()
>>> next(i)
1
>>> next(i)
2
>>> next(i)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
StopIteration

StopIteration is automatically handled by tools that calls next for you:

>>> list(one_two())
[1, 2]

The problem is, any unexpected StopIteration that is raised within a generator causes it to stop silently instead of actually raising an exception:

def one_two():
    yield 1
    yield 2

def one_two_repeat(n):
    for _ in range(n):
        i = one_two()
        yield next(i)
        yield next(i)
        yield next(i)

print(list(one_two_repeat(3)))

The last yield here is a mistake: StopIteration is raised and makes list(...) to stop the iteration. The result is [1, 2], surprisingly. However, that was changed in Python 3.7. Such foreign StopIteration is now replaced with RuntimeError:

Traceback (most recent call last):
  File "test.py", line 10, in one_two_repeat
    yield next(i)
StopIteration

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 12, in <module>
    print(list(one_two_repeat(3)))
RuntimeError: generator raised StopIteration

You can enable the same behavior since python3.5 by from __future__ import generator_stop.

itertools.cycle(x) gives you an iterator that repeats all that x yields indefinitely:

In : c = cycle([1, 2, 3])
In : next(c)
Out: 1
In : next(c)
Out: 2
In : next(c)
Out: 3
In : next(c)
Out: 1
In : next(c)
Out: 2
In : next(c)
Out: 3
In : next(c)
Out: 1

Note, that not all iterables can be reiterated, so cycle makes a copy of all elements so it can yield them again. That can be unnecessarily inefficient for iterables that can, e. g. lists. You probably shouldn't care about it unless your list is big enough. If this is the case, you should reimplement cycle somehow like this:

def safe_cycle(iterable):
    while True:
        empty = True
        for x in iterable:
            empty = False
            yield x

        if empty:
            return

In Python, an object is physically destroyed and unloaded from the memory when nobody has a reference to it anymore. That is also true for any number of objects that have cross-references but are not available for the rest of the objects (so-called reference cycles). There might be a case when you want to have a reference to an object but don't want to prevent its destruction if your reference is the last one. The reference you want to have in this case is called weak. Weak references are extremely helpful for old kind of caches or indexes. The weakref module allows you to create weak references explicitly or you dictionaries with them inside. Unfortunately, not all types support weak referencing; sometimes you have to create trivial subclasses:

class List(list):
    pass

weakref.ref creates an object that you must call to get the original value:

>>> x = List()
>>> r = weakref.ref(x)
>>> r()
[]
>>> del x
>>> r
<weakref at 0x7f302db036d8; dead>
>>> r()
>>>

weakref.proxy creates an object that acts almost like a standard reference:

>>> x = List()
>>> p = weakref.proxy(x)
>>> p
<weakproxy at 0x7f302db03688 to List at 0x7f302db87ea8>
>>> list(p)
[]
>>> p.append(42)
>>> p[0]
42
>>> del x
>>> p
<weakproxy at 0x7f302db03688 to NoneType at 0x8a1a80>
>>> p[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ReferenceError: weakly-referenced object no longer exists

Some code you are using may print data you are interested in stdout instead of providing some API that is usable within a program (returning a string, for example). Instead of refactoring such code you may use the contextlib.redirect_stdout context manager that allows temporary redirecting stdout to any custom file-like object. In conjuncture with io.StringIO, it allows capturing output to a variable.

from contextlib import redirect_stdout
from io import StringIO

s = StringIO()
with redirect_stdout(s):
    print(42)

print(s.getvalue())

There is also contextlib.redirect_stderr available for redirecting sys.stderr.

Sometimes you need to have a queue in your program, i. e. a container where you put elements from one side and remove them from another. list can be such a container:

In : lst = [1, 2, 3]
In : lst.pop()
Out: 3
In : lst
Out: [1, 2]
In : lst[:0] = [4]  # push
In : lst
Out: [4, 1, 2]

However, using list doesn't only look eerie (look at that push), but also is quite inefficient.

In : lst = [0] * 10_000_000

In : %timeit lst[:0] = [1]
9.5 ms ± 111 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

In : %timeit lst.pop()
84.3 ns ± 4.01 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

As you can see, on my machine pop is 100 times faster than “push”. This is how list works: elements can be easily added to or removed from the end of the list, but to remove the first element, Python needs to create a new list from scratch. What you really want to use for this problem is collections.deque. It's designed to be used as a queue:

In : d = deque([1] * 100_000_000)
In : %timeit d.popleft()
65 ns ± 0.436 ns per loop (mean ± std. dev. of 7 runs, 10000000 loops each)

The standard json module has a command line interface that can be useful to prettify JSON by python alone. The module for this is called json.tool and is meant to be called like this:

$ echo '{"a": [], "b": "c"}' | python -m json.tool
{
    "a": [],
    "b": "c"
}

Python functions can return multiple values:

def edges(lst):
    return lst[0], lst[-1]

first, last = edges([1, 2, 3])
assert first == 1
assert last == 3

In truth, lst[0], lst[-1] is a simple tuple. It's returned as usual and then unpacked to first and last:

result = edges([1, 2, 3])
assert isinstance(result, tuple)
first, last = result
assert first == 1
assert last == 3

Usually, you don't care about it at all. However, all these things come to the surface when you use type hints. You have to define the function return value as tuple:

def edges(lst) -> Tuple[int, int]:
    return lst[0], lst[-1]

Calling that function is even harder. You may think that you can do something like this:

first: int, last: int = edges([1, 2, 3])

Or at least this:

first, last: Tuple[int, int] = edges([1, 2, 3])

But both ways are incorrect. This is the only reasonable thing you can do to annotate these variables:

first: int
last: int
first, last = edges([1, 2, 3])

Python provides the powerful library to work with date and time: datetime. The interesting part is, datetime objects have the special interface for timezone support (namely the tzinfo attribute), but this module only has limited support of its interface, leaving the rest of the job to different modules. The most popular module for this job is pytz. The tricky part is, pytz doesn't fully satisfy tzinfo interface. The pytz documentation states this at one of the first lines: “This library differs from the documented Python API for tzinfo implementations.” You can't use pytz timezone objects as the tzinfo attribute. If you try, you may get the absolute insane results:

In : paris = pytz.timezone('Europe/Paris')
In : str(datetime(2017, 1, 1, tzinfo=paris))
Out: '2017-01-01 00:00:00+00:09'

Look at that +00:09 offset. The proper use of pytz is following:

In : str(paris.localize(datetime(2017, 1, 1)))
Out: '2017-01-01 00:00:00+01:00'

Also, after any arithmetic operations, you should normalize your datetime object in case of offset changes (on the edge of the DST period for instance).

In : new_time = time + timedelta(days=2)
In : str(new_time)
Out: '2018-03-27 00:00:00+01:00'
In : str(paris.normalize(new_time))
Out: '2018-03-27 01:00:00+02:00'

Since Python 3.6, it's recommended to use dateutil.tz instead of pytz. It's fully compatible with tzinfo, can be passed as an attribute, doesn't require normalize, though works a bit slower. If you are interested why pytz doesn't support datetime API, or you wish to see more examples, consider reading the decent article on the topic.

It's a pretty common task to choose the right subclass according to some parameter provided by a user. For example, a user may choose between ChatMessage and EmailMessage by sending 'chat' or 'email' respectively. You don't have to write all that ifs manually. Python metaprogramming tools allow you to do it more elegant. The standard practices include generation a class name by the parameter provided, putting all available descendants into a dictionary (manually or with decorators magic), using metaclasses and more. The more detailed review is in my article on the subject.

mypy lets you view an inferred type of any expression. That could be useful if you don't understand why mypy is not happy with your code. It's done with the reveal_type function:

class User:
   def __init__(self, name: str) -> None:
       self._name = name
   def get_name_length(self) -> int:
       reveal_type(self._name)
       return len(self._name)

$ mypy test.py
test.py:6: error: Revealed type is 'builtins.str'

Note, that reveal_type is a pseudo-function that is only understood by mypy and executed during the analysis, not in runtime. There is no such function for Python itself, so you have to remove all reveal_type calls before running the program. For the same reason, you don't need to import reveal_type, it's always available for mypy. Another useful pseudo-function is reveal_locals, it shows types of all local variables at once.

To create a class method, you should use the @classmethod decorator. This method can be called from the class directly, not from its instances, and accepts the class as a first argument (usually called cls, not self). However, there are two implicit class methods in Python data model: __new__ and __init_subclass__. They work exactly as though they are decorated with @classmethod except they aren't. (__new__ creates new instances of a class, __init_subclass__ is a hook that is called when a derived class is created.)

class Foo:
    def __new__(cls, *args, **kwargs):
        print(cls)
        return super().__new__(
            cls, *args, **kwargs
        )

Foo()  # <class '__main__.Foo'>

matplotlib is a complex and flexible Python plotting library. It's supported by a wide range of products, Jupyter and Pycharm including. This is how you draw a simple fractal figure with matplotlib: https://repl.it/@VadimPushtaev/myplotlib

When you use the multiprocessing module, and there is an exception in one of the processes, it's propagated to the main program using pickling. The exception is pickled, passed to another process and then unpickled back. However, pickling exceptions may be tricky. Exception is created with any number of arguments that are stored in the attribute named args. The same arguments are used to recreate an Exception object during unpickling. However, that might not work as you expect if inheritance is in use. Look at the example:

import pickle

class TooMuchWeightError(Exception):
    def __init__(self, weight):
        super().__init__()
        self._weight = weight

pickled = pickle.dumps(TooMuchWeightError(42))
pickle.loads(pickled)

TooMuchWeightError.__init__ calls Exception.__init__. Exception.__init__ sets args equal to the empty tuple. This empty tuple is used as arguments during unpickling which obviously leads to:

TypeError: __init__() missing 1 required positional argument: 'weight'

A workaround is either not to call super().__init__() at all (which is usually not a nice thing to do for a subclass) or pass all arguments to the parent's constructor explicitly:

class TooMuchWeightError(Exception):
    def __init__(self, weight):
        super().__init__(weight)
        self._weight = weight