Python etc

Today's post is written by @orsinium, the author of @itgram_channel.

The standard json module has a command line interface that can be useful to prettify JSON by python alone. The module for this is called json.tool and is meant to be called like this:

$ echo '{"a": [], "b": "c"}' | python -m json.tool
{
    "a": [],
    "b": "c"
}

In the previous example, we can save some time by avoiding slicing the string again and again but asking the re module to search starting from a different position instead. That requires some changes. First, re.search doesn' support searching from a custom position, so we have to compile the regular expression manually. Second, ^ means the real start for the string, not the position where the search started, so we have to manually check that the match happened at the same position.

In [1]: from example import *

In [2]: %timeit xml_to_tree_slow(text)
356 µs ± 16.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In [3]: %timeit xml_to_tree_fast(text)
294 µs ± 6.15 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

In some languages, you can use \G assertion. It matches at the position where the previous match is ended. That allows writing finite automata that walk through string word by word (where word is defined by the regex). However, there is no such thing in Python. The proper workaround is to manually track the position and pass the substring to regex functions:

import re
import json

text = '<a><b>foo</b><c>bar</c></a><z>bar</z>'
regex = '^(?:<([a-z]+)>|</([a-z]+)>|([a-z]+))'

stack = []
tree = []

pos = 0
while len(text) > pos:
    error = f'Error at {text[pos:]}'
    found = re.search(regex, text[pos:])
    assert found, error
    pos += len(found[0])
    start, stop, data = found.groups()

    if start:
        tree.append(dict(
            tag=start,
            children=[],
        ))
        stack.append(tree)
        tree = tree[-1]['children']
    elif stop:
        tree = stack.pop()
        assert tree[-1]['tag'] == stop, error
        if not tree[-1]['children']:
            tree[-1].pop('children')
    elif data:
        stack[-1][-1]['data'] = data


print(json.dumps(tree, indent=4))

Already imported modules will not be loaded again. import foo just does nothing. However, it proved to be useful to reimport modules while working in an interactive environment. The proper way to do this in Python 3.4+ is to use importlib:

In [1]: import importlib
In [2]: with open('foo.py', 'w') as f:
   ...:     f.write('a = 1')
   ...:

In [3]: import foo
In [4]: foo.a
Out[4]: 1
In [5]: with open('foo.py', 'w') as f:
   ...:     f.write('a = 2')
   ...:
In [6]: foo.a
Out[6]: 1
In [7]: import foo
In [8]: foo.a
Out[8]: 1
In [9]: importlib.reload(foo)
Out[9]: <module 'foo' from '/home/v.pushtaev/foo.py'>
In [10]: foo.a
Out[10]: 2

ipython also has the autoreload extension that automatically reimports modules if necessary:

In [1]: %load_ext autoreload
In [2]: %autoreload 2
In [3]: with open('foo.py', 'w') as f:
   ...:     f.write('print("LOADED"); a=1')
   ...:
In [4]: import foo
LOADED
In [5]: foo.a
Out[5]: 1
In [6]: with open('foo.py', 'w') as f:
   ...:     f.write('print("LOADED"); a=2')
   ...:
In [7]: import foo
LOADED
In [8]: foo.a
Out[8]: 2
In [9]: with open('foo.py', 'w') as f:
   ...:     f.write('print("LOADED"); a=3')
   ...:
In [10]: foo.a
LOADED
Out[10]: 3

Lambdas in Python can't do a lot of things that ordinary functions can. You can only have one expression as a lambda body, you can't use statements (a = b, yield, await etc.), lambdas are not allowed to have type hints or be declared async. However, if you really need to turn lambda into an asynchronous function, you can use the asyncio.coroutine decorator. It was useful until Python 3.4 before async keyword was introduced, but has no much use in the modern Python.

In : f = asyncio.coroutine(lambda x: x ** 2)
In : asyncio.get_event_loop().run_until_complete(f(12))
Out: 144

Of course, that doesn't allow you to use await inside the lambda.

In Python, an object is physically destroyed and unloaded from the memory when nobody has a reference to it anymore. That is also true for any number of objects that have cross-references but are not available for the rest of the objects (so-called reference cycles). There might be a case when you want to have a reference to an object but don't want to prevent its destruction if your reference is the last one. The reference you want to have in this case is called weak. Weak references are extremely helpful for old kind of caches or indexes. The weakref module allows you to create weak references explicitly or you dictionaries with them inside. Unfortunately, not all types don't support weak referencing; sometimes you have to create trivial subclasses:

class List(list):
    pass

weakref.ref creates an object that you must call to get the original value:

>>> x = List()
>>> r = weakref.ref(x)
>>> r()
[]
>>> del x
>>> r
<weakref at 0x7f302db036d8; dead>
>>> r()
>>>

weakref.proxy creates an object that acts almost like a standard reference:

>>> x = List()
>>> p = weakref.proxy(x)
>>> p
<weakproxy at 0x7f302db03688 to List at 0x7f302db87ea8>
>>> list(p)
[]
>>> p.append(42)
>>> p[0]
42
>>> del x
>>> p
<weakproxy at 0x7f302db03688 to NoneType at 0x8a1a80>
>>> p[0]
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ReferenceError: weakly-referenced object no longer exists

The attributes of classes are stored in dictionaries, and that could be a problem since they don't preserve order in Python 3.5 and older:

$ cat test.py
class M:
    def __new__(meta, cls, bases, ns):
        print(ns)

class A(metaclass=M):
    a = 1
    b = 2
$ python3.4 test.py
{'__module__': '__main__', 'b': 2, '__qualname__': 'A', 'a': 1}
$ python3.4 test.py
{'a': 1, 'b': 2, '__module__': '__main__', '__qualname__': 'A'}
$ python3.4 test.py
{'__module__': '__main__', 'b': 2, '__qualname__': 'A', 'a': 1}
$ python3.4 test.py
{'__qualname__': 'A', 'a': 1, '__module__': '__main__', 'b': 2}
$ python3.4 test.py
{'b': 2, 'a': 1, '__module__': '__main__', '__qualname__': 'A'}
$ python3.4 test.py
{'b': 2, '__qualname__': 'A', '__module__': '__main__', 'a': 1}

However, you can replace the attribute container by using the __prepare__ metaclass method:

$ cat test.py
from collections import OrderedDict


class M:
    def __new__(meta, cls, bases, ns):
        print(ns)

    @classmethod
    def __prepare__(metacls, cls, bases):
        return OrderedDict()

class A(metaclass=M):
    a = 1
    b = 2
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])

Again, there no need to do such thing in Python 3.6+, since dictionaries now preserve order:

$ cat test.py
class M:
    def __new__(meta, cls, bases, ns):
        print(ns)

class A(metaclass=M):
    a = 1
    b = 2
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}

To immediately stop a Python program one should use sys.exit(). Another solution is the exit() function; however, its purpose is to work in the interactive mode. Thanks to its string representation, it could help users who try to end the session using exit (which is supported by many shells):

>>> exit
Use exit() or Ctrl-D (i.e. EOF) to exit
>>> str(exit)
'Use exit() or Ctrl-D (i.e. EOF) to exit'

Both exit() and sys.exit() don't really end the program, they merely raise the SystemExit exception. SystemExit is a direct subclass of BaseException which means it can't be caught by except Exception but can be by except BaseException or bare except:.

>>> try:
...     exit()
... except:
...     'Nothing'
...
'Nothing'
>>>

As long as this is a problem, you can use the os._exit function. It doesn't raise any exception; it just terminates the current process. That means, however, that none of your finally blocks are executed as well as exit routines of context managers.

$ python3
Python 3.4.3 (default, Apr 28 2015, 13:37:07)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> try:
...     os._exit(42)
... finally:
...     print('Bye!')
...
$ ...

Sometimes software starts to behave weirdly in the production. Instead of simply restarting it, you probably wish to understand what exactly is happening so you can fix it later. The obvious way to do it is to analyze what a program does and try to guess which piece of code is being executed. Surely proper logging makes that task easier, but your application's logs may be not verbose enough, either by design or because the high level of logging is set in the configuration. In that case, strace may be quite beneficial. It's a Unix utility which traces system calls for you. You can run it in advance — strace python script.py — but usually connecting to the already executing application is more suitable: strace -p PID.

$ cat test.py
with open('/tmp/test', 'w') as f:
    f.write('test')
$ strace python test.py 2>&1 | grep open | tail -n 1
open("/tmp/test", O_WRONLY|O_CREAT|O_TRUNC|O_CLOEXEC, 0666) = 3

Each line in the trace contains the system call name, followed by its arguments in parentheses and its return value. Since some arguments are output parameters (they are used for returning a result from the system call, not for passing data into it), line outputting may be interrupted until system call is finished. In this example, the output is interrupted until someone writes to STDIN:

$ strace python -c 'input()'
...
read(0,

However, in Python 2 Ellipsis can't be written as .... The only exception is a[...] that means a[Ellpsis]. All of the following syntaxes are valid for Python 3, but only the first line is valid for Python 2:

a[...]
a[...:2:...]
[..., ...]
{...:...}
a = ...
... is ...
def a(x=...): ...

Python has a very short list of built-in constants. One of them is Ellipsis which is also can be written as .... This constant has no special meaning for the interpreter but is used in places where such syntax looks appropriate. numpy support Ellipsis as a __getitem__ argument, e. g. x[...] returns all elements of x. PEP 484 defines additional meaning: Callable[..., type] is a way to define a type of callables with no argument types specified. Finally, you can use ... to indicate that function is not yet implemented. This is a completely valid Python code:

def x():
    ...

If you want to iterate over several iterables at once, the zip function may be a good choice. It returns a generator that yields tuples containing one element from every original iterables:

In : eng = ['one', 'two', 'three']
In : ger = ['eins', 'zwei', 'drei']
In : for e, g in zip(eng, ger):
    ...:     print('{e} = {g}'.format(e=e, g=g))
    ...:
one = eins
two = zwei
three = drei

Notice, that zip accepts iterables as separate arguments, not a list of arguments. To unzip values, you can use the * operator:

In : list(zip(*zip(eng, ger)))
Out: [('one', 'two', 'three'), ('eins', 'zwei', 'drei')]

The functools.reduce function is a powerful tool, but it can't return intermediate results and therefore can't be used with infinite generators. Since Python 3.3 you can use itertools.accumulate to do such things:

>>> a = accumulate(sys.stdin, lambda a, b: int(a) * int(b))
>>> next(a)
1  # this line is input
'1\n'
>>> next(a)
2  # this line is input
2
>>> next(a)
10  # this line is input
20

sys.stdout is a wrapper that allows you to write strings instead of raw bytes. The string is encoded automatically using sys.stdout.encoding:

>>> _ = sys.stdout.write('Straße\n')
Straße
>>> sys.stdout.encoding
'UTF-8'

sys.stdout.encoding is read-only and is equal to Python default encoding, which can be changed by setting the PYTHONIOENCODING environment variable:

$ PYTHONIOENCODING=cp1251 python3
Python 3.6.6 (default, Aug 13 2018, 18:24:23)
[GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.stdout.encoding
'cp1251'

If you want to write bytes to stdout you can bypass automatic encoding by accessing the wrapped buffer with sys.stdout.buffer:

>>> sys.stdout
<_io.TextIOWrapper name='<stdout>' mode='w' encoding='cp1251'>
>>> sys.stdout.buffer
<_io.BufferedWriter name='<stdout>'>
>>> _ = sys.stdout.buffer.write(b'Stra\xc3\x9fe\n')
Straße

sys.stdout.buffer is also a wrapper that does buffering for you. It can be bypassed by accessing the raw file handler with sys.stdout.buffer.raw:

>>> _ = sys.stdout.buffer.raw.write(b'Stra\xc3\x9fe')
Straße

Sometimes you want to compare complex structures in tests ignoring some values. Usually, it can be done by comparing particular values with the structure:

>>> d = dict(a=1, b=2, c=3)
>>> assert d['a'] == 1
>>> assert d['c'] == 3

However, you can create special value that reports being equal to any other value:

>>> assert d == dict(a=1, b=ANY, c=3)

That can be easily done by defining the __eq__ method:

>>> class AnyClass:
...     def __eq__(self, another):
...         return True
... 
>>> ANY = AnyClass()

Welcome to PyCon Belarus 2019 — the two-days conference about Python development and data science in Python ecosystem. Save the dates! 🗓 February 15 (Event SPACE). Junior Day Talks and workshops for beginners about ML, DeepPavlov, Dialogue Systems, NLP, ChatBots, Dependencies Management. 🗓 February 16 (IBB Hotel). Advanced Day Talks by Python experts for skilled developers and data scientists with 2 tracks: 🔴 ML/DS: Jupyter Notebooks, Luigi, GeoPython, Data Vizualization 🔴 Python development: Deployment-Friendly Apps, Application Security, Testing & Legacy, GraphQL, Poetry, Flit, Pipenv. Stay tuned for updates ➡ https://by.pycon.org

Hello everyone. PyCon Belarus offers one of you a free ticket to the conference this year. (More info about the event below.) If you are interested, write a post for @pythonetc until 2019-02-03 23:59:00 CET and send it to me (@pushtaev). As long as it's good enough it will be published with your name specified. The author of the best post will win the ticket.

The MagicMock object allows you to get any attribute from it or call any method. New mock will be returned upon such access. What is more, you get the same mock object if access the same attribute (or call the same method):

>>> from unittest.mock import MagicMock
>>> m = MagicMock()
>>> a = m.a
>>> b = m.b
>>> a is m.a
True
>>> m.x() is m.x()
True
>>> m.x()
<MagicMock name='mock.x()' id='139769776427752'>

This obviously will work with sequential attribute access of any deep. Method arguments are ignored though:

>>> m.a.b.c.d
<MagicMock name='mock.a.b.c.d' id='139769776473480'>
>>> m.a.b.c.d
<MagicMock name='mock.a.b.c.d' id='139769776473480'>
>>> m.x().y().z()
<MagicMock name='mock.x().y().z()' id='139769776450024'>
>>> m.x(1).y(1).z(1)
<MagicMock name='mock.x().y().z()' id='139769776450024'>

Once you set a value for any attribute, it doesn't return mock anymore:

>>> m.a.b.c.d = 42
>>> m.a.b.c.d
42
>>> m.x.return_value.y.return_value = 13
>>> m.x().y()
13

However, it doesn't work with m[1][2]. The reason is, the item access is not treated specially by MagicMock, it's merely a method call:

>>> m[1][2] = 3
>>> m[1][2]
<MagicMock name='mock.__getitem__().__getitem__()' id='139769776049848'>
>>> m.__getitem__.return_value.__getitem__.return_value = 50
>>> m[1][2]
50

pip --freeze allows you to get the full list of installed packages with fixed versions. That could be helpful if once want to recreate the current environment. Create and freeze the environment:

vadim:~$ virtualenv ~/.ve/pythonetc1
Using base prefix '/usr/local'
New python executable in /home/vadim/.ve/pythonetc1/bin/python3.6
Also creating executable in /home/vadim/.ve/pythonetc1/bin/python
Installing setuptools, pip, wheel...done.
vadim:~$ source ~/.ve/pythonetc1/bin/activate
(pythonetc1) vadim:~$ pip install flask
Collecting flask
  Downloading
...
Successfully installed Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 click-7.0 flask-1.0.2 itsdangerous-1.1.0
(pythonetc1) vadim:~$ pip freeze
Click==7.0
Flask==1.0.2
itsdangerous==1.1.0
Jinja2==2.10
MarkupSafe==1.1.0
Werkzeug==0.14.1
(pythonetc1) vadim:~$ pip freeze > /tmp/freezed

Recreate it:

vadim:~$ virtualenv ~/.ve/pythonetc2
Using base prefix '/usr/local'
New python executable in /home/vadim/.ve/pythonetc2/bin/python3.6
Also creating executable in /home/vadim/.ve/pythonetc2/bin/python
Installing setuptools, pip, wheel...done.
$ source ~/.ve/pythonetc2/bin/activate
(pythonetc2) vadim:~$ pip install --no-deps -r /tmp/freezed
...
Installing collected packages: Click, itsdangerous, MarkupSafe, Jinja2, Werkzeug, Flask
Successfully installed Click-7.0 Flask-1.0.2 Jinja2-2.10 MarkupSafe-1.1.0 Werkzeug-0.14.1 itsdangerous-1.1.0

However, you should be aware that pip doesn't freeze pip, setuptools, distribute, wheel packages unless --all is specified. The point is they are usually useful only for development environments and are not needed in the actaul production installation. Also, pip allows you to install packages that are actually part of standard library, e. g. argparse. The reason is argparse was indeed third-party package until Python 3.2 and 2.7. Such modules are never printed by pip freeze though however could be listed as a requirement by some packages.