Python etc

Usually, you communicate with a generator by asking for data with next(gen). You also can send some values back with g.send(x) in Python 3. But the technique you probably don't use every day, or maybe even isn't aware of, is throwing exceptions inside a generator. With gen.throw(e) you may raise an exception at the point where the gen generator is paused, i. e. at the point of some yield. If gen catches the exception, get.throw(e) returns the next value yielded (or StopIteration is raised). If gen doesn't catch the exception, it propagates back to you.

In : def gen():
...:     try:
...:         yield 1
...:     except ValueError:
...:         yield 2
...: 
...: g = gen()
...: 

In : next(g)
Out: 1

In : g.throw(ValueError)
Out: 2

In : g.throw(RuntimeError('TEST'))
...
RuntimeError: TEST

You can use it to control generator behavior more precisely, not only be sending data to it but by notifying about some problems with values yielded for example. But this is rarely required, and you have a little chance to encounter g.throw in the wild. However, @contextmanager decorator from contextlib does exactly this to let the code inside the context catch exceptions.

In : from contextlib import contextmanager
...: 
...: @contextmanager
...: def atomic():
...:     print('BEGIN')
...: 
...:     try:
...:         yield
...:     except Exception:
...:         print('ROLLBACK')
...:     else:
...:         print('COMMIT')
...: 

In : with atomic():
...:     print('ERROR')
...:     raise RuntimeError()
...: 
BEGIN
ERROR
ROLLBACK

Since BNF is a context-free language itself, you can represent its syntax with a BNF :).

<syntax>         ::= <rule> | <rule> <syntax>
<rule>           ::= <opt-whitespace> "<" <rule-name>
                     ">" <opt-whitespace>
                     "::=" <opt-whitespace> <expression>
                     <line-end>
<opt-whitespace> ::= " " <opt-whitespace> | ""
<expression>     ::= <list> | <list> <opt-whitespace>
                     "|" <opt-whitespace> <expression>
<line-end>       ::= <opt-whitespace> <EOL> |
                     <line-end> <line-end>
<list>           ::= <term> |
                     <term> <opt-whitespace> <list>
<term>           ::= <literal> | "<" <rule-name> ">"
<literal>        ::= '"' <text1> '"' | "'" <text2> "'"
<text1>          ::= "" | <character1> <text1>
<text2>          ::= "" | <character2> <text2>
<character>      ::= <letter> | <digit> | <symbol>
<letter>         ::= "A" | "B" | "C" | "D" | "E" | "F" |
                     "G" | "H" | "I" | "J" | "K" | "L" |
                     "M" | "N" | "O" | "P" | "Q" | "R" |
                     "S" | "T" | "U" | "V" | "W" | "X" |
                     "Y" | "Z" | "a" | "b" | "c" | "d" |
                     "e" | "f" | "g" | "h" | "i" | "j" |
                     "k" | "l" | "m" | "n" | "o" | "p" |
                     "q" | "r" | "s" | "t" | "u" | "v" |
                     "w" | "x" | "y" | "z"
<digit>          ::= "0" | "1" | "2" | "3" | "4" | "5" |
                     "6" | "7" | "8" | "9"
symbol>         ::=  "|" | " " | "!" | "#" | "$" | "%" |
                     "&" | "(" | ")" | "*" | "+" | "," |
                     "-" | "." | "/" | ":" | ";" | ">" |
                     "=" | "<" | "?" | "@" | "[" | "\" |
                     "]" | "^" | "_" | "`" | "{" | "}" |
                     "~"
<character1>     ::= <character> | "'"
<character2>     ::= <character> | '"'
<rule-name>      ::= <letter> | <rule-name> <rule-char>
<rule-char>      ::= <letter> | <digit> | "-"

Apart from regular languages, Chomsky distinguishes three more types (ordered by descending strictness): context-free, context-sensitive, and unrestricted. Context-free languages are more powerful than regular ones but still can be efficiently parsed by a program. XML, JSON and SQL are context-free for example. Many tools allow you to parse such languages easily. Usually, they require you to define some grammar, the rules on how to parse and create a parser automatically. The most popular way to define such grammar is the BNF language. Here is the grammar to parse simple arithmetical expressions (only + supported) defined in BNF:

 <expr> ::= <operand> "+" <expr>  |  <operand>
 <operand> ::= "(" <expr> ")"  |  <const>
 <const> ::= integer

This is the set of rules. An expression is an operand plus another expression or just operand. An operand is either a constant or an expression enclosed in brackets. This way we can see the recursive nature of this language, which makes it non-regular. The example of a context-free grammar parser for Python is lark. It is what you want if regexes are not enough or code that does the parsing gets messy.

The machine starts at (1), possibly matches minus sign, then processes as many digits as required. After that, it may match a dot (3->4) which must be followed by one digit (4->5), but maybe more. The classic example of a non-regular language is a family of strings like:

a-b
aaa-bbb
aaaaa-bbbbb

Formally, we need a line that consists of N occurrences of a, then -, then N occurrences of b. N is any integer greater than zero. You can't do it with a finite machine, because you have to remember the number of a chars you encountered which leads you to the infinite number of states. Regular expressions can match only regular languages. Remember to check whether the line you are trying to process can be handled by FSM at all. JSON, XML or even simple arithmetic expression with nested brackets cannot be. Mind, however, that a lot of modern regular expression engines are not regular. For example, Python regex module supports recursion (which will help with that aaa-bbb problem).

A regular language is a formal language that can be recognized by a finite-state machine (FSM). That means that reading text, character by character, you only need memory to remember current state, and the number of such states is finite. The beautiful and simple example is a machine that checks whether an input is a simple number like -3, 2.2 or 001. The following diagram is an FSM diagram. Double circles mean accept states, they identify where the machine can stop.

In Python, you can override square brackets operator ([]) by defining __getitem__ magic method. The example is Cycle object that virtually contains an infinite number of repeated elements:

class Cycle:
    def __init__(self, lst):
        self._lst = lst

    def __getitem__(self, index):
        return self._lst[index % len(self._lst)]

print(Cycle(['a', 'b', 'c'])[100])  # prints 'b'

The unusual thing here is [] operator supports a unique syntax. It can be used not only like this — [2], but also like this — [2:10], or [2:10:2], or [2::2], or even [:]. The semantic is [start:stop:step] but you can use it any way you want for your custom objects. But what __getitem__ gets as an index parameter if you call it using that syntax? The slice objects exist precisely for this case.

In : class Inspector:
...:     def __getitem__(self, index):
...:         print(index)
...:
In : Inspector()[1]
1
In : Inspector()[1:2]
slice(1, 2, None)
In : Inspector()[1:2:3]
slice(1, 2, 3)
In : Inspector()[:]
slice(None, None, None)

You can even combine tuple and slice syntaxes:

In : Inspector()[:, 0, :]
(slice(None, None, None), 0, slice(None, None, None))

slice is not doing anything for you except simply storing start, stop and step attributes.

In : s = slice(1, 2, 3)
In : s.start
Out: 1
In : s.stop
Out: 2
In : s.step
Out: 3

Generators are one of the most influential Python mechanics. They have many uses, and one of them is to create context managers easily. Usually, you have to manually define __enter__ and __exit__ magic methods, but @contextmanager decorator from contextlib makes it far more convenient:

from contextlib import contextmanager

@contextmanager
def atomic():
    print('BEGIN')

    try:
        yield
    except Exception:
        print('ROLLBACK')
    else:
        print('COMMIT')

Now atomic is a context manager that can be used like this:

In : with atomic():
...:     print('ERROR')
...:     raise RuntimeError()
...:
BEGIN
ERROR
ROLLBACK

Additionally, the @contextmanager magic allows to use it as a decorator as well as a context manager:

In : @atomic()
...: def ok():
...:     print('OK')
...:
In : ok()
...:
BEGIN
OK
COMMIT

Python provides the powerful library to work with date and time: datetime. The interesting part is datetime objects have the special interface for timezone support (namely tzinfo attribute), but this module only has limited support of its interface, leaving the rest of the job to different modules. The most popular module for this job is pytz. The tricky part pytz don't fully satisfy tzinfo interface. The pytz documentation states this at one the first lines: “This library differs from the documented Python API for tzinfo implementations.” You can't use pytz timezone objects as a tzinfo attribute. If you try, you may get the absolute insane results:

In : paris = pytz.timezone('Europe/Paris')
In : str(datetime(2017, 1, 1, tzinfo=paris))
Out: '2017-01-01 00:00:00+00:09'

Look at this +00:09 offset. The proper use of pytz is following:

In : str(paris.localize(datetime(2017, 1, 1)))
Out: '2017-01-01 00:00:00+01:00'

Also, after any arithmetic operations, you should normalize your datetime object in case of offset changes (on the edge of DST period for instance).

In : new_time = time + timedelta(days=2)
In : str(new_time)
Out: '2018-03-27 00:00:00+01:00'
In : str(paris.normalize(new_time))
Out: '2018-03-27 01:00:00+02:00'

Since Python 3.6 it's recommended to use dateutil.tz instead of pytz. It's fully compatible with tzinfo, can be passed as an attribute, doesn't require normalize, though works a bit slower. If you are interested why pytz doesn't support datetime API, or you wish to see more examples, consider reading the decent article on the topic.

Let’s delve into a pretty typical situation. You have some subclasses, and you have to pick one according to some parameter provided by a user. Here is my article on how to do it in Python.

Objects in Python store their attributes in dictionaries that can be accessed by __dict__ magic attribute:

In [1]: class A: pass
In [2]: a = A()
In [3]: a.x = 1
In [4]: a.__dict__
Out[4]: {'x': 1}

By direct accessing it you can even create attributes that are not Python identifiers (which means you can't get them with a standard obj.attr syntax):

In [6]: a.__dict__[' '] = ' '
In [7]: getattr(a, ' ')
Out[7]: ' '

You can also ask Python to store attributes directly in memory (like a simple C struct) using __slots__. It will save some memory and some CPU cycles that are used for dictionary lookups.

class Point:
    __slots__ = ['x', 'y']

There are some things you should remember while using slots. First, you can't set any attributes that are not specified in __slots__ (unless you add __dict__ there as well). Second, if you inherit from a class with slots, your own __slots__ don't override parental __slots__ but are added to it:

class Parent: __slots__ = ['x']
class Child(Parent): __slots__ = ['y']
c = Child()
c.x = 1
c.y = 2

Third, you can't inherit from two different classes with nonempty __slots__, even if they are identical. You can get more information from this excellent Stack Overflow answer. Remeber, that __slots__ is meant for optimization, not for constraining attributes.

I bet you often ask yourself: “How do I make a script that can be run not only by Python interpreter but by Perl and Ruby as well?” Calm yourself down; I've got a solution for you:

"@{[sub {while (<DATA>) {last if /^\"\"\"__PERL__/}; eval join '', <DATA>}->()]}"
__DATA__ = 0
"""#{

    # Place Ruby code here
    if (2 > 1)
        puts "Hi, I'm Ruby!"
    end

}""";
__END__ = 0
__END__

    # Place Python code here
    if 2 > 1:
       print("Hi, I'm Python!")

"""__PERL__

    # Place perl code here
    use feature 'say';
    if (2 > 1) {
        say "Hi, I'm Perl!";
    }

__END__
"""

Here is how it works:

$ ruby script && python script && perl script
Hi, I'm Ruby!
Hi, I'm Python!
Hi, I'm Perl!

Converting datetime object to the number of seconds since the start of the epoch is not a simple task until Python 3.3. The most natural solution for the problem is to use strftime method that can format the datetime. Using %s as a format you can get a timestamp. Look a the example:

naive_time = datetime(2018, 3, 31, 12, 0, 0)
utc_time = pytz.utc.localize(naive_time)
ny_time = utc_time.astimezone(
    pytz.timezone('US/Eastern'))

ny_time is the exact the same moment as utc_time, but written as New Yorkers see it:

# utc_time
datetime.datetime(2018, 3, 31, 12, 0,
    tzinfo=<UTC>)
# ny_time
datetime.datetime(2018, 3, 31, 8, 0,
    tzinfo=<DstTzInfo 'US/Eastern' ...>)

Since they are the same moments, their timestamps should be equal:

In : int(utc_time.strftime('%s')),
     int(ny_time.strftime('%s'))
Out: (1522486800, 1522468800)

Wait, what? They are not the same at all. In fact, you can't use strftime as a solution for this problem. Python's strftime doesn't even support %s as an argument, it merely works because internally the platform C library’s strftime() is called. But, as you can see, the timezone of datetime object is wholly ignored. The proper result can be achieved with straightforward subtraction:

In : epoch_start = pytz.utc.localize(
      datetime(1970, 1, 1))

In : (utc_time - epoch_start).total_seconds()
Out: 1522497600.0

In : (utc_time - epoch_start).total_seconds()
Out: 1522497600.0

Again, if you use Python 3.3+, you can solve the problem with timestamp() method of datetime: utc_time.timestamp().

Sometimes you need to create JSON from a big pile of data you can stream from some source, file or socket for instance. Sadly you can't encode generator as is using the Python json library:

In [1]: json.dumps(range(10))
...
TypeError: Object of type 'range' is not JSON serializable

The simple solution here is to derive from list and override __iter__ method:

In [1]: class LazyList(list):
   ...:     def __init__(self, gen):
   ...:         self.__gen = gen
   ...:
   ...:     def __iter__(self):
   ...:         return iter(self.__gen)
   ...:
   ...:
In [2]: json.dumps(LazyList(range(10)))
Out[2]: '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]'

Mind that the solution is not problem-free. It might not work correctly for indent parameter, and also some versions of json require you you to override __len__ as well. The solution described is a more or less hack, the clear one is to use simplejson instead. It explicitly supports iterable_as_array flag:

In [1]: import simplejson as json
In [2]: json.dumps(range(10), iterable_as_array=True)
Out[3]: '[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]'

You also may put all data of the generator into a list and encode it afterward but will take some time and additional memory.

Python data model contains two methods with similar names: __getattr__ and __getattribute__. The key difference is that __getattribute__ is called unconditionally on an every attribute access, but __getattr__ is only called when __getattribute__ fails to find an attribute (raises AttributeError). The default __getattribute__ behavior (the one you use every day) is to return the attribute if it exists or to raise AttributeError. There are no default for __getattr__. That actually means that by default __getattr__ is a way to handle attributes that couldn't be found by ordinary means. That can be helpful for some sorts of metaprogramming or creating DSLs. With this code you can get hex version of any attribute by prepending hex_ to its name (that's not really helpful but good enough as an example):

class Point:
    def __init__(self, x, y):
        self._x = x
        self._y = y

    @property
    def x(self):
        return self._x

    @property
    def y(self):
        return self._y

    def __getattr__(self, attr):
        prefix, orig_attr = attr.split('_', 2)
        if prefix == 'hex' and hasattr(self, orig_attr):
            return hex(getattr(self, orig_attr))
        else:
            raise AttributeError

p = Point(16, 20)
print(p.hex_x, p.hex_y)

There are also __setattr__ and __delattr__ methods for setting and deleting attributes, and they both are called unconditionally. That means that the __getattr__ method stands out here, not __getattrbiute__ like it may seem.

heapq also contains general purpose functions: merge, nlargest and nsmallest. Guido van Rossum used the first one in his 10-year-old article, which one of the readers sent me today. Feel free to do the same — @pushtaev.

A priority queue is a data structure that supports two operations: add element and extract the minimum of all elements among previously added. One of the most common implementations of a priority queue is a binary heap. It's a complete binary tree with the following property: the key stored in each node is equal to or less than (≤) the keys in the node's children. The minimum of all elements is a root of such tree.

              1

      3               7

  5       4       9       8

15 16   17 18   19

In a binary heap, both inserting and extraction operations' complexity is O(log n). The common way of storing a complete binary tree in memory is an array, where children of x[i] are x[2*i+1] and x[2*i+2]:

[1, 3, 7, 5, 4, 9, 8, 15, 16, 17, 18, 19]

Python doesn't provide a binary heap as a class, but it does provide a number of functions that treat list like a binary heap. They are placed in the heapq module.

In [1]: from heapq import *
In [2]: heap = [3,2,1]
In [3]: heapify(heap)
In [4]: heap
Out[4]: [1, 2, 3]
In [5]: heappush(heap, 0)
In [6]: heap
Out[6]: [0, 1, 3, 2]
In [7]: heappop(heap)
Out[7]: 0
In [8]: heap
Out[8]: [1, 2, 3]

Remember that NotImplemented is not the same that NotImplementedError. It's not even an exception. It's a special value (like True and False) that has an absolutely diffrent meaning. It should be returned by the binary special methods (e.g. __eq__(), __add__() etc.) so Python tries to reflect operation. If a.__add__(b) returns NotImplemented, Python tries to call b.__radd__.

The popular method to declare an abstract method in Python is to use NotImplentedError exception:

def human_name(self):
    raise NotImplementedError

Though it's pretty popular and even has IDE support (Pycharm consider such method to be abstract) this approach has a downside. You get the error only upon method call, not upon class instantiation. Use abc to avoid this problem:

from abc import ABCMeta, abstractmethod
class Service(metaclass=ABCMeta):
    @abstractmethod
    def human_name(self):
        pass

py.test is a simple yet powerful tool that allows you to run tests. It may be useful not only for big projects but even for one-off scripts. Let's say you write a small utility to parse some log, and you have a function to detect GET requests.

def is_get(line):
    return re.search(r'\bGET\b', line)

To test it you may put some debug statements in your script, copy the function to another file or Python shell and run it manually, or create a stand-alone test script. Or you may just define test_is_get along is_get that doesn't interfere with your script unless it is executed with py.test.

def test_is_get():
    assert is_get('12:00 GET url')
    assert is_get('00:00 url GET params')
    assert not is_get('07:00 GETTER restart')

Once the script is started with py.test log_parser.py, all test_* will be executed. This way you can actually have two modes to run your script: with python to parser log or with py.test to test some things you want to be tested.