Python etc

The difference between function definition and generator definition is the presence of the yield keyword in the function body:

In : def f():
...:     pass
...:

In : def g():
...:     yield
...:

In : type(f())
Out: NoneType

In : type(g())
Out: generator

That means that in order to create an empty generator you have to do something like this:

In : def g():
...:     if False:
...:         yield
...:

In : list(g())
Out: []

However, since yield from supports simple iterators that better looking version would be this:

def g():
    yield from []

Good day everyone. I'm happy to announce that the project I'm working for is now open for beta testing. I'm talking about Marusia, the Russian speaking intelligent personal assistant for Android and iOS. You can get it on Google Play Store or the App Store. It currently requires invite codes. I have a bunch of them available, PM me to get one.

Every method can be treated as a plain function and called with a custom self:

In : class A:
   ...:     def foo(self):
   ...:         return self
   ...:

In : A().foo
Out: <bound method A.foo of <...>>

In : A.foo
Out: <function __main__.A.foo>

In : A.foo(A())
Out: <__main__.A at 0x7f55ddd32898>

You can even convert a function back to the bound method. Any function is a descriptor, so it can be abused by calling __get__:

In [8]: b = A()

In [9]: A.foo.__get__(b, A)
Out[9]: <bound method A.foo of <...>>

In Python, range() defines all integers in a half-open interval. So range(2, 10) means, speaking mathematically, [2, 10). Or, speaking Python, [2, 3, 4, 5, 6, 7, 8, 9]. Despite asymmetry, that is not a mistake nor an accident. It makes perfect sense since it allows you to glue together two adjacent intervals without risk of one-off errors:

[a, c) = [a, b) + [b, c)

Compare to closed intervals that feel more “natural”:

[a, c] = [a, b] + [b+1, c]

This is also a reason for indexing to start from zero: [0, N) has exactly N elements. Dijkstra wrote an excellent article on the subject back in 1982.

Since Python 3.5, it's actually possible to use unpacking with dictionary and list literals.

In : {**{'a': 1}, 'b': 2, **{'c': 3}}
Out: {'a': 1, 'b': 2, 'c': 3}

In : [1, 2, *[3, 4]]
Out: [1, 2, 3, 4]

For dictionaries, this form is even more powerful than the dict function, since it allows values to be overridden:

In : {**{'a': 1, 'b': 1}, 'a': 2, **{'b': 3}}
Out: {'a': 2, 'b': 3}

Feel like you versed in Data Science but can't organize your knowledges properly and don't have enough practice? Online-education center SkillFactory launches Data Science course https://clc.to/I9sEKQ This course will allow you not only to increase your existing skills but learn somethig new. All our teachers are industry professionals, who worked for Yandex and NVIDIA. They will share with you industry insides and intricacies of the job, which are not written in any books. Our course program includes a Python learning block (including Pandas for Data Analysis), mathematics, statistics probability theory with solving problems in NumPy and landing on Data Science, introduction to Machine Learning, course of Data Engineering, Neural Networks and AI and management for a data scientist: the skill of implementing data analysis systems, machine learning and neural networks. Want to become a one of a kind specialist and take part in a variety of competitions on Kaggle (with solutions analyses and various models of Machine Learning and Neural Networks training) - our course starts 11.06, click to register now https://clc.to/I9sEKQ

Ads time!

The sorted function allows you to provide custom method for sorting. It’s done with the key argument, which describes how to convert original values to values that are actually compared:

>>> x = [dict(name='Vadim', age=29), dict(name='Alex', age=4)]
>>> sorted(x, key=lambda v: v['age'])
[{'age': 4, 'name': 'Alex'}, {'age': 29, 'name': 'Vadim'}]

Alas, not all libraries that work with comparison support something like this key argument. Notable examples are heapq (partial support) and bisect (no support). There are two ways to deal with the situation. The first is to use custom objects that do support proper comparsion:

>>> class User:
...     def __init__(self, name, age):
...         self.name = name
...         self.age = age
...     def __lt__(self, other):
...         return self.age < other.age
...
>>> x = [User('Vadim', 29), User('Alex', 4)]
>>> [x.name for x in sorted(x)]
['Alex', 'Vadim']

However, you may have to create several versions of such classes since there are more than one way to compare objects. It can be tiresome, but can be easily solved by the second way. Instead of creating custom objects you may use tuples (a, b) where a is the value to compare (a.k.a. prioirty) and b is the original value:

>>> users = [dict(name='Vadim', age=29), dict(name='Alex', age=4)]
>>> to_sort = [(u['age'], u) for u in users]
>>> [x[1]['name'] for x in sorted(to_sort)]
['Alex', 'Vadim']

List comprehensions may contain more than one for and if clauses:

In : [(x, y) for x in range(3) for y in range(3)]
Out: [
    (0, 0), (0, 1), (0, 2),
    (1, 0), (1, 1), (1, 2),
    (2, 0), (2, 1), (2, 2)
]

In : [
    (x, y)
    for x in range(3)
    for y in range(3)
    if x != 0
    if y != 0
]
Out: [(1, 1), (1, 2), (2, 1), (2, 2)]

Also, any expression within for and if may use all the variables that are defined before:

In : [
    (x, y)
    for x in range(3)
    for y in range(x + 2)
    if x != y
]
Out: [
    (0, 1),
    (1, 0), (1, 2),
    (2, 0), (2, 1), (2, 3)
]

You can mix ifs and fors however you want:

In : [
    (x, y)
    for x in range(5)
    if x % 2
    for y in range(x + 2)
    if x != y
]
Out: [
    (1, 0), (1, 2),
    (3, 0), (3, 1), (3, 2), (3, 4)
]

The \ symbol in regular string have special meaning. \t is tab character, \r is carriage return and so on. You can use raw-strings to disable this behaviour. r'\t' is just backslash and t. You obviously can’t use ' inside r'...'. However, it sill can be escaped by \, but \ is preserved in the string:

>>> print(r'It\'s insane!')
It\'s insane!

The company I'm working for regularly holds championships in machine learning and programming of artificial intelligence. These championships are a great opportunity to try yourself in solving interesting non-standard AI and ML problems. ML Boot Camp 9 (machine learning championship) and Mini AI Cup 4 (artificial intelligence championship) will start very soon. Championships announcements will be published in the official channel (Russian only). In the channel you can also find the analysis of previous tasks, read useful articles and the most important news.

Some code you are using may print data you are interested in to stdout instead of providing some API that is usable within a program (returning a string, for example). Instead of refactoring such code you may use the contextlib.redirect_stdout context manager that allows temporary redirecting stdout to any custom file-like object. In conjuncture with io.StringIO, it allows capturing output to a variable.

from contextlib import redirect_stdout
from io import StringIO

s = StringIO()
with redirect_stdout(s):
    print(42)

print(s.getvalue())

There is also contextlib.redirect_stderr available for redirecting sys.stderr.

The in operator can be used with generators: x in g. Python will iterate over g until x is found or g is exhausted.

>>> def g():
...     print(1)
...     yield 1
...     print(2)
...     yield 2
...     print(3)
...     yield 3
...
>>> 2 in g()
1
2
True

However, range() does more than this for you. It has the __contains__ magic method overriden that allows in to work with the O(1) complexity:

In [1]: %timeit 10**20 in range(10**30)
375 ns ± 10.7 ns per loop

Mind that it doesn't work for the Python 2 xrange() function.

In Python, you can override square brackets operator ([]) by defining the __getitem__ magic method. This is how you create an object that virtually contains an infinite number of repeated elements:

class Cycle:
    def __init__(self, lst):
        self._lst = lst

    def __getitem__(self, index):
        return self._lst[
            index % len(self._lst)
        ]

print(Cycle(['a', 'b', 'c'])[100])  # 'b'

The unusual thing here is that the [] operator supports a unique syntax. It can be used not only like this — [2], but also like this — [2:10], or [2:10:2], or [2::2], or even [:]. The default semantic is [start:stop:step], but you can use any for your custom objects. But what does __getitem__ get as an index parameter if you call it using that syntax? The slice objects exist precisely for that.

In : class Inspector:
...:     def __getitem__(self, index):
...:         print(index)
...:
In : Inspector()[1]
1
In : Inspector()[1:2]
slice(1, 2, None)
In : Inspector()[1:2:3]
slice(1, 2, 3)
In : Inspector()[:]
slice(None, None, None)

You can even combine tuple and slice syntaxes:

In : Inspector()[:, 0, :]
(slice(None, None, None), 0, slice(None, None, None))

slice is not doing anything for you except simply storing start, stop and step attributes.

In : s = slice(1, 2, 3)
In : s.start
Out: 1
In : s.stop
Out: 2
In : s.step
Out: 3

To store any information in memory or on a storage device, you should represent it in bytes. Python usually provides the level of abstraction where you can think about data itself, not its byte form. Still, when you write, say, a string to a file, you deal with a physical structure of data. To put characters into a file you should transform them into bytes; that is called encoding. When you get bytes from a file, you probably want to convert them into meaningful characters; that is call decoding. There are hundreds of encoding methods out there. The most popular one is probably Unicode, but you can't transform anything to bytes with it. In the sense of byte representation, Unicode is not even an encoding. Unicode defines a mapping between characters and their integer codes. 🐍 is 128 013, for example. But to put integers into a file, you need a real encoding. Unicode is usually used with utf-8, which is (usually) a default in Python. When you read from a file, Python automatically decodes utf-8. You can choose any other encoding with encoding= parameter of the open function, or you can read plane bytes by appending b to its mode.

All information published in this channel is licensed under Attribution-ShareAlike 4.0 International (CC BY-SA 4.0). That means that you have to mention the source of the content upon reposting (translations included). You are also requried to have the same CC BY-SA license for all products based on this content. Thanks for your cooperation and understanding.

There are six magic methods for Python objects that define comparison rules: ‣ __lt__ for < ‣ __gt__ for > ‣ __le__ for <= ‣ __ge__ for >= ‣ __eq__ for == ‣ __ne__ for != If some of these methods are not defined or return NotImplemented, the following rules applied: ‣ a.__lt__(b) is the same as b.__gt__(a) ‣ a.__le__(b) is the same as b.__ge__(a) ‣ a.__eq__(b) is the same as not a.__ne__(b) (mind that a and b are not swapped in this case) However, a >= b and a != b don’t automatically imply a > b. The functools.total_ordering decorator create all six methods based on __eq__ and one of the following: __lt__, __gt__, __le__, or __ge__.

from functools import total_ordering           
                                               
                                               
@total_ordering                                
class User:                                    
    def __init__(self, pk, name):              
        self.pk = pk                           
        self.name = name                       
                                               
    def __le__(self, other):                   
        return self.pk <= other.pk             
                                               
    def __eq__(self, other):                   
        return self.pk == other.pk             
                                               
                                               
assert User(2, 'Vadim') < User(13, 'Catherine')

Python provides the powerful library to work with date and time: datetime. The interesting part is, datetime objects have the special interface for timezone support (namely the tzinfo attribute), but this module only has limited support of its interface, leaving the rest of the job to different modules. The most popular module for this job is pytz. The tricky part is, pytz doesn't fully satisfy tzinfo interface. The pytz documentation states this at one of the first lines: “This library differs from the documented Python API for tzinfo implementations.” You can't use pytz timezone objects as the tzinfo attribute. If you try, you may get the absolute insane results:

In : paris = pytz.timezone('Europe/Paris')
In : str(datetime(2017, 1, 1, tzinfo=paris))
Out: '2017-01-01 00:00:00+00:09'

Look at that +00:09 offset. The proper use of pytz is following:

In : str(paris.localize(datetime(2017, 1, 1)))
Out: '2017-01-01 00:00:00+01:00'

Also, after any arithmetic operations, you should normalize your datetime object in case of offset changes (on the borderline of the DST period for instance).

In : new_time = time + timedelta(days=2)
In : str(new_time)
Out: '2018-03-27 00:00:00+01:00'
In : str(paris.normalize(new_time))
Out: '2018-03-27 01:00:00+02:00'

Since Python 3.6, it's recommended to use dateutil.tz instead of pytz. It's fully compatible with tzinfo, can be passed as an attribute, doesn't require normalize, though works a bit slower. If you are interested why pytz doesn't support datetime API, or you wish to see more examples, consider reading the decent article on the topic.

Decorators may be used not only to modify existing functions but also to create new ones without altering originals.

class A:
    def __init__(self, x):
        self._x = x

    def get_x_len(self):
        return len(self._x)

    x_len = property(get_x_len)


a = A([1, 2, 3])
print(a.get_x_len()) # 3
print(a.x_len) # 3