Python etc

typing allows you to define type for generators. You can additionally specify what type is yielded, what type can be sent into a generator and what is returned. Generator[int, None, bool] is a generator that yields integers, returns boolean value and doesn’t support g.send(). Here is slightly more complicated example. chain_while yields from other generators until one of them returns something that is a signal to stop according to the condition function:

from typing import Generator, Callable, Iterable, TypeVar

Y = TypeVar('Y')
S = TypeVar('S')
R = TypeVar('R')


def chain_while(
    iterables: Iterable[Generator[Y, S, R]],
    condition: Callable[[R], bool],
) -> Generator[Y, S, None]:
    for it in iterables:
        result = yield from it
        if not condition(result):
            break


def r(x: int) -> Generator[int, None, bool]:
    yield from range(x)
    return x % 2 == 1


print(list(chain_while(
    [
        r(5),
        r(4),
        r(3),
    ],
    lambda x: x is True,
)))

Your task is to create a file-like object that allows you to read from several files as though they are glued into a single one. Write a class that produces such instances.

a = IOChain(f1, f2).read()
# a is now all the bytes from f1 and f2 concatenated
b = IOChain(f3, f4).read(1)
# b is now the first byte of f3,
# or the first by of f4 if f3 is empty

Welcome to the weekend task section. Below is the task that you can solve in Python. My solution will be published in 36 hours.

Hello! I turned 30 today. It’s a big time for me and also for the channel. I now proudly introduce the new type of posts we will have every weekend. Every Saturday morning the simple programming problem will be posted. After exactly 36 hours I’ll share my solution written in Python. I hope it will be fun for both people who try to solve the problem by their own and for those who don’t. There are more than four thousand subscribers by now, that I’m really happy to see. I want to thank you all for your attention and feedback. If you have something to say, feel free to contact me any time you wish to — @pushtaev. If you enjoy @pythonetc you can support my work at ko-fi, yasobe.ru or simply by clicking the cake below.

There is a service with python library it helps you to process cryptocurrency payments and donations. This service is called https://cryptopay.click. Docs: http://docs.cryptopay.click

Sponsorship time!

Python floats can have NaN values. You can get one with math.nan. nan is not equal to anything including itself:

>>> math.nan == math.nan
False

Also, NaN object is not unique, you can have several different NaN objects from different sources:

>>> float('nan')
nan
>>> float('nan') is float('nan')
False

That means that you generally can’t use NaN as a dictionary key:

>>> d = {}
>>> d[float('nan')] = 1
>>> d[float('nan')] = 2
>>> d
{nan: 1, nan: 2}

In Python, None is equal to None so it looks like you can check for None with ==:

ES_TAILS = ('s', 'x', 'z', 'ch', 'sh')


def make_plural(word, exceptions=None):
    if exceptions == None:  # ← ← ←
        exceptions = {}

    if word in exceptions:
        return exceptions[word]
    elif any(word.endswith(t) for t in ES_TAILS):
        return word + 'es'
    elif word.endswith('y'):
        return word[0:-1] + 'ies'
    else:
        return word + 's'

exceptions = dict(
    mouse='mice',
)

print(make_plural('python'))
print(make_plural('bash'))
print(make_plural('ruby'))
print(make_plural('mouse', exceptions=exceptions))

This is a wrong thing to do though. None is indeed is equal to None, but it’s not the only thing that is. Custom objects may be equal to None too:

>>> class A:
...     def __eq__(self, other):
...             return True
...
>>> A() == None
True
>>> A() is None
False

The only proper way to compare with None is to use is None.

Different data structures are merged by different means in Python. Lists use +:

>>> [1, 2] + [2, 3]
[1, 2, 2, 3]

Tuples and strings use + as well:

>>> (1, 2) + (2, 3)
(1, 2, 2, 3)
>>> "12" + "23"
'1223'

The same is true for deque:

>>> deque([1,2]) + deque([2,3])
deque([1, 2, 2, 3])

Sets, on the other hand, use |:

>>> {1, 2} | {2, 3}
{1, 2, 3}

Dicts really stand apart here. Merging dicts is not that simple: the order is important if both operands contain the same key:

>>> {**dict(a=1, b=2), **dict(b=3, c=4)}
{'a': 1, 'b': 3, 'c': 4}
>>> {**dict(b=3, c=4), **dict(a=1, b=2)}
{'b': 2, 'c': 4, 'a': 1}

Counters can be merged with +, the values are summed in the case:

>>> Counter(dict(a=1, b=2)) + Counter(dict(b=3, c=4))
Counter({'b': 5, 'c': 4, 'a': 1})

Python has a very short list of built-in constants. One of them is Ellipsis which is also can be written as .... This constant has no special meaning for the interpreter but is used in places where such syntax looks appropriate. numpy supports Ellipsis as a __getitem__ argument, e. g. x[...] returns all elements of x. PEP 484 defines additional meaning: Callable[..., type] is a way to define a type of callables with no argument types specified. Finally, you can use ... to indicate that function is not yet implemented. This is a completely valid Python code:

def x():
    ...

Functions declared in a class body can’t see the class scope. It makes sense since the class scope only exists during class creation.

>>> class A:
...     x = 2
...     def f():
...         print(x)
...     f()
...
[...]
NameError: name 'x' is not defined

That is usually not a problem: methods are declared inside a class only to become methods and be called later:

>>> class A:
...     x = 2
...     def f(self):
...         print(self.x)
...
>>>
>>>
>>> A().f()
2

Somewhat surprisingly, the same is true for comprehensions. They have their own scopes and can’t access the class scope as well. That really make sense for generator comprehensions: they evaluate expressions after the class creation is already finished.

>>> class A:
...     x = 2
...     y = [x for _ in range(5)]
...
[...]
NameError: name 'x' is not defined

Comprehensions, however, have no access to self. The only way to make it work is to add one more scope (yep, that’s ugly):

>>> class A:
...     x = 2
...     y = (lambda x=x: [x for _ in range(5)])()
...
>>> A.y
[2, 2, 2, 2, 2]

If an instance of a class doesn’t have an attribute with the given name, it tires to access the class attribute with the same name.

>>> class A:
...     x = 2
... 
>>> A.x
2
>>> A().x
2

It’s fairly simple for an instance to have attribute that a class doesn’t or have the attribute with the different value:

>>> class A:
...     x = 2
...     def __init__(self):
...         self.x = 3
...         self.y = 4
... 
>>> A().x
3
>>> A.x
2
>>> A().y
4
>>> A.y
AttributeError: type object 'A' has no attribute 'y'

If it’s not that simple, however, if you want an instance behave like it doesn’t have an attribute despite the class having it. To make it happen you have to create custom descriptor that doesn’t allow access from the instance:

class ClassOnlyDescriptor:
    def __init__(self, value):
        self._value = value
        self._name = None  # see __set_name__

    def __get__(self, instance, owner):
        if instance is not None:
            raise AttributeError(
                f'{instance} has no attribute {self._name}'
            )

        return self._value

    def __set_name__(self, owner, name):
        self._name = name


class_only = ClassOnlyDescriptor


class A:
    x = class_only(2)


print(A.x)  # 2
A().x       # raises AttributeError

See also how the Django classonlymethod decorator works: https://github.com/django/django/blob/b709d701303b3877387020c1558a590713b09853/django/utils/decorators.py#L6

Note that the method doesn't get the looked up string as an argument.

You can customize index completions in Jupiter notebook by providing the _ipython_key_completions_ method. This way you can control what is displayed when you press tab after something like d["x:

__slots__ can be used to speed up attributes access and reduce size of an object. However, if you want to be able to assign new attributes, you need to use mutable container inside the __slots__ to store values in. The problem is that using such container might be a way slower than using class with __dict__:

class A:
    __slots__ = ('__items_type__', '__values__')

    def __init__(self, items_type, **values):
        object.__setattr__(self, '__values__',  values)
        object.__setattr__(self, '__items_type__',  items_type)

    def __getattr__(self, item):
        try:
            return self.__values__[item]
        except KeyError:
            raise AttributeError(f'{self} does not have attribute {item}')

    def __setattr__(self, key, value):
        if type(value) is not self.__items_type__:
            raise TypeError(f'Value has wrong type {type(value)}, {self.__items_type__} expected')
        self.__values__[key] = value

class B:
    def __init__(self, items_type, **values):
        object.__setattr__(self, '__dict__',  values)
        object.__setattr__(self, '__items_type__',  items_type)

    def __setattr__(self, key, value):
        if type(value) is not self.__items_type__:
            raise TypeError(f'Value has wrong type {type(value)}, {self.__items_type__} expected')
        self.__dict__[key] = value

a = A(int, first=1)
b = B(int, first=1)

print(timeit('a.first; a.second=2', globals={'a': a}, number=10000000))
print(timeit('b.first; b.second=2', globals={'b': b}, number=10000000))

# 4.367306873999951
# 1.7298872390001634

To solve this problem without sacrificing faster access to attributes that stored in __slots__ you can mention __dict__ inside, so new attributes will be stored in there:

class C:
    __slots__ = ('__items_type__', '__dict__')

    def __init__(self, items_type, **values):
        object.__setattr__(self, '__dict__', values)
        object.__setattr__(self, '__items_type__', items_type)

    def __setattr__(self, key, value):
        if type(value) is not self.__items_type__:
            raise TypeError(f'Value has wrong type {type(value)}, {self.__items_type__} expected')
        self.__dict__[key] = value

c = C(int, first=1)

print(timeit('c.first; c.second=2', globals={'c': c}, number=10000000))
# 1.5695846090002306

Here's an example of boost attrs access speed up to 14 times, just by renaming one attribute to __dict__: https://github.com/samuelcolvin/pydantic/issues/711

Today's post is written by @MrMrRobat

Dictionaries that are used for storing object attributes are not the same that you create with dict though they look exactly the same:

>>> from sys import getsizeof
>>> class A:
...     pass
... 
>>> a = dict()
>>> b = A().__dict__
>>> type(a)
<class 'dict'>
>>> type(b)
<class 'dict'>
>>> a
{}
>>> b
{}
>>> getsizeof(a)
240
>>> getsizeof(b)
112

For reduction in memory used, dictionaries for __dict__ are implemented differently. They are sharing keys across all instances of A. Mind, however, that b is not actually smaller than a, it’s just how getsizeof works. Read all the details here: https://www.python.org/dev/peps/pep-0412/

The attributes of classes are stored in dictionaries, and that could be a problem since they don't preserve order in Python 3.5 and older:

$ cat test.py
class M:
    def __new__(meta, cls, bases, ns):
        print(ns)

class A(metaclass=M):
    a = 1
    b = 2
$ python3.4 test.py
{'__module__': '__main__', 'b': 2, '__qualname__': 'A', 'a': 1}
$ python3.4 test.py
{'a': 1, 'b': 2, '__module__': '__main__', '__qualname__': 'A'}
$ python3.4 test.py
{'__module__': '__main__', 'b': 2, '__qualname__': 'A', 'a': 1}
$ python3.4 test.py
{'__qualname__': 'A', 'a': 1, '__module__': '__main__', 'b': 2}
$ python3.4 test.py
{'b': 2, 'a': 1, '__module__': '__main__', '__qualname__': 'A'}
$ python3.4 test.py
{'b': 2, '__qualname__': 'A', '__module__': '__main__', 'a': 1}

However, you can replace the attribute container by using the __prepare__ metaclass method:

$ cat test.py
from collections import OrderedDict


class M:
    def __new__(meta, cls, bases, ns):
        print(ns)

    @classmethod
    def __prepare__(metacls, cls, bases):
        return OrderedDict()

class A(metaclass=M):
    a = 1
    b = 2
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])

There is no need to do such thing in Python 3.6+, since dictionaries now preserve order:

$ cat test.py
class M:
    def __new__(meta, cls, bases, ns):
        print(ns)

class A(metaclass=M):
    a = 1
    b = 2
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}