Python etc

Ir al canal en Telegram

Regular tips about Python and programming in general Owner — @pushtaev The current season is run by @orsinium Tips are appreciated: https://ko-fi.com/pythonetc / https://sobe.ru/na/pythonetc © CC BY-SA 4.0 — mention if repost

El país no está especificadoTecnologías y Aplicaciones16 318

6 179

Suscriptores

Sin datos24 horas

Sin datos7 días

Sin datos30 días

Sin datos

Visitas de la publicación

Sin datos24 horas

Sin datos48 horas

Sin datos

Tasa de compromiso

Sin datos

Mensajes por día

Ads index

beta

Archivo de publicaciones

6 179

The popular method to declare an abstract method in Python is to use NotImplentedError exception:

def human_name(self):
    raise NotImplementedError

Though it's pretty popular and even has IDE support (PyCharm considers such method to be abstract), this approach has a downside. You get the error only upon method call, not on class instantiation. Use abc to avoid this problem:

from abc import ABCMeta, abstractmethod
class Service(metaclass=ABCMeta):
    @abstractmethod
    def human_name(self):
        pass

Also be aware that NotImplemented is not the same that NotImplementedError. It's not even an exception. It's a special value (like True and False) that has an absolutely different meaning. Some special methods may return it (e.g., __eq__(), __add__(), etc.) so Python tries to reflect operation. If a.__add__(b) returns NotImplemented, Python tries to call b.__radd__.

6 179

Python substitution does not fallback to addition with negative value. Consider the example:

class Velocity:
    SPEED_OF_LIGHT = 299_792_458

    def __init__(self, amount):
        self.amount = amount

    def __add__(self, other):
        return type(self)(
            (self.amount + other.amount) /
            (
                1 +
                self.amount * other.amount /
                self.SPEED_OF_LIGHT ** 2
            )
        )

    def __neg__(self):
        return type(self)(-self.amount)

    def __str__(self):
        amount = int(self.amount)
        return f'{amount} m/s'

That doesn’t work:

v1 = Velocity(20_000_000)
v2 = Velocity(10_000_000)

print(v1 - v2)
# TypeError: unsupported operand type(s) for -: 'Velocity' and 'Velocity

Funny enough, that does:

v1 = Velocity(20_000_000)
v2 = Velocity(10_000_000)

print(v1 +- v2)
# 10022302 m/s

6 179

Python supports parallel assignment meaning that all variables are modified at once after all expressions are evaluated. Moreover, you can use any expression that supports assignment, not only variables:

def shift_inplace(lst, k):
    size = len(lst)
    lst[k:], lst[0:k] = lst[0:-k], lst[-k:]

lst = list(range(10))

shift_inplace(lst, -3)
print(lst)
# [3, 4, 5, 6, 7, 8, 9, 0, 1, 2]

shift_inplace(lst, 5)
print(lst)
# [8, 9, 0, 1, 2, 3, 4, 5, 6, 7]

6 179

If you access the class attribute that is a descriptor object you get what its __get__ returns, not the object itself:

class Descriptor:
    def __get__(self, *args):
        return 42

class A:
    x = Descriptor()

a = A()
print(a.x)  # 42
print(getattr(a, 'x')) # 42

If you want to get the descriptor object you have to manually check __dict__ of object and all its classes (according to MRO):

class Descriptor:
    def __get__(self, *args):
        return 42

class A:
    x = Descriptor()

class B(A):
    pass

def getattr_raw(obj, attr):
    for x in [obj] + type(obj).mro():
        if attr in x.__dict__:
            return x.__dict__[attr]

    raise AttributeError()

b = B()
print(getattr_raw(b, 'x'))

6 179

The order of except blocks matter: if exceptions can be caught by more than one block, the higher block applies. The following code doesn’t work as intended:

import logging


def get(storage, key, default):
    try:
        return storage[key]
    except LookupError:
        return default
    except IndexError:
        return get(storage, 0, default)
    except TypeError:
        logging.exception('unsupported key')
        return default


print(get([1], 0, 42))  # 1
print(get([1], 10, 42))  # 42
print(get([1], 'x', 42))  # error msg, 42

except IndexError never works since IndexError is a subclass of LookupError. More concrete exception should always be higher:

import logging


def get(storage, key, default):
    try:
        return storage[key]
    except IndexError:
        return get(storage, 0, default)
    except LookupError:
        return default
    except TypeError:
        logging.exception('unsupported key')
        return default


print(get([1], 0, 42))  # 1
print(get([1], 10, 42))  # 1
print(get([1], 'x', 42))  # error msg, 42

6 179

Holiday season in Russia! See you later :).

6 179

The difference between function definition and generator definition is the presence of the yield keyword in the function body:

In : def f():
...:     pass
...:

In : def g():
...:     yield
...:

In : type(f())
Out: NoneType

In : type(g())
Out: generator

That means that in order to create an empty generator you have to do something like this:

In : def g():
...:     if False:
...:         yield
...:

In : list(g())
Out: []

However, since yield from supports simple iterators that better looking version would be this:

def g():
    yield from []

6 179

You can use for not only with variables but with any expression. It’s evaluated on every iteration:

>>> log2 = {}
>>> key = 1
>>> for log2[key] in range(100):
...     key *= 2
... 
>>> log2[16]
4
>>> log2[1024]
10

6 179

Using sorted with the key argument is usually more efficient than providing custom comparison method since key is calculated only once for every value:

>>> sorted([-4, -2, 3, 1], key=lambda x: (print(x), abs(x)))
-4
-2
3
1
[1, -2, 3, -4]

6 179

Ordinary function just needs to call itself to become recursive. It’s not so simple for generators: you usually have to use yield from for recursive generators:

from operator import itemgetter


tree = {
    'imgs': {
        '1.png': None,
        '2.png': None,
        'photos': {
            'me.jpg': None
        },
    },
    'MANIFEST': None,
}


def flatten_tree(tree):
    for name, children in sorted(
        tree.items(),
        key=itemgetter(0)
    ):
        yield name
        if children:
            yield from flatten_tree(children)


print(list(flatten_tree(tree)))

6 179

In asyncio, the common practice to schedule execution of some code at a later time is to spawn a task that does await asyncio.sleep(x):

import asyncio

async def do(n=0):
    print(n)
    await asyncio.sleep(1)
    loop.create_task(do(n + 1))
    loop.create_task(do(n + 1))

loop = asyncio.get_event_loop()
loop.create_task(do())
loop.run_forever()

However, creating a new task may be expensive and is not necessary if you aren't planning to any asynchronous operations (like the do function in the example). Another way to do this is to use loop.call_later and loop.call_at functions that schedule an asynchronous callback to be called:

import asyncio                     
                                   
def do(n=0):                       
    print(n)                       
    loop = asyncio.get_event_loop()
    loop.call_later(1, do, n+1)    
    loop.call_later(1, do, n+1)    
                                   
loop = asyncio.get_event_loop()    
do()                               
loop.run_forever()

6 179

mypy doesn’t yet support recursive types definitions:

from typing import Optional, Dict
from pathlib import Path

TreeDict = Dict[str, 'TreeDict']


def tree(path: Path) -> TreeDict:
    return {
        f.name: tree(f) if f.is_dir() else None
        for f in path.iterdir()
    }

The error is Cannot resolve name "TreeDict" (possible cyclic definition). Stay tuned here: https://github.com/python/mypy/issues/731

6 179

Since Python 3.5, it's actually possible to use unpacking with dictionary and list literals.

In : {**{'a': 1}, 'b': 2, **{'c': 3}}
Out: {'a': 1, 'b': 2, 'c': 3}

In : [1, 2, *[3, 4]]
Out: [1, 2, 3, 4]

For dictionaries, this form is even more powerful than the dict function, since it allows values to be overridden:

In : {**{'a': 1, 'b': 1}, 'a': 2, **{'b': 3}}
Out: {'a': 2, 'b': 3}

6 179

You can define dictionaries in two ways, using literals or the dict function:

>>> dict(a=1, b=2)
{'a': 1, 'b': 2}
>>> {'a': 1, 'b': 2}
{'a': 1, 'b': 2}

Literals work faster than dict, but the function has some advantages. First, you don't need to add additional quotes. However, it only works as long as all keys are valid Python identifiers.

>>> dict(a=1)
{'a': 1}
>>> dict(1='a')
  File "<stdin>", line 1
SyntaxError: keyword can't be an expression

Second, you can't accidentally provide the same key twice:

>>> {'a': 1, 'a': 1}
{'a': 1}
>>> dict(a=1, a=1)
  File "<stdin>", line 1
SyntaxError: keyword argument repeated

Third, you can easily create new the new dictionary based on some already existed one.

>>> d = dict(b=2)
>>> dict(a=1, **d)
{'a': 1, 'b': 2}

Mind, however, that keys can't be redefined with this syntax:

>>> dict(b=3, **d)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: type object got multiple values for keyword argument 'b'

6 179

The attributes of classes are stored in dictionaries, and that could be a problem since they don't preserve order in Python 3.5 and older:

$ cat test.py
class M:
    def __new__(meta, cls, bases, ns):
        print(ns)

class A(metaclass=M):
    a = 1
    b = 2
$ python3.4 test.py
{'__module__': '__main__', 'b': 2, '__qualname__': 'A', 'a': 1}
$ python3.4 test.py
{'a': 1, 'b': 2, '__module__': '__main__', '__qualname__': 'A'}
$ python3.4 test.py
{'__module__': '__main__', 'b': 2, '__qualname__': 'A', 'a': 1}
$ python3.4 test.py
{'__qualname__': 'A', 'a': 1, '__module__': '__main__', 'b': 2}
$ python3.4 test.py
{'b': 2, 'a': 1, '__module__': '__main__', '__qualname__': 'A'}
$ python3.4 test.py
{'b': 2, '__qualname__': 'A', '__module__': '__main__', 'a': 1}

However, you can replace the attribute container by using the __prepare__ metaclass method:

$ cat test.py
from collections import OrderedDict


class M:
    def __new__(meta, cls, bases, ns):
        print(ns)

    @classmethod
    def __prepare__(metacls, cls, bases):
        return OrderedDict()

class A(metaclass=M):
    a = 1
    b = 2
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])
$ python3.4 test.py
OrderedDict([('__module__', '__main__'), ('__qualname__', 'A'), ('a', 1), ('b', 2)])

There is no need to do such thing in Python 3.6+, since dictionaries now preserve order:

$ cat test.py
class M:
    def __new__(meta, cls, bases, ns):
        print(ns)

class A(metaclass=M):
    a = 1
    b = 2
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}
$ python3.6 test.py
{'__module__': '__main__', '__qualname__': 'A', 'a': 1, 'b': 2}

6 179

You can translate or delete characters of a string (like the tr utility does) with the translate method of str:

>>> 'Hello, world!'.translate({
...     ord(','): ';',
...     ord('o'): '0',
... })
'Hell0; w0rld!'

The only argument of translate is a dictionary mapping character codes to characters (or codes). It’s usually more convenient to create such a dictionary with str.maketrans static method:

>>> 'Hello, world!'.translate(str.maketrans({
...     ',': ';',
...     'o': '0',
... }))
'Hell0; w0rld!'

Or even:

>>> 'Hello, world!'.translate(str.maketrans(
...     ',o', ';0'
... ))
'Hell0; w0rld!'

The third argument is for deleting characters:

>>> tr = str.maketrans(',o', ';0', '!')
>>> tr
{44: 59, 111: 48, 33: None}
>>> 'Hello, world!'.translate(tr)
'Hell0; w0rld'

6 179

python -m webbrowser -t 'http://www.python.org'

6 179

A familiar situation: you open a social site and see a block with the accounts of people which you may know. Making such a feature is an example of a Data Scientist task. Do you have a desire to learn the profession and do really cool things? The SkillFactory has a course for you – the Data Science specialization, where you will develop the skills with which you can take up the tasks of training a speech recognition service, identifying fraudulent transactions, forecasting demand for goods and even generating music or poems in the future. Here you will go through the data scientist’s must-have: Python, machine learning, neural networks and deep learning, Big Data and Data engineering. And also mathematics, statistics and a management module. 📍 Stop pulling, get the opportunity in 12 months to work on cool projects in the popular field: https://clc.to/hdwc7A

6 179

Ads time!

6 179

Different asyncio tasks obviously have different stacks. You can view at all of them at any moment using asyncio.all_tasks() to get all currently running tasks and task.get_stack() to get a stack for each task.

import linecache
import asyncio
import random


async def producer(queue):
    while True:
        await queue.put(random.random())
        await asyncio.sleep(0.01)

async def avg_printer(queue):
    total = 0
    cnt = 0
    while True:
        while queue.qsize():
            x = await queue.get()
            total += x
            cnt += 1
            queue.task_done()
        print(total / cnt)
        await asyncio.sleep(1)

async def monitor():
    while True:
        await asyncio.sleep(1.9)
        for task in asyncio.all_tasks():
            if task is not asyncio.current_task():
                f  = task.get_stack()[-1]
                last_line = linecache.getline(
                    f.f_code.co_filename,
                    f.f_lineno,
                    f.f_globals,
                )
                print(task)
                print('\t', last_line.strip())

        print()

async def main():
    loop = asyncio.get_event_loop()

    queue = asyncio.Queue()

    loop.create_task(producer(queue))
    loop.create_task(producer(queue))
    loop.create_task(producer(queue))

    loop.create_task(avg_printer(queue))
    loop.create_task(monitor())



loop = asyncio.get_event_loop()
loop.create_task(main())
loop.run_forever()

To avoid messing with the stack object directly and using the linecache module you can call task.print_stack() instead.