Python etc

An interesting story is behind types module. Until Python 2.7, it had types like IntType, TupleType, UnicodeType, and so on. The motivation behind is that before Python 2.4 built-in functions like int and str were constructors for the types but not the types itself. Hence they couldn't be used in type checks:

# before 2.4:
isinstance(1, int)
# False

import types
isinstance(1, types.IntType)
# True

# Python 2.4 and later:
isinstance(1, int)
# True

The interesting thing is that the same story with typing module. Before Python 3.9 we had typing.Dict but now we don't need it because the dict type itself can be used in the same way in type annotations.

More cool things from Python 3.9. PEP-585 introduced generic types support for the built-in types:

# before 3.9:
from typing import List, Type
lst: List[int] = [1, 2, 3]
t: Type[int] = float

# from python 3.9:
lst: list[int] = [1, 2, 3]
t: type[int] = float

So, now, from typing import will become much shorter! Hooray! The next step would be to support int & str instead of Union[int, str]. The only purpose of these types is type annotations. They don't make any runtime type checks:

list[str]({1, 2, 3})
# [1, 2, 3]

isinstance([1, 2, 3], list[str])
# TypeError: isinstance() arg 2 cannot be a parameterized generic

There are a lot of ways to merge two dicts: 1. Long but simple:

merged = d1.copy()
merged.update(d2)

2. Unpacking:

merged = {**d1, **d2}

3. Unpacking again (keys must be strings):

merged = dict(d1, **d2)

4. collections.ChainMap. Result is not dict but so. In python 3.9, PEP-584 introduced the 5th way. Meet the | operator for dict!

merged = d1 | d2

Basically, that is the same as the first way but shorter and can be inlined.

Python from the very first release and until Python 2.7 supported backquotes as a shortcut for repr(...):

>>> a = 1
>>> `a + 2`
'3'
>>> `int`
"<type 'int'>"

In Python 3, it was removed because it's easy to confuse with single quotes and hard to type on some keyboards.

In python 3.9, PEP-616 introduced str.removeprefix and str.removesuffix methods:

'abcd'.removeprefix('ab')
# 'cd'

'abcd'.removeprefix('fg')
# 'abcd'

The implementation is simple (it's implemented on C, of course, but the idea is the same):

def removeprefix(self: str, prefix: str) -> str:
    if self.startswith(prefix):
        return self[len(prefix):]
    return self

Welcome to season 2! In the next episodes: 1. New cool features in Python 3.9 (the final release is planned on 2020-10-05). 2. Python history and features that were removed. 3. PEPs in the "draft" status that aren't accepted nor rejected yet. 4. Short and useful code snippets to empower your code. 5. A lot more features and tricks, like in good old season 1. Also, a few updates: 1. @orsinium joins the @pythonetc team. 2. No ad in this season. Support us on ko-fi if you like it! The show must go on!

Hi again! Here come the news, hope you are still subscribed :). As you may know I'm working on Marusia virtual assistant. I'll share my experience on the “Oh, My Code” show tomorrow. We are also up to discuss my teaching practice and this very channel. Welcome to the live stream. (The show is in Russian with no translation available.) Speaking of the channel. I have great news. There will be the second season of @pythonetc, and it will be very soon. Can't share any details right now, but it's gonna be great (I guess).

It’s time to announce the official end of season 1. It was fun but I’ve said pretty much everything I had to say. I hope that season 2 is yet to come. I'm up to come with something interesting off-season, stay tuned.

Python float literals can have integer or decimal part omitted:

>>> 0.2
0.2
>>> .2
0.2
>>> 2.
2.0

Omitting decimal part but not the period can be useful when having float instead of integer is important:

>>> type(2)
<class 'int'>
>>> type(2.)
<class 'float'>

Python floats can have NaN values. You can get one with math.nan. nan is not equal to anything including itself:

>>> math.nan == math.nan
False

Also, NaN object is not unique, you can have several different NaN objects from different sources:

>>> float('nan')
nan
>>> float('nan') is float('nan')
False

That means that you generally can’t use NaN as a dictionary key:

>>> d = {}
>>> d[float('nan')] = 1
>>> d[float('nan')] = 2
>>> d
{nan: 1, nan: 2}

If you want to pass some information down the call chain, you usually use the most straightforward way possible: you pass it as functions arguments. However, in some cases, it may be highly inconvenient to modify all functions in the chain to propagate some new piece of data. Instead, you may want to set up some kind of context to be used by all functions down the chain. How can this context be technically done? The simplest solution is a global variable. In Python, you also may use modules and classes as context holders since they are, strictly speaking, global variables too. You probably do it on a daily basis for things like loggers. If your application is multi-threaded, a bare global variable won't work for you since they are not thread-safe. You may have more than one call chain running at the same time, and each of them needs its own context. The threading module gets you covered, it provides the threading.local() object that is thread-safe. Store there any data by simply accessing attributes: threading.local().symbol = '@'. Still, both of that approaches are concurrency-unsafe meaning they won't work for coroutine call-chain where functions are not only called but can be awaited too. Once a coroutine does await, an event loop may run a completely different coroutine from a completely different chain. That won't work:

import asyncio
import sys

global_symbol = '.'

async def indication(timeout):
    while True:
        print(global_symbol, end='')
        sys.stdout.flush()
        await asyncio.sleep(timeout)

async def sleep(t, indication_t, symbol='.'):
    loop = asyncio.get_event_loop()

    global global_symbol
    global_symbol = symbol
    task = loop.create_task(
        indication(indication_t)
    )
    await asyncio.sleep(t)
    task.cancel()

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(
    sleep(1, 0.1, '0'),
    sleep(1, 0.1, 'a'),
    sleep(1, 0.1, 'b'),
    sleep(1, 0.1, 'c'),
))

You can fix that by having the loop set and restore the context every time it switches between coroutines. You can do it with the contextvars module since Python 3.7.

import asyncio
import sys
import contextvars

global_symbol = contextvars.ContextVar('symbol')

async def indication(timeout):
    while True:
        print(global_symbol.get(), end='')
        sys.stdout.flush()
        await asyncio.sleep(timeout)

async def sleep(t, indication_t, symbol='.'):
    loop = asyncio.get_event_loop()

    global_symbol.set(symbol)
    task = loop.create_task(indication(indication_t))
    await asyncio.sleep(t)
    task.cancel()

loop = asyncio.get_event_loop()
loop.run_until_complete(asyncio.gather(
    sleep(1, 0.1, '0'),
    sleep(1, 0.1, 'a'),
    sleep(1, 0.1, 'b'),
    sleep(1, 0.1, 'c'),
))

Sometimes you want to keep track of N largest elements in some stream. The perfect data structure for this is a heap. To limit the size of a heap with heapq you should use the heappushpop function:

from heapq import heappush, heappushpop
import random

LIMIT = 10

heap = []
for _ in range(1_000_000):
    f = heappush if len(heap) < LIMIT else heappushpop
    f(heap, random.random())

print('\n'.join(str(x) for x in heap))

The similar function is heapreplace. The difference is it makes pop before push, so the value removed may be larger than the item added. Both heapreplace and heappushpop are more efficient than heappush and heappop called separately.

The \ symbol in regular string have special meaning. \t is tab character, \r is carriage return and so on. You can use raw-strings to disable this behaviour. r'\t' is just backslash and t. You obviously can’t use ' inside r'...'. However, it sill can be escaped by \, but \ is preserved in the string:

>>> print(r'It\'s insane!')
It\'s insane!

Brackets are required to create a generator comprehension:

>>> g = x**x for x in range(10)
  File "<stdin>", line 1
    g = x**x for x in range(10)
               ^
SyntaxError: invalid syntax
>>> g = (x**x for x in range(10))
>>> g
<generator object <genexpr> at 0x7f90ed650258>

However they can be omitted if a generator comprehension is the only argument for the function:

>>> list((x**x for x in range(4)))
[1, 1, 4, 27]
>>> list(x**x for x in range(4))
[1, 1, 4, 27]

That doesn’t work for function with more than one argument:

>>> print((x**x for x in range(4)), end='\n')
<generator object <genexpr> at 0x7f90ed650468>
>>>
>>>
>>> print(x**x for x in range(4), end='\n')
  File "<stdin>", line 1
SyntaxError: Generator expression must be parenthesized if not sole argument

Any JSON is syntactically correct Python code. However, true, false and null are not defined by default. Defining them makes it possible to use eval as a JSON parser (which isn’t a good idea anyway):

$ cat json
{"$id":"1","currentDateTime":"2019-04-25T14:16Z","utcOffset":"00:00:00","isDayLightSavingsTime":false,"dayOfTheWeek":"Thursday","timeZoneName":"UTC","currentFileTime":132006753872039629,"ordinalDate":"2019-115","serviceResponse":null}

>>> null = None
>>> true = True
>>> false = False
>>> with open('json') as f:
...     j = eval(f.read())
...
>>> j
{'currentFileTime': 132006753872039629, 'isDayLightSavingsTime': False, 'dayOfTheWeek': 'Thursday', 'utcOffset': '00:00:00', 'serviceResponse': None, '$id': '1', 'timeZoneName': 'UTC', 'ordinalDate': '2019-115', 'currentDateTime': '2019-04-25T14:16Z'}

Function can't be generator and regular function at the same time. If yield is presented anywhere in the function body, the function turns into generator:

def zeros(*, count: int, lazy: bool):
    if lazy:
        for _ in range(count):
            yield 0
    else:
        return [0] * count

zeros(count=10, lazy=True) 
# <generator object zeros at 0x7ff0062f2a98>

zeros(count=10, lazy=False)
# <generator object zeros at 0x7ff0073da570>

list(zeros(count=10, lazy=False))
# []

However, regular function can return another iterator:

def _lazy_zeros(*, count: int):
    for _ in range(count):
        yield 0
    
def zeros(*, count: int, lazy: bool):
    if lazy:
        return _lazy_zeros(count=count)
    return [0] * count

zeros(count=10, lazy=True)
# <generator object _lazy_zeros at 0x7ff0062f2750>

zeros(count=10, lazy=False)
# [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Also, for simple cases generator expressions could be useful:

def zeros(*, count: int, lazy: bool):
    if lazy:
        return (0 for _ in range(count))
    return [0] * count

Today's post is written by @orsinium.

Do you know how your voice assistant recognizes speech, how the bank protects your account from fraudsters and how the online store determines the product for recommendations? These tasks are solved by Data Scientist. It is not surprising that today it is a breakthrough profession with a montly salary of 130,000 rubles. Sounds interesting but complicated, doesn't it? With the Data Science specialization at SkillFactory Data School you can learn the profession online in 12 months. It won't be easy, but it's interesting and super-promising. On the course 20% of theory and 80% of practice on real data are waiting for you: Python, machine learning, neural networks and deep learning, big data and data engineering, mathematics and statistics for the Data Scene + management module. Git repository with your ready-made cases! Find out the details: https://clc.to/kZP5uA

Ads time!

Python provides the powerful library to work with date and time: datetime. The interesting part is, datetime objects have the special interface for timezone support (namely the tzinfo attribute), but this module only has limited support of its interface, leaving the rest of the job to different modules. The most popular module for this job is pytz. The tricky part is, pytz doesn't fully satisfy tzinfo interface. The pytz documentation states this at one of the first lines: “This library differs from the documented Python API for tzinfo implementations.” You can't use pytz timezone objects as the tzinfo attribute. If you try, you may get the absolute insane results:

In : paris = pytz.timezone('Europe/Paris')
In : str(datetime(2017, 1, 1, tzinfo=paris))
Out: '2017-01-01 00:00:00+00:09'

Look at that +00:09 offset. The proper use of pytz is following:

In : str(paris.localize(datetime(2017, 1, 1)))
Out: '2017-01-01 00:00:00+01:00'

Also, after any arithmetic operations, you should normalize your datetime object in case of offset changes (on the borderline of the DST period for instance).

In : new_time = time + timedelta(days=2)
In : str(new_time)
Out: '2018-03-27 00:00:00+01:00'
In : str(paris.normalize(new_time))
Out: '2018-03-27 01:00:00+02:00'

Since Python 3.6, it's recommended to use dateutil.tz instead of pytz. It's fully compatible with tzinfo, can be passed as an attribute, doesn't require normalize, though works a bit slower. If you are interested why pytz doesn't support datetime API, or you wish to see more examples, consider reading the decent article on the topic.