Python etc

Today is the first anniversary of @pythonetc. During last year I've created more than one and a half hundreds of unique posts with original content only, published eight monthly digests translated to Russian, shared more than fifty links, and got more than three thousand subscribers. I want to thank you for your attention and feedback; if you have something to say, feel free to contact me any time you wish to — @pushtaev. If you enjoy @pythonetc you can support my work at ko-fi or yasobe.ru.

If you use mltiprocessing.Queue, you should be aware that it uses locks and semaphores to synchronize access to itself. It has a separate lock for reading for the queue, a lock for write to the queue and a semaphore for limiting its size. Both locks are also implemented as semaphores internally, that leaves us with three semaphores for each queue. Here is how lsof for the example program's master process looks like:

$ sudo lsof -p 42716 | grep sem
python  42716 vadim  DEL    REG   0,17           146565 /dev/shm/sem.emkiph
python  42716 vadim  DEL    REG   0,17           146564 /dev/shm/sem.QLpSti
python  42716 vadim  DEL    REG   0,17           146563 /dev/shm/sem.Yeutyj

The problem is, semaphores are not automatically released if the process using it is killed. The example shows how killing one of the subprocesses lead to malfunctioning of the whole program:

$ python master.py
0 -> 0: 0
1 -> 0: 0
{44529, 44530} {44531, 44533} 44535
1 -> 0: 1
0 -> 1: 1
1 -> 0: 2
0 -> 0: 2
1 -> 0: 3
0 -> 0: 3
1 -> 0: 4
0 -> 0: 4
Killing 44531
Consumer 44531 is killed!
{44529, 44530} {44536, 44533} 44535
0 -> 1: 5
1 -> 2: 5
Restarting killer
{44529, 44530} {44536, 44533} 44537
0 -> 1: 6
1 -> 2: 6
0 -> 1: 7
1 -> 2: 7
Killing 44530
Producer 44530 is killed!
{44529, 44538} {44536, 44533} 44537
2 -> 1: 0
2 -> 2: 1
Restarting killer
{44529, 44538} {44536, 44533} 44540
2 -> 2: 2
0 -> 2: 8
0 -> 2: 9
0 -> 2: 0
0 -> 2: 1
2 -> 2: 3
0 -> 2: 2
2 -> 2: 4
0 -> 2: 3
2 -> 2: 5
Killing 44536
Consumer 44536 is killed!
{44529, 44538} {44533, 44543} 44540
# Should work here, but doesn't
Restarting killer
{44529, 44538} {44533, 44543} 44544
Killing 44543
Consumer 44543 is killed!
{44529, 44538} {44545, 44533} 44544
# Should work here, but doesn't
Restarting killer
...

When you fork your process, the random seed you are using is copying across processes. That may lead to processes producing the same “random” result. To avoid this, you have to manually call random.seed() in every process. However, that is not the case if you are using the multiprocessing module, it is doing exactly that for you. Here is the example:

import multiprocessing              
import random                       
import os                           
import sys                          
                                    
def test(a):                        
    print(random.choice(a), end=' ')
                                    
a = [1, 2, 3, 4, 5]                 
                                    
for _ in range(5):                  
    test(a)                         
print()                             
                                    
                                    
for _ in range(5):                  
    p = multiprocessing.Process(    
        target=test, args=(a,)      
    )                               
    p.start()                       
    p.join()                        
print()                             
                                    
for _ in range(5):                  
    pid = os.fork()                 
    if pid == 0:                    
        test(a)                     
        sys.exit()                  
    else:                           
        os.wait()                   
print()

The result is something like:

4 4 4 5 5
1 4 1 3 3
2 2 2 2 2

Moreover, if you are using Python 3.7 or newer, os.fork does the same as well, thanks to the new at_fork hook. The output of the above code for Python 3.7 is:

1 2 2 1 5
4 4 4 5 5
2 4 1 3 1

The unittest.mock.patch decorator can replace any attribute of any module with MagicMock. That can be used for unit-testing as a replacement for other code isolation techniques such as dependency injection.

@patch('sms.send')
def test_now(patched_send):
    create_client('31205551111')
    patched_send.assert_called_with(
        '31205551111',
        'client created'
    )

The significant limitation is that patch doesn't work with built-ins:

# foo.py
from datetime import datetime
def is_odd_hour_now():
    return datetime.now().hour % 2 == 1

# test_foo.py
from datetime import datetime
from unittest.mock import patch
from foo import is_odd_hour_now

@patch('datetime.datetime.now')
def test_is_odd_hour_now(patched_now):
    patched_now.return_value = \
        datetime(2010, 1, 1, 13, 0, 0)
    assert is_odd_hour_now()

That will cause TypeError: can't set attributes of built-in/extension type 'datetime.datetime'. As a workaround you can patch not the original datetime, but the reference that foo holds:

from datetime import datetime
from unittest.mock import patch
from foo import is_odd_hour_now

@patch('foo.datetime')
def test_is_odd_hour_now(patched_datetime):
    patched_datetime.now.return_value = \
        datetime(2010, 1, 1, 13, 0, 0)
    assert is_odd_hour_now()

Under Python 2.7:

>>> import __hello__
Hello world...
>>> import __phello__
Hello world...
>>> import __phello__.spam
Hello world...

Under Python 3.x:

>>> import __hello__
Hello World!

If you check a file of these modules this result will be returned:

>>> __hello__.__file__
'<frozen>'

The byte-code of these modules (see Python source file ./Python/frozen.c) is compiled into Python lib (python27.dll on Windows and libpython2.7.so on Linux). To check whether the module is frozen it's possible to use imp.is_frozen:

>>> import imp
>>> imp.is_frozen('__hello__')
True
>>> imp.is_frozen('__phello__')
True
>>> imp.is_frozen('__phello__.spam')
True

It's also possible to get the code object of these modules and for example get the bytecode:

>>> imp.get_frozen_object('__phello__.spam').co_code
'd\x00\x00GHd\x01\x00S'

Or get the code object filename:

>>> imp.get_frozen_object('__hello__').co_filename
'hello.py'
>>> imp.get_frozen_object('__phello__').co_filename
'hello.py'
>>> imp.get_frozen_object('__phello__.spam').co_filename
'hello.py'

To load a frozen module Python C API function, PyImport_ImportFrozenModule is used.

Another guest story, by @delimitry.

Sorting a list with None values can be challenging:

In [1]: data = [
   ...:     dict(a=1),
   ...:     None,
   ...:     dict(a=-3),
   ...:     dict(a=2),
   ...:     None,
   ...: ]

In [2]: sorted(data, key=lambda x: x['a'])
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-4-91ebdd6ce23f> in <module>
----> 1 sorted(data, key=lambda x: x['a'])

<ipython-input-4-91ebdd6ce23f> in <lambda>(x)
----> 1 sorted(data, key=lambda x: x['a'])

TypeError: 'NoneType' object is not subscriptable

You may try to remove Nones and put them back after sorting (to the end or the beginning of the list depending on your task):

In [3]: sorted(
   ...:     (d for d in data if d is not None),
   ...:     key=lambda x: x['a']
   ...: ) + [
   ...:     d for d in data if d is None
   ...: ]
Out[3]: [{'a': -3}, {'a': 1}, {'a': 2}, None, None]

That's a mouthful. The better solution is to use more complex key:

In [4]: sorted(data, key=lambda x: float('inf') if x is None else x['a'])
Out[4]: [{'a': -3}, {'a': 1}, {'a': 2}, None, None]

For types where no infinity is available you can sort tuples instead:

In [5]: sorted(data, key=lambda x: (1, None) if x is None else (0, x['a']))
Out[5]: [{'a': -3}, {'a': 1}, {'a': 2}, None, None]

Python developers used to initialize dictionary in different ways:

>>> foo = {}
>>> foo = dict()

By far, the result is the same, but let's dive deeper:

>>> timeit.timeit('{}', number=10**8)
2.65754420599842
>>> timeit.timeit('dict()', number=10**8)
6.245466648993897

Seems like the second dictionary initialization is at least two times slower. Let's see why it happens by using the dis module which allows inspecting the guts of the python bytecode:

>>> dis.dis('foo={}')
  1           0 BUILD_MAP                0
              2 STORE_NAME               0 (foo)
              4 LOAD_CONST               0 (None)
              6 RETURN_VALUE
>>> dis.dis('foo=dict()')
  1           0 LOAD_NAME                0 (dict)
              2 CALL_FUNCTION            0
              4 STORE_NAME               1 (foo)
              6 LOAD_CONST               0 (None)
              8 RETURN_VALUE

The answer is on the surface — whenever you prefer dict() to {}, you lose additional time on a redundant function call. But in most python applications it doesn't really matter because we have a deal with microseconds. Anyway — keep in mind.

Another guest story today. This one is written by @karbachinsky.

0_0 is a totally valid Python expression.

If you are feeling old-school, you might like it 🔥

Ads time! If you are full-stack developer or interested in frontend development, checkout out @thefrontend. They post tips, tutorials and news as well as cute things like that:

Comments are welcome, as always.

Example Before applying the pattern:

from unittest.mock import MagicMock, call

# We have several interactions with the "values" argument.
def do_work(values): # Clients: do_work(values)
    res = 0
    res += values.get_next()  # One interaction
    res /= values.get_next()  # Another interaction
    return res

# Unfortunately, this may lead to various issues,
# caused by high coupling between the function and its arguments.
# Such issues are usually unclear, multiple and diverse.
def test_do_work():
    values = MagicMock()
    # Recording the calls looks innocent,
    # but it is likely to backfire eventually.
    values.get_next.side_effect = [10, 2]

    assert do_work(values) == 5  # (0+10)/2 = 5
    values.get_next.assert_has_calls([call(), call()])

# We commit the code with the test to the repository
# but the project keeps evolving.

After code change:

from unittest.mock import MagicMock, call

# For example, now we need to change the function
# by reordering the calls to "data".
# But we depend on the order of the values stored in the test double,
# which has no business meaning at all.
# It requires us to do a simultaneous update to all the tests as well.
def do_work(values):
    res = 0
    res /= values.get_next()  # These two line are now swapped
    res += values.get_next()
    return res

def test_do_work():
    values = MagicMock()
    values.get_next.side_effect = [2, 10]  # Making update to the order of data

    assert do_work(values) == 10 # 0/2 + 10 = 10
    values.get_next.assert_has_calls([call(), call()])

# If we don't change the order, it would be 0/10 + 2 = 2
# Such simultaneous updates unnecessarily
# consume the development resources,
# but are unavoiable with the current architecture.
# Another case that likely to reveal coupling problems is multi-threading,
# when "values" may be accessed in a virtually random order.

After applying the pattern:

from unittest.mock import MagicMock, call

# However, if we treat each interaction individually,
# the method could be changed more freely.
def do_work(additive, denominator):  # Clients: do_work(values, values)
    res = 0
    res += additive.get_next()
    res /= denominator.get_next()
    return res

def test_do_work():
    additive = MagicMock()
    additive.get_next.side_effect = [10]
    denominator = MagicMock()
    denominator.get_next.side_effect = [2]

    assert do_work(additive, denominator) == 5  # (0+10)/2 = 5

    additive.get_next.assert_called_with()
    denominator.get_next.assert_called_with()

After code change:

from unittest.mock import MagicMock, call

def do_work(additive, denominator):
    res = 0
    res /= denominator.get_next()  # The order here is also swapped.
    res += additive.get_next()     # Now we can keep fewer things in mind,
                                   # when changing it. 
    return res

def test_do_work():
    # Now the order of test data can not affect how the method works,
    # which is a benefit of a looser coupling.
    additive = MagicMock()
    additive.get_next.side_effect = [10]
    denominator = MagicMock()
    denominator.get_next.side_effect = [2]

    # We update only what has actually changed - the output.
    assert do_work(additive, denominator) == 10

    additive.get_next.assert_called_with()
    denominator.get_next.assert_called_with()

Approach Make a copy of an argument for each interaction where it participates, and give them new names, that makes sense for how they are used. When the function is called, provide the same pointer for all new duplicated arguments. At very least you always can split the arguments into the ones dedicated exclusively for output and those for input. On the technical side, this is also an affordable refactoring, because it involves only the function itself and its direct callers, so it does not require massive codebase changes. If you end up with a huge function signature after this, it may be interpreted as a symptom of an overly high coupling between the function and the initial arguments, or that the function simply does a too much and can benefit from a breakdown into smaller ones. Anyway, such an outcome could serve as a hint for the next codebase rework if deemed necessary. Another way to deal with such bloated signatures is to serve all new arguments as elements of a collection through a single original argument. This solves the issue, but will requires more changes inside the function body, because now you'll need to retrieve the relevant value each time from the collection.

Motivation A function is easier to use and test if each argument participates in no more than a single interaction inside its body. Otherwise, when an argument does not have a narrow, well-defined purpose, it ends up having an overcomplicated type, which is required to handle all the use cases. This is a case of high coupling, which is not a new concept for the majority. Articles and papers are usually very abstract in discussing it, but in testing, it manifests straightforwardly - by complicating the mocks and causing false negative results. Moreover, those issues are easy to miss initially, because they are not immediately apparent until somebody tries to change a high-coupled component. This pattern suggests substituting a complex argument type with multiple, but elementary ones. It’s done by spreading the interactions across new arguments, each capable of handling just a single case, so the types stay as simple as possible. Even though this technique can provide a huge convenience in tests, it may also produce not-so-pretty method signatures, so it’s important not to overdo it. However, when working on a legacy code without any coverage, it may be efficiently applied to the full extent simply to enable testing. Besides that, it is an essential intermediary step in achieving adherence to the Law of Demeter in your system. It breaks down the knowledge that a function has about its arguments (namely, the types), which creates the initial outlines for the possible breakdown of all participating objects.

Type breakdown Thesis: The Single Responsibility Principle applies also to data types

This post is the 8th installment of the series of 16 software design patterns dedicated to improving code testability. Published every Tuesday. Provided by @NikolayRys.

The round function rounds a number to a given precision in decimal digits.

>>> round(1.2)
1
>>> round(1.8)
2
>>> round(1.228, 1)
1.2

Also you can set up negative precision:

>>> round(413.77, -1)
410.0
>>> round(413.77, -2)
400.0

round returns value of type of input number:

>>> type(round(2, 1))
<class 'int'>

>>> type(round(2.0, 1))
<class 'float'>

>>> type(round(Decimal(2), 1))
<class 'decimal.Decimal'>

>>> type(round(Fraction(2), 1))
<class 'fractions.Fraction'>

For your own classes you can define round processing with the __round__ method:

>>> class Number(int):
...   def __round__(self, p=-1000):
...     return p
...
>>> round(Number(2))
-1000
>>> round(Number(2), -2)
-2

Values are rounded to the closest multiple of 10 ** (-precision). For example, for precision=1 value will be rounded to multiple of 0.1: round(0.63, 1) returns 0.6. If two multiples are equally close, rounding is done toward the even choice:

>>> round(0.5)
0
>>> round(1.5)
2

Sometimes rounding of floats can be a little bit surprising:

>>> round(2.85, 1)
2.9

This is because most decimal fractions can't be represented exactly as a float:

>>> format(2.85, '.64f')
'2.8500000000000000888178419700125232338905334472656250000000000000'

If you want to round half up you can use decimal.Decimal:

>>> from decimal import Decimal, ROUND_HALF_UP
>>> Decimal(1.5).quantize(0, ROUND_HALF_UP)
Decimal('2')
>>> Decimal(2.85).quantize(Decimal('1.0'), ROUND_HALF_UP)
Decimal('2.9')
>>> Decimal(2.84).quantize(Decimal('1.0'), ROUND_HALF_UP)
Decimal('2.8')