Python etc

Some things should be closed after the use. Some of them are provided as context managers (open is a notable example), and some of them aren't (say, socket.socket). Writing such context manager is trivial:

@contextmanager
def socket_context(*args, **kwargs):
    try:
        sock = socket(*args, **kwargs)
        yield sock
    finally:
        sock.close()

To avoid writing a context manager for every type of closing object , you can you universal contextlib.closing:

with closing(socket.socket()) as sock:
    sock.connect(addr)
    sock.sendall(data)

If you still like to have a socket_context name, but don't want to write the monotonous try-yield-finally-close, you should wrap closing:

def socket_context(*args, **kwargs):
    return closing(socket.socket(*args, **kwargs))

Content managers and function decorators are pretty similar and usually interchangeable. Any context manager can be used as a function decorator as long as it's derived from contextlib.ContextDecorator. (Mind, that the @contextlib.contextmanager decorator inherits from that class automatically.) On the other hand, you can't use any decorator as a context manager due to severe limitation: a context manager always runs a code block exactly once while a decorated function may call original one as many times as it wants (zero included). PEP 377 proposed the change that allows __enter__ to ask Python not to run a code block at all, but it was rejected. As a workaround, you should extract a function and apply a decorator to it:

def retry(attempts, exceptions=(Exception,)):       
    def decorator(func):                            
        def decorated(*args, **kwargs):             
            for _ in range(attempts - 1):           
                with suppress(*exceptions):          
                    return func(*args, **kwargs)    
            return func(*args, **kwargs)  # last try
        return decorated                            
    return decorator                                

@retry(10, HTTPError) 
def get(url):
    requests.get(url).raise_for_status()

Unit-tests you write may require some temporary files or directories. The tempfile module can help you to achieve that. Since temporary stuff usually should be removed after use, tempfile provides context manager as well as plain functions:

with tempfile.TemporaryDirectory() as dir_path:
    open(os.path.join(dir_path, 'a'), 'w').close()
    open(os.path.join(dir_path, 'b'), 'w').close()
    open(os.path.join(dir_path, 'c'), 'w').close()

    assert files_of(dir_path) == ['a', 'b', 'c']

Python lets you know the path to any source file. Within that file, __file__ returns the relative path to it:

$ cat test/foo.py
print(__file__)
$ python test/foo.py
test/foo.py

The typical usage for that is to find the path where the script is located. It can be helpful for finding other files such as configs, assets, etc. To get the absolute path form the relative one you can use os.path.abspath. So the common idiom to get the script directory path is:

dir_path = os.path.dirname(os.path.abspath(__file__))

The unittest.mock.patch decorator can replace any attribute of any module with MagicMock. That can be used for unit-testing as a replacement for other code isolation techniques such as dependency injection.

@patch('sms.send')
def test_now(patched_send):
    create_client('31205551111')
    patched_send.assert_called_with(
        '31205551111',
        'client created'
    )   

The significant limitation is that patch doesn't work with built-ins:

# foo.py
from datetime import datetime
def is_odd_hour_now():
    return datetime.now().hour % 2 == 1

# test_foo.py
from datetime import datetime
from unittest.mock import patch
from foo import is_odd_hour_now

@patch('datetime.datetime.now')
def test_is_odd_hour_now(patched_now):
    patched_now.return_value = \
        datetime(2010, 1, 1, 13, 0, 0)
    assert is_odd_hour_now()

That will cause TypeError: can't set attributes of built-in/extension type 'datetime.datetime'. As a workaround you can patch not the original datetime, but the reference that foo holds:

from datetime import datetime
from unittest.mock import patch
from foo import is_odd_hour_now

@patch('foo.datetime')
def test_is_odd_hour_now(patched_datetime):
    patched_datetime.now.return_value = \
        datetime(2010, 1, 1, 13, 0, 0)
    assert is_odd_hour_now()

When you write custom __repr__ for some object, you usually want to include representation of its attributes. You should be careful to call repr() explicitly, since formatting calls str() instead. Here is a simple example:

class Pair:
    def __init__(self, left, right):
        self.left = left
        self.right = right
        
    def __repr__(self):
        class_name = type(self).__name__
        repr_left = repr(self.left)
        repr_right = repr(self.right)
        return f'{class_name}({repr_left}, {repr_right})'

The problem with calling repr on some other objects is that you can't guarantee it's not the same object and the call isn't recursive:

In : p = Pair(1, 2)
In : p
Out: Pair(1, 2)
In : p.right = p
In : p
Out: [...]
RecursionError: maximum recursion depth exceeded while calling a Python object

To easily solve this problem you can use reprlib.recursive_repr decorator:

@reprlib.recursive_repr()
def __repr__(self):
    class_name = type(self).__name__
    repr_left = repr(self.left)
    repr_right = repr(self.right)
    return f'{class_name}({repr_left}, {repr_right})'

Now it works:

In : p = Pair(1, 2)
In : p.right = p
In : p
Out: Pair(1, ...)

Applying a function to the result of another function is called function composition. If you have f : X → Y and g : Y → Z, you can create the composition function h: X → Z: h(x) = g(f(x)). This is also denoted as h = g ∘ f. In python, there is no ∘ operator, but you still can create composition functions via bare lambdas:

In : from math import sqrt
In : sqrt_abs = lambda x: sqrt(abs(x))
In : sqrt_abs(-4)
Out: 2.0

To make it more semantically precise, you can create your own compose function. Adding clear repr is also a good idea. Here is an example:

class compose:
    def __init__(self, *functions):
        self._functions = functions

    def __call__(self, *args, **kwargs):
        result = None
        for f in reversed(self._functions):
            result = f(*args, **kwargs)
            args = [result]
            kwargs = dict()
        return result

    def __repr__(self):
        return '{}({})'.format(
            type(self),
            ', '.join(repr(f) for f in self._functions)
        )

To read a line from stdin before Python 3, you had to use the raw_input function instead of input. Usage of input was pretty dangerous since it executes the input line:

$ echo '[x ** 2 for x in range(10)]' | python2 -c 'print input()'
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

In Python 3 input just reads the line and raw_input is gone. If you want to support both Python 2 and Python 3, you can do something like this:

from contextlib import suppress

with suppress(NameError):
    input = raw_input

The popular six module already does this for you. It provides input function that only reads the line.

Since loops in Python don't create scopes, you usually need an extra function to create closures. The straightforward way doesn't work:

multipliers = []
for i in range(10):
    multipliers.append(lambda x: x * i)

[multipliers[i](2) for i in range(5)]
# [18, 18, 18, 18, 18]

Let's add the extra function:

multiplier_creator = lambda i: lambda x: x * i
for i in range(10):
    multipliers.append(multiplier_creator(i))

It works this way, but the code can be clumsy, especially if you need def, not lambda:

def multiplier_creator(i):
    def multiplier(x):
        return x * i
for i in range(10):
        return multiplier

    multipliers.append(multiplier_creator(i))

To make it slightly more readable, you can write universal function and get partials of it:

multiplier = lambda x, i: x * i
for i in range(10):
    multipliers.append(partial(multiplier, i=i))

You can always emulate partial with custom lambda, but repr of partials are generally more readable:

In : partial(int, base=2)
Out: functools.partial(<class 'int'>, base=2)

In : lambda x: int(x, base=2)
Out: <function __main__.<lambda>>

Fun fact: thanks to the operator module this particular example can be expressed even more appealing:

for i in range(10):
    multipliers.append(partial(operator.mul, i))

In Python, an else block could be presented not only after if, but after for and while as well. The code inside else is executed unless the loop was interrupted by break. The common usage for this is to search something in a loop and use break when found:

In : first_odd = None
In : for x in [2,3,4,5]:
...:     if x % 2 == 1:
...:         first_odd = x
...:         break
...: else:
...:     raise ValueError('No odd elements in list')
...:
In : first_odd
Out: 3

In : for x in [2,4,6]:
...:     if x % 2 == 1:
...:         first_odd = x
...:         break
...: else:
...:     raise ValueError('No odd elements in list')
...:
...
ValueError: No odd elements in list

The itertools.chain function is a way to iterate over many iterables as though they are glued together:

In : list(chain(['a', 'b'], range(3), set('xyz')))
Out: ['a', 'b', 0, 1, 2, 'x', 'z', 'y']

Sometimes you want to know whether a generator is empty (rather say, exhausted). To do this, you have to try getting the next element from the generator. If it works, you would like to put element back in the generator, which of course is not possible. You can glue it back with chain instead:

def sum_of_odd(gen):
    try:
        first = next(gen)
    except StopIteration:
        raise ValueError('Empty generator')

    return sum(
        x for x in chain([first], gen)
        if x % 2 == 1
    )

Usage example:

In : sum_of_odd(x for x in range(1, 6))
Out: 9
In : sum_of_odd(x for x in range(2, 3))
Out: 0
In : sum_of_odd(x for x in range(2, 2))
...
ValueError: Empty generator

In Python, for lets you withdraw elements from a collection without thinking about their indexes:

def find_odd(lst):
    for x in lst:
        if x % 2 == 1:
            return x

    return None

If you do care about indexes you can iterate over range(len(lst)):

def find_odd(lst):
    for i in range(len(lst)):
        x = lst[i]
        if x % 2 == 1:
            return i, x

    return None, None

But perhaps the more semantically correct and expressive way to do the same it to use enumerate:

def find_odd(lst):
    for i, x in enumerate(lst):
        if x % 2 == 1:
            return i, x

    return None, None

PEP 424 allows generators and other iterable objects that don't have the exact predefined size to expose a length hint. For example, the following generator will likely return ~50 elements:

(x for x in range(100) if random() > 0.5)

If you write an iterable and want to add the hint, define the __length_hint__ method. If the length is known for sure, use __len__ instead. If you use an iterable and want to know its expected length, use operator.length_hint.

Let's suppose you have some datetime object and want to know how much time has passed since the start of the day. How do you do that? First of all, to know what day we are talking about, we need to have the timezone; having the datetime is not enough. As long you have the timezone you need to convert the datetime and strip time with the date() method:

def date_of_time(datetime_object, tz):
    return datetime_object.astimezone(tz).date()

Having the date, we can get its midnight time. To do this, we glue 00:00:00 back to the datetime object and assign the original timezone:

def midnight_of_date(date_in_given_timezone, tz):
    midnight = datetime.datetime.combine(
        date_in_given_timezone, datetime.time()
    )

    return tz.localize(midnight)

And now we put things together:

def midnight(datetime_object, tz):
    return midnight_of_date(
        date_of_time(datetime_object, tz), tz
    )

The format method of Python string is a mighty tool that supports a lot of things that you are probably not even aware of. Each replacement placeholder ({...}) may contain three parts: field name, conversion and format specification. The field name is used to specify which argument exactly should be used as a replacement:

>>> '{}'.format(42)
'42'
>>> '{1}'.format(1, 2)
'2'
>>> '{y}'.format(x=1, y=2)
'2'

The conversion let you ask format to use repr() (or ascii()) instead of str() while converting objects to strings:

>>> '{!r}'.format(datetime.now())
'datetime.datetime(2018, 5, 3, 23, 48, 49, 157037)'
>>> '{}'.format(datetime.now())
'2018-05-03 23:49:01.060852'

Finally, the format specification is a way to define how values are presented:

>>> '{:+,}'.format(1234567)
'+1,234,567'
>>> '{:>19}'.format(1234567)
'            1234567'

This specification may be applied to a single object with format function (not the str method):

format(5000000, '+,')
'+5,000,000'

The format function calls __format__ method of the object internally so you can alter its behavior for your types.

Sometimes you want to know whether something is a function or not. The obvious solution is to check the object class with isinstance. The class of functions is function, but you can't access directly. You can instead get type of any existing function:

FunctionType = type(lambda: None)

Now you can do checking:

def isfunction(object):
    return isinstance(object, FunctionType)

Luckily all above code is already written for you: FunctionType is an existing member of types and isfunction already exists in the inspect module. Note, that you usually don't care whether something is a function, but rather if it's callable or not. It can be done with callable:

>>> callable(int)
True
>>> callable(42)
False
>>> callable(callable)
True

Liskov substitution principle tells us that if S is a subtype of T, then all occurrences of T may be replaced with S without breaking any code. That means that S should satisfy all the guarantees that T introduces. Whenever you work with dict you usually assume that as long as x in d returns False, d[x] raises KeyError. But if your dict is a defaultdict, that it's simply not true. Does defaultdict violate the LSP then? Strictly speaking, it doesn't. The dict documentation explicitly says that d[x] may return something even though x is not present in d (if the __missing__ method defined). So you still potentially get something from the dictionary even if the key is not in it. That means that you shouldn't ever do x in d to check the existence of element as long as you want to support all dict subclasses. It may seem that defaultdict violates the LSP, but it “breaks” the guarantees that were never there.

Sometimes you need to create a function from a more universal one. For example, int() has a base parameter which we would like to freeze to have new base2 function:

>>> int("10")
10
>>> int("10", 2)
2
>>> def base2(x):
...     return int(x, 2)
...
>>> base2("10")
2

The functools.partial allows you to do the same more accurate and semantically clear:

base2 = partial(int, base=2)

It can be helpful when you need to pass a function as an argument to another higher order function, but some arguments should be locked:

>>> map(partial(int, base=2), ["1", "10", "100"])
[1, 2, 4]

Without partial you do something like this:

>>> map(lambda x: int(x, base=2), ["1", "10", "100"])
[1, 2, 4]

Creating an external process in Python is an easy task, you can do it with subprocess module. However, reading both stdout and stderr of the spawned process may be more challenging. Let's suppose we ask Popen to create two pipes, one for stdout and one for stderr:

p = subprocess.Popen(
    ["python", "-c", "..."],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

Now we have to read from them. The problem is, you can't just do readline() for any of those pipes since it can cause deadlocks. Consider the more concrete example:

import subprocess

SUBPROCESS_CODE = """
import sys
sys.stderr.write('err')
print('out')
"""

p = subprocess.Popen(
    ["python", "-c", SUBPROCESS_CODE],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

print(p.stdout.readline())

The primary process creates the child and waits for stdout. The child process first writes to stderr pipe and then to stdout. The main process successfully receives 'out' from the pipe. But what about 'err'? It's stored in the pipe buffer until the main process does p.stderr.readline(). But what if the buffer is full?

import subprocess

SUBPROCESS_CODE = """
import sys
for _ in range(100000):
    sys.stderr.write('err')
print('out')
"""

p = subprocess.Popen(
    ["python", "-c", SUBPROCESS_CODE],
    stdout=subprocess.PIPE,
    stderr=subprocess.PIPE,
)

print(p.stdout.readline())

In this case sys.stderr.write('err') will be blocked at some point until someone reads from the buffer. But no one ever will: the main process waits for data in stdout. This is the deadlock we are talking about. To solve this problem, you should read from both stdout and stderr at once. You can do it with select module or simply use p.communicate(). The second approach is much more straightforward but doesn't let you read data line by line.