Python etc

You can use any object as a dictionary key in Python as long as it implements the __hash__ method. This method can return any integer as long as the only requirement is met: equal objects should have equal hashes (not vice versa). You also should avoid using mutable objects as keys, because once the object becomes not equal to the old self, it can't be found in a dictionary anymore. There is also one bizarre thing that might surprise you during debugging or unit testing:

In : class A:
...:     def __init__(self, x):
...:         self.x = x
...:
...:     def __hash__(self):
...:         return self.x
...:
In : hash(A(2))
Out: 2
In : hash(A(1))
Out: 1
In : hash(A(0))
Out: 0
In : hash(A(-1))  # sic!
Out: -2
In : hash(A(-2))
Out: -2

In CPython -1 is internally reserved for error states, so it's implicitly converted to -2.

Is Python interpreted or compiled? The simple answer here is interpreted; the right one is — it's both. Python compiles your source code to bytecode (.pyc files). It does that implicitly, but it's still an essential phase of Python code execution. Java, for example, does the same but explicitly: you compile with javac and run with java. Despite that, Python is usually called interpreted language while Java is called compiled language, which is, strictly speaking, not entirely correct. Here is an article on the subject with more details and explanations.

To store any information in memory or on a storage device, you should represent it in bytes. Python usually provides the level of abstraction where you can think about data itself, not its byte form. Still, when you write, say, a string to a file, you deal with a physical structure of data. To put characters into a file you should transform them into bytes; that is called encoding. When you get bytes from a file, you probably want to convert them into meaningful characters; that is call decoding. There are hundreds of encoding methods out there. The most popular one is probably Unicode, but you can't transform anything to bytes with it. In the sense of byte representation, Unicode is not even an encoding. Unicode defines a mapping between characters and their integer codes. 🐍 is 128 013, for example. But to put integers into a file, you need a real encoding. Unicode is usually used with utf-8, which is (usually) a default in Python. When you read form a file, Python automatically decodes utf-8. You can choose any other encoding with encoding= parameter of the open function, or you can read plane bytes by appending b to its mode.

If you need to search through a sorted collection, binary search is what you need. This simple algorithm compares the target value to the middle of the array; the result determines which half should be searched next. Python standard library provides a way to use binary search without directly implementing it. bisect_left function returns the leftmost position in a sorted list for the element, while bisect_right return the rightmost one.

In : from random import randrange
In : from bisect import bisect_left
In : n = 1000000
In : look_for = 555555
In : lst = sorted(randrange(0, n) for _ in range(n))

In : %timeit look_for in lst
69.7 ms ± 449 µs per loop

In : %timeit look_for == lst[bisect_left(lst, look_for)]
927 ns ± 2.28 ns per loop

Though decorators and context managers are quite similar and often interchangeable, context managers are severely more limited. You can't skip a block or execute it twice; it always runs exactly one time. However, you can control whether the exception that is raised inside a context should be propagated to the caller or not. It's done by the slightly obscure way: the exception is suppressed if __exit__ return a true value:

class Atomic:
    def __enter__(self):
        print('BEGIN')

    def __exit__(self, exc_type, exc_value, traceback):
        if exc_type:
            print(
                'ROLLBACK due to {}({})'.format(
                    exc_type, exc_value
                )
            )
        else:
            print('COMMIT')

        return True

with Atomic():
    print('A')

with Atomic():
    print('B')
    raise RuntimeError('C')

The output is:

BEGIN
A
COMMIT
BEGIN
B
ROLLBACK due to <type 'exceptions.RuntimeError'>(C)

When Python executes a method call, say a.f(b, c, d), it should first select the right f function. Due to polymorphism, it depends on the type of a. The process of choosing the method is usually called dynamic dispatch. Python supports only single-dispatch polymorphism because a single object alone (a in the example) affects the method selection. Some other languages, however, may also consider a type of b, c and d. This mechanism is called multiple disaptch. C# is a notable example of languages that support that technique. However, multiple dispatch can be emulated via single-dispatch. The visitor design pattern is created exactly for this. What visitor do is essentially calling single-dispatch twice to imitate double-dispatch. Mind, that the ability to overload methods (like in Java and C++) is not the same as multiple dispatch. Dynamic dispatch works in runtime while overloading solely affects compile time.

Python multiprocessing module allows you to spawn not only processes but threads as well. Mind, however, than CPython is notorious for its GIL (global interpreter lock), the interpreter feature that doesn't allow different threads run Python bytecode simultaneously. That means that threads are only useful when your program spends time outside of Python interpreter, usually waiting for IO. For example, downloading three different Wikipedia articles at once with threads will be as efficient as with processes (and thrice as efficient as downloading using only one process):

from multiprocessing import Pool
from multiprocessing.pool import ThreadPool

def download_wiki_article(article):
    url = 'http://de.wikipedia.org/wiki/'
    return requests.get(url + article)

process_pool = Pool(3)
thread_pool = ThreadPool(3)

thread_pool.map(download_wiki_article, ['a', 'b', 'c'])
# 376 ms ± 11 ms

process_pool.map(download_wiki_article, ['a', 'b', 'c'])
# 373 ms ± 3.17 ms

[download_wiki_article(a) for a in ['a', 'b', 'c']]
# 1.09 s ± 27.9 ms

On the other hand, it doesn't make much sense to solve CPU-heavy tasks with threads:

import math
from multiprocessing import Pool
from multiprocessing.pool import ThreadPool

def f(x):
    return len(str(math.factorial(x)))

process_pool = Pool(4)
thread_pool = ThreadPool(4)
inputs = [i ** 2 for i in range(100, 130)]

[f(x) for x in inputs]
# 1.48 s ± 7.61 ms

thread_pool.map(f, inputs)
# 1.48 s ± 7.78 ms

process_pool.map(f, inputs)
# 478 ms ± 7.55 ms

If you have a CPU-heavy task and want to utilize all the cores you have, then multiprocessing.Pool is for you. It spawns multiple processes and delegates tasks to them automatically. Simply create a pool with Pool(number_of_processes) and run p.map with the list of inputs.

In : import math
In : from multiprocessing import Pool
In : inputs = [i ** 2 for i in range(100, 130)]
In : def f(x):
...:     return len(str(math.factorial(x)))
...:

In : %timeit [f(x) for x in inputs]
1.44 s ± 19.2 ms per loop (...)

In : p = Pool(4)
In : %timeit p.map(f, inputs)
451 ms ± 34 ms per loop (...)

Pagination is the pretty standard problem that countless developers solve every day. If you use a relational database, you can explicitly set the offset with LIMIT:

SELECT *
FROM table
LIMIT 1001, 1100

That indeed returns a hundred of records, from 1001 to 1100. The thing is, it's as hard for a database as selecting all 1001 tuples. So the later page your user requests, the slower you return the result. The solution is to use WHERE instead of LIMIT, asking a client to provide the last result of her current page ($last_seen_id in the example):

SELECT *
FROM table
WHERE id > $last_seen_id
ORDER BY id ASC
LIMIT 100

See the perfect article regarding the subject.

A lot of Python classes start with a similar boilerplate: straightforward constructor, trivial repr and stuff like that:

class Server:
    def __init__(self, ip, version=4):
        self.ip = ip
        self._version = version

    def __repr__(self):
        return '{klass}("{ip}", {version})'.format(
            klass=type(self).__name__,
            ip=self.ip,
            version=self._version,
        )

One way to deal with it is to use popular attrs package, which does a lot of default things automatically driving by few declarations:

@attrs
class Server:
    ip = attrib()
    _version = attrib(default=4)

server = Server(ip='192.168.0.0.1', version=4)

It not only creates initializer and repr for you but a complete set of comparison methods as well. That said, there is the upcoming change in Python 3.7, that brings us data classes, the standard library addition that should solve the same problem (and more). It uses the variable annotations, another comparably new Python feature. Here is an example:

@dataclass
class InventoryItem:
    name: str
    unit_price: float
    quantity_on_hand: int = 0

    def total_cost(self) -> float:
        return self.unit_price * self.quantity_on_hand

When you want to empty a list in Python, you probably do lst = []. In fact, you just create a new empty list and assign it to lst, while all others owners of the same list still have the same content:

In : lst = [1, 2, 3]
In : lst2 = lst
In : lst = []
In : lst2
Out: [1, 2, 3]

While this may seem pretty obvious, the correct solution wasn't straightforward until lst.clear() was introduced in Python 3.3. Before that, you should do del lst[:] or lst[:] = []. It works since slice syntax allows you to modify part of the list, and that part is the whole list in case of [:].

Sometimes you need to know the size of a generator without retrieving the actual values. Some generators support len(), but this is not the rule:

In : len(range(10000))
Out: 10000

In : gen = (x ** 2 for x in range(10000))
In : len(gen)
...
TypeError: object of type 'generator' has no len()

The straightforward solution is to use an intermediate list:

In : len(list(gen))
Out: 10000

Though fully functional, this solution requires enough memory to store all the yielded values. The simple idiom allows to avoid such a waste:

In : sum(1 for _ in gen)
Out: 10000

coverage is a simple tool that can tell which part of your code was run and which was not during program execution. It's usually useful for unit-testing to detect parts that are probably not tested thoroughly enough. Say, we need coverage result for the following script:

if 2 > 1:
    print(':)')
else:
    print(':(')

After we install coverage with pip install coverage we just run:

$ coverage run test.py
:)

As a result, the .coverage file is created in the current directory. Now we want to see the actual report:

$ coverage report
Name      Stmts   Miss  Cover
-----------------------------
test.py       3      1    67%

It says that out of three statements we have, one was never executed (hence total coverage is ~67%). For prettier and more detailed representation we can use coverage html:

In Python, range() defines all integers in a half-open interval. So range(2, 10) means, speaking mathematically, [2, 10). Or, speaking Python, [2, 3, 4, 5, 6, 7, 8, 9]. Despite asymmetry, that is not a mistake nor an accident. It makes perfect sense since it allows you to glue together two adjacent intervals without risk of one-off errors:

[a, c) = [a, b) + [b, c) 

Compare to closed intervals that feel more “natural”:

[a, c] = [a, b] + [b+1, c]

This is also a reason for indexing to start from zero: range(0, N) has exactly N elements. Dijkstra wrote an excellent article on the subject back in 1982.

Python 2 can unpack function parameters if you define them like a tuple:

In : def between(x, (start, stop)):
...:     return start < x < stop
...:
In : interval = (5, 10)
In : between(2, interval)
Out: False
In : between(7, interval)
Out: True

It can even do it recursively:

In : def determinant_2_x_2(((a,b), (c,d))):
...:     print a*d - c*b
...:

In : determinant_2_x_2([
...:     (1, 2),
...:     (3, 4),
...: ])
-2

However, this feature was removed in Python 3. You still can do the same by unpacking manually:

In : def determinant_2_x_2(matrix):
...:     row1, row2 = matrix
...:     a, b = row1
...:     c, d = row2
...:
...:     return a*d - c*b
...:

In : determinant_2_x_2([
...:     (1, 2),
...:     (3, 4),
...: ])
Out: -2

If you want to ignore some exception, you probably do something like this:

try:
    lst = [1, 2, 3, 4, 5]
    print(lst[10])
except IndexError:
    pass

That will work (without printing anything), but contextlib let you do the same more expressively and semantically correct:

from contextlib import suppress
with suppress(IndexError):
    lst = [1, 2, 3, 4, 5]
    lst[10]

When you write a decorator, you almost always should use @functools.wraps:

def atomic(func):
    @functools.wraps(func)
    def wrapper():
        print('BEGIN')
        func()
        print('COMMIT')

    return wrapper

It updates wrapper, so it looks like an original func. It copies __name__, __module__ and __doc__ from func to wrapper. It may help if you generate documentation by pydoc, practice doctest or use some introspection tools. Mind, however, that you still see the original name of the function in a stack trace (it's stored in wrapper.__code__.co_name).

Reduce is a higher-order function that processes an iterable recursively, applying some operation to the next element of the iterable and the already calculated value. You also may know it termed fold, inject, accumulate or somehow else. Reduce with result = result + element brings you the sum of all elements, result = min(result, element) gives you the minimum and result = element works for getting the last element of a sequence. Python provides reduce function (that was moved to functools.reduce in Python 3):

In : reduce(lambda s, i: s + i, range(10))
Out: 45
In : reduce(lambda s, i: min(s, i), range(10))
Out: 0
In : reduce(lambda s, i: i, range(10))
Out: 9

Also, if you ever need such simple lambdas like a, b: a + b, Python got you covered with operator module:

In : from operator import add
In : reduce(add, range(10))
Out: 45

The default list slice in Python creates a copy. It may be undesirable if a slice is too big to be copied, you want a slice to reflect changes in the list, or even want to modify a slice to affect the original object. To solve the problem with copying a lot of data, one can use itertools.islice. It lets you iterate over the part of the list, but doesn't support indexing or modification. The way to have a class for modifiable slices is to create it. Luckily Python provides the suitable abstract base class: collections.abc.MutableSequence (just collections.MutableSequence in Python 2). You only need to override __getitem__, __setitem__, __delitem__, __len__ and insert. The example below doesn't support deletion and inserting, but supports slicing slices and modifications.