Python etc

A decorator creates a new object (usually function) using another single function as an argument. However, you may want to provide more than one function. This can't be done straightforwardly due to limitations of Python syntax, but you could use a simple trick to solve the issue. The returned function may contain another decorator that can be reapplied to additional functions to bring in additional behavior. That is kinda what @property does:

@property
def x(self):
    return self._x

@x.setter
def x(self, value):
    self._x = value

This is the example of how you can define a function that uses additional functions for special cases:

from functools import wraps

def make_case_decorator(func):
    def case_decorator(*case_decorator_args):
        def decorator(special_case_func):
            @wraps(func)
            def decorated(*args):
                if case_decorator_args == args:
                    return special_case_func(*args)
                return func(*args)

            decorated.case = make_case_decorator(decorated)

            return decorated

        return decorator

    return case_decorator


def special_cases(func):
    @wraps(func)
    def decorated(*args):
        return func(*args)

    decorated.case = make_case_decorator(decorated)

    return decorated

@special_cases
def fact(x):
    return x * fact(x - 1)

@fact.case(0)
def fact(x):
    return 1

@fact.case(10)
def fact(x):
    print(f'(optimization worked for {x})')
    return 3628800

Python lacks syntax to define positional-only arguments of a function. There are many built-in functions which arguments are positional only though:

In : len([])
Out: 0
In : len(obj=[])
TypeError: len() takes no keyword arguments

Documentation for such cases is somewhat inconsistent, but the proper conventional notation for such arguments is to be followed by / in the list of arguments:

>>> help(len)
Help on built-in function len in module builtins:

len(obj, /)
    Return the number of items in a container.

However, positional-only arguments can be achieved by using *args and manual unpacking:

In : def my_len(*args):
...:     obj, = args
...:     return len(obj)
...:

In : my_len([1, 2])
Out: 2

There are three situations in which a newly created variable cannot be annotated. It's a tuple unpacking, for loops and with statements. All of these examples are invalid:

name: str, age: int = student

for x: int in numbers: 
    ...

with connection() as conn: Connection:
    ...

The proper way to define a type of such variables is to define it upfront without initializing them:

conn: Connection
with connection() as conn:
    ...

Every well-behaved command line utility should accept arguments in the form of options (e. g. -h or --help), options with parameters (--log-level 2) or positional parameters (cp file1 file2). Options are distinguished from positional parameters by having a leading dash (or two dashes). Problems start when positional arguments have to start with a dash, e. g. you want to remove a file with the name of -rf: rm -rf doesn't work this way. The conventional way to solve this problem is to support -- delimiter. Arguments after -- are never interpreted as options:

$ echo test > -rf
$ cat -rf
cat: invalid option -- 'r'
Try 'cat --help' for more information.
$ cat -- -rf
test
$ rm -- -rf
$ cat -- -rf
cat: -rf: No such file or directory

The argparse module automatically handles -- for you.

Some modules may contain such cryptic constructions:

try:
    cache
except NameError:
    cache = {}

Looks like there is no point to do something like this. cache definitely causes the NameError at the beginning of the module since it wasn't assigned before. However, it's not the case if the module is reloaded. When this happens, the dictionary containing all module attributes is reused, giving the module opportunity to reuse attributes of its previous incarnation. If the module is designed to be reloaded, it can rely on this feature. For example, the above code helps it to keep some cache intact upon reloading.

Python supports set literals since 2.7:

>>> s = {1, 2, 3}
>>> s
set([1, 2, 3])
>>> type(s)
<type 'set'>

They use the same curly braces as dictionary literals, the only difference is their content:

>>> type({1, 2, 3})
<type 'set'>
>>> type({1: 1, 2: 2})
<type 'dict'>

The same is true for set and dict comprehensions:

>>> {int(x) for x in '123'}
set([1, 2, 3])
>>> {int(x): x for x in '123'}
{1: '1', 2: '2', 3: '3'}

The only problem is empty curly braces — {} — that has no content and set and dict cannot be distinguished. Historically, they mean empty dictionary, not set, since that syntax was available long before set literals. To create an empty set, use set() instead.

To read a line from stdin before Python 3, you had to use the raw_input function instead of input. Usage of input was pretty dangerous since it executes the input line:

$ echo '[x ** 2 for x in range(10)]' | python2 -c 'print input()'
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

In Python 3 input just reads the line and raw_input is gone. If you want to support both Python 2 and Python 3, you can do something like this:

from contextlib import suppress

with suppress(NameError):
    input = raw_input

The six module already does this for you. It provides input function that only reads the line.

In asyncio, the common practice to schedule execution of some code at a later time is to spawn a task that does await asyncio.sleep(x):

import asyncio

async def do(n=0):
    print(n)
    await asyncio.sleep(1)
    loop.create_task(do(n + 1))
    loop.create_task(do(n + 1))

loop = asyncio.get_event_loop()
loop.create_task(do())
loop.run_forever()

However, creating a new task may be expensive and is not necessary if you aren't planning to any asynchronous operations (like the do function in the example). Another way to do this is to use loop.call_later and loop.call_at functions that schedule an asynchronous callback to be called:

import asyncio                     
                                   
def do(n=0):                       
    print(n)                       
    loop = asyncio.get_event_loop()
    loop.call_later(1, do, n+1)    
    loop.call_later(1, do, n+1)    
                                   
loop = asyncio.get_event_loop()    
do()                               
loop.run_forever()

Any running asyncio coroutine can be cancelled via the cancel() method. CancelledError will be thrown into the coroutine that will lead for it and all wrapping coroutines to be terminated, unless the error is caught and suppressed. CancelledError is a subclass of Exception that means that it can be accidentally caught by try ... except Exception that is meant to catch “any error”. To safely do this within a coroutine, you stuck with something like this:

try:
    await action()
except asyncio.CancelledError:
    raise
except Exception:
    logging.exception('action failed')

If you import a module that was already imported, it doesn't do anything, since Python tracks what modules were already loaded. All such modules are placed into the sys.modules dictionary:

In : import sys
In : 'sys' in sys.modules.keys()
Out: True

If you really need to reload a module, you should use the importlib.reload(m) function. m is an object of a module that was successfully imported before, not a string with its name:

In : import importlib
In : importlib.reload(importlib)
Out[5]: <module 'importlib' from '/home/pushtaev/.ve/pythonetc/lib/python3.6/importlib/__init__.py'>

In Python, a variable name may consist of a single underscore: _. Though usually such names are not descriptive enough and should not be used, there are at least three cases when _ has a conventional meaning. First, interactive Python interpreters use _ to store the result of the last executed expression:

>>> 2 + 2
4
>>> _
4

Second, the gettext module's manual recommends to alias its gettext() function to _() as a way to minimize cluttering your code. Third, _ is used when you have to come up with names for values you don't' care about:

>>> log_entry = '10:50:24 14234 GET /api/v1/test'
>>> time, _, method, location = log_entry.split()

Sometimes you want to run a piece of code and ignore all exceptions that it may raise. It's reasonable for plugins, foreign modules, and other units you don't understand nor trust. The proper way to do this is to use try with except Exception, not bare except:

try:
   foreign()
except Exception:
   logging.warn('fail', exc_info=True)

except without explicit exception type is an equivalent for except BaseException. The difference between BaseException and Exception is that the former includes exceptions you usually don't want to be caught, such as KeyboardInterrupt.

In Python, you can override square brackets operator ([]) by defining __getitem__ magic method. This is how you create an object that virtually contains an infinite number of repeated elements:

class Cycle:
    def __init__(self, lst):
        self._lst = lst

    def __getitem__(self, index):
        return self._lst[
            index % len(self._lst)
        ]

print(Cycle(['a', 'b', 'c'])[100])  # 'b'

The unusual thing here is that the [] operator supports a unique syntax. It can be used not only like this — [2], but also like this — [2:10], or [2:10:2], or [2::2], or even [:]. The semantic is [start:stop:step], but you can use it any way you want for your custom objects. But what __getitem__ gets as an index parameter if you call it using that syntax? The slice objects exist precisely for that.

In : class Inspector:
...:     def __getitem__(self, index):
...:         print(index)
...:
In : Inspector()[1]
1
In : Inspector()[1:2]
slice(1, 2, None)
In : Inspector()[1:2:3]
slice(1, 2, 3)
In : Inspector()[:]
slice(None, None, None)

You can even combine tuple and slice syntaxes:

In : Inspector()[:, 0, :]
(slice(None, None, None), 0, slice(None, None, None))

slice is not doing anything for you except simply storing start, stop and step attributes.

In : s = slice(1, 2, 3)
In : s.start
Out: 1
In : s.stop
Out: 2
In : s.step
Out: 3

Storing users' passwords is a big deal. You can't be sure no intruder ever accesses your database, but you must be sure they can't learn password that your clients may use in some other places. So you can't store the password as is, but you have to save enough information to verify a password once a user reaccesses your service. The simple solution is to store y =h(x) where h is a hash function, and x is a password. To verify that some p is a valid password, you check whether h(p) == y. An intruder may have h(x) calculated for a huge amount of simple x values. As a countermeasure, you should store y = h(x + s), where s is salt. Salt has to be unique for every password stored; if it's not, h(x + s) is just some other hash function: g(x). Salt has to be long enough; if it's not, x + s might be in the intruder's dictionary as easily as x. However, it's not that simple. You can't use any general purpose hash functions as h for that task. They are fast enough and allow intruders to brute-force your whole database. Functions like md5 and sha- of any kind don't suit this purpose. You need a function that is deliberately designed for hashing passwords, e. g. bcrypt or scrypt. Both of them are available through modules of the same name.

There are two concepts with similar names that can be easily confused: overriding and overloading. Overriding happens when a child class defines a method that is already provided by its parents effectively replacing it. In some languages you have to explicitly mark the overriding method (C# requires the override modifier), in some languages it's optional (the @Override annotation in Java). Python doesn't require any special modifier nor does it have a standard way to mark such methods (some people like to use a custom @override decorator that does virtually nothing, just for the sake of readability). Overloading is another story. Overloading is having multiple functions with the same name but different signatures. It's supported by languages like Java and C++ and is often used as a way to provide default arguments:

class Foo {
    public static void main(String[] args) {
        System.out.println(Hello());
    }

    public static String Hello() {
        return Hello("world");
    }

    public static String Hello(String name) {
        return "Hello, " + name;
    }
}

Python doesn't support finding functions by their signatures, only be their names. You can write code that analyzes the types and number of arguments explicitly. That usually looks clumsy and generally is not a nice thing to do:

def quadrilateral_area(*args):
    if len(args) == 4:
        quadrilateral = Quadrilateral(*args)
    elif len(args) == 1:
        quadrilateral = args[0]
    else:
        raise TypeError()

    return quadrilateral.area()

If you need type hints for this, the typing module can help you with the @overload decorator:

from typing import overload

@overload
def quadrilateral_area(
    q: Quadrilateral
) -> float: ...

@overload
def quadrilateral_area(
    p1: Point, p2: Point,
    p3: Point, p4: Point
) -> float: ...

Python lets you know the path to any source file. Within a file, __file__ returns the relative path to it:

$ cat test/foo.py
print(__file__)
$ python test/foo.py
test/foo.py

The typical usage for that is to find the path where the script is located. It can be helpful for finding other files such as configs, assets, etc. To get the absolute path form the relative one you can use os.path.abspath. So the common idiom to get the script directory path is:

dir_path = os.path.dirname(
    os.path.abspath(__file__)
)

The simplest way to use the logging module is to call functions directly from it, without creating a logger object.

import logging
logging.error('xxx')

This global logger can be configured via the logging.basicConfig() call:

import logging
logging.basicConfig(format='-- %(message)s --')
logging.error('xxx')  # -- xxx --

Due to its global nature, basicConfig has some limitation. First, only the first call actually does something, any further calls of basicConfig are entirely ignored. Second, any function that writes a log message calls basicConfig, so you must configure logging before logging any messages:

import logging
logging.error('xxx')  # ERROR:root:xxx
logging.basicConfig(format='-- %(message)s --')
logging.error('xxx')  # ERROR:root:xxx

To set the default values of attributes in a constructor, you usually use a simple if:

def __init__(self, cache=None):
    if cache is None:
        cache = {}
    self._cache = cache

It can be rewritten a little shorter:

def __init__(self, cache=None):
    self._cache = cache or {}

This method a couple of drawbacks though. First, the intent of such or may not be clean enough since it is usually used in boolean context. Second, or checks for False, not for None, that can lead to obscure bugs.

Native Python float values use your computer hardware directly, so any value is represented internally as a binary fraction. That means that you usually work with approximations, not exact values:

In : format(0.1, '.17f')
Out: '0.10000000000000001'

The decimal module lets you use decimal floating point arithmetic with arbitrary precision:

In : Decimal(1) / Decimal(3)
Out: Decimal('0.3333333333333333333333333333')

That's still can be not enough:

In [61]: Decimal(1) / Decimal(3) * Decimal(3) == Decimal(1)
Out[61]: False

For perfect computations, you can use fractions, that stores any number as a rational one:

In : Fraction(1) / Fraction(3) * Fraction(3) == Fraction(1)
Out: True

The obvious limitation is you still have to use approximations to irrational numbers (such as π).

An object instantiation includes two significant steps. First, the __new__ method of a class is called. It creates and returns a brand new object. Second, Python calls the __init__ method of that object. Its work is to set up the initial state of the object. However, __init__ isn't called if __new__ returns an object that is not an instance of the original class. The reason for this is that it was probably created by another class, hence __init__ was already called for that object:

class Foo:
    def __new__(cls, x):
        return dict(x=x)

    def __init__(self, x):
        print(x)  # Never called

print(Foo(0))

That also means that you should not ever create instances of the same class in __new__ with a regular constructor (Foo(...)). It could lead to the double __init__ execution or even infinite recursion. Infinite recursion:

class Foo:
    def __new__(cls, x):
        return Foo(-x)  # Recursion

Double __init__:

class Foo:
    def __new__(cls, x):
        if x < 0:
            return Foo(-x)
        return super().__new__(cls)

    def __init__(self, x):
        print(x)
        self._x = x

The proper way:

class Foo:
    def __new__(cls, x):
        if x < 0:
            return cls.__new__(cls, -x)
        return super().__new__(cls)

    def __init__(self, x):
        print(x)
        self._x = x