es
Feedback
Python etc

Python etc

Ir al canal en Telegram

Regular tips about Python and programming in general Owner — @pushtaev The current season is run by @orsinium Tips are appreciated: https://ko-fi.com/pythonetc / https://sobe.ru/na/pythonetc © CC BY-SA 4.0 — mention if repost

Mostrar más
El país no está especificadoTecnologías y Aplicaciones16 318
6 179
Suscriptores
Sin datos24 horas
Sin datos7 días
Sin datos30 días
Archivo de publicaciones
Example Before refactoring:
# A class that has some logic and data
class WorkDoer:
    def __init__(self, a, b):
        self._a = a
        self._b = b
        self._c = None
        self._d = None
  
    # Some methods that affect the state of the object
    def prepare_c(self):
        self._c = self.prepare_c_from_ab(self._a, self._b)
      
    def prepare_d(self):
        self._d = self.prepare_d_from_ab(self._a, self._b)
      
    def prepare_ab(self):
        self._a = self.make_new_a_from_c(self._c)
        self._b = self.make_new_b_from_d(self._d)

    def do_job(self):
        # The method which we intend to test.
        # Let's assume its outcome depends on the object's internal state. 
        # It also changes this state.
        # We will need to efficiently recreate desireable starting point.

# In tests
def test():
    a = Factory.create_a()
    b = Factory.create_b()
    work_doer = WorkDoer.new(a, b)
    # We are doing a lot of calls to get into the desired state.
    # It can be prohibitively complex. 
    work_doer.prepare_c()
    work_doer.prepare_d()
    work_doer.prepare_ab()

    # Finally exercising it and checking the returned value
    assert work_doer.do_job() == correct_result
    # It's problematic to validate the changes in :a, :b, :c and :d, since they are private
    # We need to expose them, use reflection or add more methods just for testing.

Approach You need to substitute the attributes of the object with a single map-like attribute, containing all of them. They have to be entirely public because the decorator already manages access. For such data containers you may either use any library with the collections from your language, or even create a separate class, but this is mostly a matter of taste. In the mentioned case of an ORM, you may use existing DB model classes for it. This refactoring may be done in two different flavors - the data object reference could be made either public or private on the decorator object. The public option already works similarly to the “soft private,” because to manipulate it you need to type obj.data.field, which is similar to obj._field, where _ and .data serve as a reminder of its internal purpose. If you, contrarywise, want to make it private, then you would need methods to access it. I’ve already mentioned locate and position, but also it may be useful to make the constructor accept a data object, so the initialization would already put it in the desired state. However, these two methods are optional, since you may simply keep the link to data object after instantiating the decorator with it. This whole approach may be generalized by also moving the private methods from the decorator to its data object. They would be public, but their visibility is managed by the wrapper as well.

Motivation (Applies only to object-oriented paradigm.) Encapsulation makes testing harder. It is so because you need to access private data in order to set up a test and then validate the outcome. This pattern allows you to bypass it gracefully, without sacrificing the benefits of private in production. I’m not sure if it has already a specific name, so I’m proposing my own - Locate/Position. It comes from the perception of an object as a dot in the multi-dimensional space, to which we apply the locate operation to find its coordinates in the form of a data vector representing its state. And then we can use the position method to place it wherever we want by providing a new data vector, constructed by us. It's accomplished by splitting the initial object into two - a data object with the attributes and a decorator with the logic. The latter also has to contain a single field with a pointer to this data object with a pair of designated methods to manipulate it. It allows you to control and observe the object state in an organized and convenient manner, even make snapshots or restore them later, while keeping it encapsulated. It also has benefits certain benefits with ORMs, because the data object could naturally represent a database tuple, isolated in tests from the decorator with custom business logic. Admittedly, it is not the only way to access private data. For example, many languages have reflection, or you can just avoid using “hard-private" at all, but they have different shortcomings. With this approach, however, you can stay to true to the intended purpose of your objects both in tests and production. As a side benefit, having both locate and position available on every object greatly helps during the debugging and analyzing the object state.

Locate/Position Thesis: manage object data as a separate entity which has only public attributes.

This post is the fifth installment of the series of 16 software design patterns dedicated to improving code testability. Published every Tuesday.

Before arguments unpacking was introduced, you could use the apply built-in function, which is now deprecated (and completely removed from Python 3). The exact equivalent of f(*a, **k) is apply(f, a, k). If you still want to use apply (for backward compatibility or due to semantic reasons) you could something like this:
def apply(func, args, kwargs):
    return func(*args, **kwargs)

There are two built-in functions that let you analyze iterables without writing trivial and redundant for-loops. These are all and any. any returns True if some of the values are true; all returns True if all of them are. all returns True for an empty iterable while any returns False in that case. Both functions are usually useful while used together with list comprehensions:
package_broken = any(
    part.is_broken() for part package.get_parts()
)
package_ok = all(
    part.ok() for part package.get_parts()
)
any and all are usually interchangeable thanks to De Morgan's laws. Choose one that is easier to understand.

def execute_wrapper(
    params_data, db_data, processed_list
):
    # --- Additional code to deal with impurities ---
    # Setting up and preserving the original values
    modified_params_data = params_data.copy()
    orig_processed_list = Global.processed_data.copy()
    DataBase.get().restore_table(db_data)
    # --- End of additional code ---
 
    # Execution
    perform_job_with_side_effecs(modified_params_data)
  
    # --- More additional code    ---
    # --- to deal with impurities ---
    # Preparing modified data for returns
    modified_processed_list = Global.processed_data
    modified_db_data = DataBase.get().dump_table()
    # Clean up and restore
    Global.processed_data = orig_processed_list
    DataBase.get().truncate_table()
    # Artificial returns
    return (
        modified_params_data,
        modified_db_data,
        modified_processed_list,
    )
    # --- End of additional code ---

Example It illustrates the issues and the additional work caused by the impurities of the function under test, which forces us to create a sandbox environment for it. The initial function:
def perform_job_with_side_effecs(params_data):
    # Mutating some global state 
    Global.processed_data += params_data
    # Modifying params
    params_data.pop()
    # Sending commands to other servers
    DataBase.get().update_table(params_data)
In this particular case, the sandbox environment with the function is encapsulated in an independent wrapper which resembles a pure function but is not thread-safe. However, if we would be testing a pure one, it would be unnecessary, sparing much effort.

Pure functions Thesis: Only pure functions can be tested, so it's better to start with them right away rather than later fake them with unpure ones. Motivation There is a known set of limitations for a function that prescribes it to stay in memory, be cacheable, modify no state and have no side effects, in which case it is called a Pure Function. Such functions are suited for testing particularly well because they give you the full control right out of the box, which lets you exercise them without any additional setup. On the other hand, when dealing with a non-pure one, you always find yourself implementing some sandbox environment, that isolates and manages side effects. To test anything, you need to be able to exercise it sustainably for as much as you want, so that the less pure your function is, the more work you need to put to overcome the gap. However, code which is already pure does not need such preparation at all. In essence, a combination of an unpure function with its sandbox environment may be regarded as an approximation of a pure function in its own right, which then gets tested in place of the original one. Unfortunately, while requiring additional work, it usually ends being just a crude imitation with multiple shortcomings, like a lack of thread-safety, for example. It all suggests that there is a definite value in having a pure function in the first place, instead of imitating it by developing a neutralizing wrapper. So is that the greater is the share of the system that is comprised of pure functions, the easier it is to test. Approach There is no generic way to address that, but there are several technics to look into. For example, Dependency Injection, but it might not always be sufficient. For example, if it is a method that modifies the state of its own object, or if you don't want something to be a part of the interface. You can also consider moving the function’s borders, change its signature, rework the architecture, or simply settle on an impure one. If you're choosing the latter, it might be reasonable to find a way not to re-test those impurities on higher levels, by encapsulating them in Boundary classes and validating them separately. You cannot always win this game since the impure features need to end up somewhere anyway, but you will be rewarded the further you succeed to get.

This post is the fourth installment of the series of 16 software design patterns dedicated to improving code testability. Published every Tuesday.

Sometimes you need to create a function from a more universal one. For example, int() has a base parameter which we would like to freeze to have new base2 function:
>>> int("10")
10
>>> int("10", 2)
2
>>> def base2(x):
...     return int(x, 2)
...
>>> base2("10")
2
The functools.partial allows you to do the same more accurate and semantically clear:
base2 = partial(int, base=2)
It can be helpful when you need to pass a function as an argument to another higher order function, but some arguments should be locked:
>>> map(partial(int, base=2), ["1", "10", "100"])
[1, 2, 4]
Without partial you do something like this:
>>> map(lambda x: int(x, base=2), ["1", "10", "100"])
[1, 2, 4]

There is no support in Python for asynchronous file operations. To make them non-blocking, you have to use separate threads. To asynchronously run code in the thread, you should use the loop.run_in_executor method. The third party aiofiles module does all this for you providing nice and simple interface:
async with aiofiles.open('filename', mode='r') as f:
    contents = await f.read()

If you want a user to specify a relative path to the file in some directory, you have to make sure none of the above files could be accessed by providing a malicious relative path:
>>> Path('/var/data') / Path('a/b')  # OK
PosixPath('/var/data/a/b')
>>> Path('/var/data') / Path('../b')  # oops
PosixPath('/var/data/../b')
>>> Path('/var/data') / Path('/etc/passwd')  # even oopsier
PosixPath('/etc/passwd')
There is a trick to do this. You can abuse the fact the path can be traversed above / and normalize relative path as though it's absolute:
def safe(path):
    return os.path.normpath('/' + path).lstrip('/')

>>> Path('/var/data') / safe('a/b')
PosixPath('/var/data/a/b')
>>> Path('/var/data') / safe('../b')
PosixPath('/var/data/b')
>>> Path('/var/data') / safe('/etc/passwd')
PosixPath('/var/data/etc/passwd')

Seam placing Thesis: Extract functions in such a way, so that they accept and return fewer data. Motivation It is a rather general topic that has a larger scope than this article permits, but it boils down to an idea that you can simplify interactions between functions and modules by aligning software seams and choosing architectural boundaries carefully. In the academical language, these considerations are often referred to as a concept of coupling vs. cohesion) Putting it simpler, we want to make interfaces as straightforward as possible at the expense of a higher complexity encapsulated inside the functions. It is useful because a close interaction which complicates usage becomes immediately apparent in the tests, where you exercise each function several more times in addition to the production code. It also affects the resulting complexity of the test doubles, so this is probably one of the more significant sources of leverage for improving the testability. Approach When designing the software architecture, there are usually numerous ways to distribute the code into functions, which differ significantly in the usage difficulty of the extracted components, so we strive to find the one that requires less input and output data. In tests, this lets us both fake less and validate less. Example
# Our task is to implement "convert_abc_to_z",
# while all the "extract"/"make" functions are
# assumed to be given.
# If some functionality is moved out from
# the "convert_abc_to_z" to another function this way,
# it would require 4 parameters:
def _mnbc_to_z(m, n, b, c):
    x = make_x(b, m)
    y = make_y(x, n)
    z = make_z(y, c)

def convert_abc_to_z(a, b, c):
    m = extract_m(a)
    n = extract_n(a)
    z = _mnbc_to_z(m, n, b, c) # 4 params


# But if we inline "_mnbc_to_z" and
# then extract some code differently,
# we may significantly reduce the data exchange.
# This is going to result in testing
# of the "_ab_to_y" being simpler
# than of the "_mnbc_to_z" above.
def _ab_to_y(a, b):
    m = extract_m(a)
    n = extract_n(a)
    x = make_x(b, m)
    y = make_y(x, n)

def convert_abc_to_z(a, b, c):
    y = _ab_to_y(a, b) # Only 2 params
    z = make_z(y, c)

This post is the third installment of the series of 16 software design patterns dedicated to improving code testability. Published every Tuesday.

>>> bool(datetime(2018, 1, 1).time())
False
>>> bool(datetime(2018, 1, 1, 13, 12, 11).time())
True
Before Python 3.5, datetime.time() objects were considered false if they represented UTC midnight. That can lead to obscure bugs. In the following examples if not may run not because create_time is None, but because it's a midnight.
def create(created_time=None) -> None:
    if not created_time:
        created_time = datetime.now().time()
You can fix that by explicitly testing for None: if created_time is None.

In : int('୧৬𝟙༣')
Out: 1613
0 1 2 3 4 5 6 7 8 9 are not the only characters that are considered digits. Python follows Unicode rules and treats several hundreds of symbols as digits, here is the full list. That affects functions like int, unicode.isdecimal and even re.match:
In : int('෯')
Out: 9

In : '٢'.isdecimal()
Out: True

In : bool(re.match('\d', '౫'))
Out: True

Since Python 3.0, raising an exception in an except block will automatically add the caught exception in the __context__ attribute of the new one. That will cause both exceptions to be printed:
try:
    1 / 0
except ZeroDivisionError:
    raise ValueError('Zero!')
 (most recent call last):
  File "test.py", line 2, in <module>
    1 / 0
ZeroDivisionError: division by zero

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "test.py", line 4, in <module>
    raise ValueError('Zero!')
ValueError: Zero!
You also can add __cause__ to any exception with the raise ... from expression:
division_error = None

try:
    1 / 0
except ZeroDivisionError as e:
    division_error = e

raise ValueError('Zero!') from division_error
 (most recent call last):
  File "test.py", line 4, in <module>
    1 / 0
ZeroDivisionError: division by zero

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 8, in <module>
    raise ValueError('Zero!') from division_error
ValueError: Zero!

All objects that currently exist in the interpreter memory can be accessed via gc.get_objects():
In : class A:
...:     def __init__(self, x):
...:         self._x = x
...:
...:     def __repr__(self):
...:         class_name = type(self).__name__
...:         x = self._x
...:         return f'{class_name}({x!r})'
...:

In : A(1)
Out: A(1)

In : A(2)
Out: A(2)

In : A(3)
Out: A(3)

In : [x for x in gc.get_objects() if isinstance(x, A)]
Out: [A(1), A(2), A(3)]