What's new in pytest 3.3

published 2017-11-27 18:00
written by nicoddemus
tags: whatsnew

This blog post provides a short overview of some of the major features and changes in pytest 3.3. See the CHANGELOG for the complete list.

Python 2.6 and 3.3 support dropped

Pytest 3.3 and onwards will no longer support Python 2.6 and 3.3. Those Python versions are EOL for some time now and incur maintenance and compatibility costs on the pytest core team, and following up with the rest of the community we decided that they will no longer be supported starting with this version. Users using those versions and with a modern enough pip should not be affected, otherwise they should pin pytest to < 3.3.

Logging capture

Pytest now captures and displays output from the standard logging module. The user can control the logging level to be captured by specifying options in pytest.ini, the command line and also during individual tests using markers. Also, a caplog fixture is available that enables users to test the captured log during specific tests (similar to capsys for example). For more information, please see the logging docs. This feature was introduced by merging the popular pytest-catchlog plugin, thanks to Thomas Hisch. Be advised that with this merge the backward compatibility interface with the defunct pytest-capturelog has been dropped.

Progress display

The pytest output now includes a percentage indicator. Here's the partial output of pytest's own test suite:

============================= test session starts =============================
platform win32 -- Python 3.6.3, pytest-3.3.0, py-1.5.2, pluggy-0.6.0
rootdir: C:\Users\appveyor\AppData\Local\Temp\1\devpi-test0\targz\pytest-3.3.0, inifile: tox.ini
plugins: hypothesis-3.38.5
collected 1985 items

testing\acceptance_test.py ...............s............................. [  2%]
...x.............                                                        [  3%]
testing\deprecated_test.py ........                                      [  3%]
testing\test_argcomplete.py ss                                           [  3%]
testing\test_assertion.py .............................................. [  5%]
...........................s.......                                      [  7%]
testing\test_assertrewrite.py ......................................s..s [  9%]
sss...........                                                           [ 10%]

The output can be changed back to its former style by configuring your pytest.ini:

[pytest]
console_output_style=classic

New capfdbinary and capsysbinary fixtures

Those two new fixtures return their contents as bytes instead of str, making them suitable to interact with programs that write binary data to stdout/stderr.

pytest.skip() at module level

pytest.skip in 3.0 was changed so it could not be used at the module level; it was a common mistake to use it as a decorator in a test function, when the user should have used pytest.mark.skip instead, which would cause the whole module to be skipped.

Some users missed this functionality, so allow_module_level can be passed to pytest.skip to skip an entire module:

import pytest

pytest.skip('this whole module is TODO', allow_module_level=True)

New dependencies

Pytest now depends on the following modules:

This change should not affect users.

Improvements and bugfixes

As usual, this release includes a lot of small improvements and bug fixes. Read the full CHANGELOG for the complete list.

What's new in pytest 3.0

published 2016-07-12 12:00
written by nicoddemus
tags: whatsnew

This blog post provides a short overview of some of the major features and changes in pytest 3.0. See the CHANGELOG for the complete list.

Additional command alias: pytest

pytest came into existence as part of the py library, providing a tool called py.test.

Even after pytest was moved to a separate project, the py.test name for the command-line tool was kept to preserve backward compatibility with existing scripts and tools.

In pytest-3.0 the command-line tool is now called pytest, which is easier to type and prevents some common misunderstandings when tools with the same name are installed in the system.

Note that py.test still works.

approx(): function for comparing floating-point numbers

The approx function makes it easy to perform floating-point comparisons using a syntax that's as intuitive and close to pytest's philosophy:

from pytest import approx

def test_similar():
    v = 0.1
    assert (v + 0.2) == approx(0.3)

yield statements in @pytest.fixture

Fixtures marked with @pytest.fixture can now use yield statements to provide values for test functions exactly like fixtures marked with @pytest.yield_fixture decorator:

import smtplib
import pytest

@pytest.fixture(scope="module")
def smtp():
    smtp = smtplib.SMTP("smtp.gmail.com")
    yield smtp
    smtp.close()

This makes it easier to change an existing fixture that uses return to use yield syntax if teardown code is needed later. Also, many users consider this style more clearer and natural than the previous method of registering a finalizer function using request.addfinalizer because the flow of the test is more explicit. Consider:

@pytest.fixture(scope="module")
def smtp(request):
    smtp = smtplib.SMTP("smtp.gmail.com")
    request.addfinalizer(smtp.close)
    return smtp

This change renders @pytest.yield_fixture deprecated and makes @pytest.fixture with yield statements the recommended way to write fixtures which require teardown code.

Note that registering finalizers in request is still supported and there's no intention of removing it in future pytest versions.

doctest_namespace fixture

The doctest_namespace fixture can be used to inject items into the namespace in which doctests run. It is intended to be used by auto-use fixtures to provide names to doctests.

doctest_namespace is a standard dict object:

# content of conftest.py
import numpy
@pytest.fixture(autouse=True)
def add_np(doctest_namespace):
    doctest_namespace['np'] = numpy

Doctests below the conftest.py file in the directory hierarchy can now use the numpy module directly:

# content of numpy.py
def arange():
    """
    >>> a = np.arange(10)
    >>> len(a)
    10
    """
    pass

Fixture name parameter

Fixtures now support a name parameter which allow them to be accessed by a different name than the function which defines it:

@pytest.fixture(name='value')
def fixture_value():
    return 10

def test_foo(value):
    assert value == 10

This solves the problem where the function argument shadows the argument name, which annoys pylint and might cause bugs if one forgets to pull a fixture into a test function.

Disable capture within a test

capsys and capfd fixtures now support a disabled context-manager method which can be used to disable output capture temporarily within a test or fixture:

def test_disabling_capturing(capsys):
    print('this output is captured')
    with capsys.disabled():
        print('output not captured, going directly to sys.stdout')
    print('this output is also captured')

pytest.raises: regular expressions and custom messages

The ExceptionInfo.match method can be used to check exception messages using regular expressions, similar to TestCase.assertRaisesRegexp method from unittest:

import pytest

def myfunc():
    raise ValueError("Exception 123 raised")

def test_match():
    with pytest.raises(ValueError) as excinfo:
        myfunc()
    excinfo.match(r'.* 123 .*')

The regexp parameter is matched with the re.search function.

Also, the context manager form accepts a message keyword parameter to raise a custom message when the expected exception does not occur:

def check_input(val):
    if input == "b":
        raise KeyError

@pytest.mark.parametrize('val', ["a", "b", "c"])
def test_inputs(val):
    with pytest.raises(KeyError, message="Key error not raised for {}".format(val)):
        check_input(val)

New hooks

pytest-3.0 adds new hooks, useful both for plugin authors and local conftest.py plugins:

  • pytest_fixture_setup(fixturedef, request): executes the fixture setup;
  • pytest_fixture_post_finalizer(fixturedef): called after the fixture's finalizer and has access to the fixture's result cache;
  • pytest_make_parametrize_id(config, val): allow plugins to provide user-friendly strings for custom types to be used by @pytest.mark.parametrize calls;

Complete documentation for all pytest hooks can be found in the hooks documentation.

Parametrize changes

Parametrize ids can now be set to None, in which case the automatically generated id will be used:

import pytest
@pytest.mark.parametrize(("a,b"), [(1,1), (1,1), (1,2)],
                         ids=["basic", None, "advanced"])  
def test_function(a, b):
    assert a == b

This will result in the following tests:

  • test_function[basic]
  • test_function[1-1]
  • test_function[advanced]

New command line options

  • --fixtures-per-test: Shows which fixtures are being used for each selected test item. Features doc strings of fixtures by default. Can also show where fixtures are defined if combined with -v.

  • --setup-show: Performs normal test execution and additionally shows the setup and teardown of fixtures.

  • --setup-plan: Performs normal collection and reports the potential setups and teardowns, but does not execute fixtures and tests.

  • --setup-only: Performs normal collection, executes setup and teardown of fixtures and reports them.

  • --override-ini/-o: Overrides values from the configuration file (pytest.ini). For example, "-o xfail_strict=True" will make all xfail markers act as strict, failing the test suite if they exit with XPASS status.

  • --pdbcls: Allow passing a custom debugger class that should be used together with the --pdb option. Syntax is in the form <module>:<class:, for example --pdbcls=IPython.core.debugger:Pdb.

  • --doctest-report: Changes the diff output format for doctests, can be one of: none, udiff, cdiff, ndiff or only_first_failure.

Assert reinterpretation is gone

Assertion reinterpretation was the preferred method to display better assertion messages before assertion rewriting was implemented. Assertion rewriting is safer because there is no risk of introducing subtle errors because of assertions with side-effects. The --assert=reinterp option has also been removed.

See this excellent blog post by Benjamin Peterson for a great overview of the assertion-rewriting mechanism.

In addition to this change, assertion rewriting is now also applied automatically in conftest.py files and plugins installed using setuptools.

Feature Deprecation

The pytest team took the opportunity to discuss how to handle feature deprecation during the pytest 2016 sprint, intending to minimize impact on the users as well as slowly fade-out some outdated features. It was decided:

Internal pytest-warnings now are displayed by default

Not showing them by default is confusing to users, as most people don't know how to display them in the first place. Also, it will help wider adoption of new ways of using pytest's features.

Deadline for deprecated features: new major version

Following semantic versioning, the pytest team will add deprecation warnings to features which are considered obsolete because there are better alternatives or usage is low while maintenance burden on the team is high. Deprecated features will only be removed on the next major release, for example pytest-4.0, giving users and plugin authors ample time to update.

Deprecated features

The following features have been considered officially deprecated in pytest-3.0, emitting pytest-warnings when used and scheduled for removal in 4.0:

  • yield-based tests; use @pytest.mark.parametrize instead;
  • pytest_funcarg__ prefix to declare fixtures; use @pytest.fixture decorator instead;
  • --resultlog: this option was rarely used and there are more modern alternatives;

Small improvements and bug-fixes

As usual, this release includes a lot of small improvements and bug fixes. Make sure to checkout the full CHANGELOG for the complete list.