pytest blog

What's new in pytest 3.3

nicoddemus — Mon, 27 Nov 2017 18:00:00 +0100

This blog post provides a short overview of some of the major features and changes in pytest 3.3. See the CHANGELOG for the complete list.

Python 2.6 and 3.3 support dropped

Pytest 3.3 and onwards will no longer support Python 2.6 and 3.3. Those Python versions are EOL for some time now and incur maintenance and compatibility costs on the pytest core team, and following up with the rest of the community we decided that they will no longer be supported starting with this version. Users using those versions and with a modern enough pip should not be affected, otherwise they should pin pytest to < 3.3.

Logging capture

Pytest now captures and displays output from the standard logging module. The user can control the logging level to be captured by specifying options in pytest.ini, the command line and also during individual tests using markers. Also, a caplog fixture is available that enables users to test the captured log during specific tests (similar to capsys for example). For more information, please see the logging docs. This feature was introduced by merging the popular pytest-catchlog plugin, thanks to Thomas Hisch. Be advised that with this merge the backward compatibility interface with the defunct pytest-capturelog has been dropped.

Progress display

The pytest output now includes a percentage indicator. Here's the partial output of pytest's own test suite:

============================= test session starts =============================
platform win32 -- Python 3.6.3, pytest-3.3.0, py-1.5.2, pluggy-0.6.0
rootdir: C:\Users\appveyor\AppData\Local\Temp\1\devpi-test0\targz\pytest-3.3.0, inifile: tox.ini
plugins: hypothesis-3.38.5
collected 1985 items

testing\acceptance_test.py ...............s............................. [  2%]
...x.............                                                        [  3%]
testing\deprecated_test.py ........                                      [  3%]
testing\test_argcomplete.py ss                                           [  3%]
testing\test_assertion.py .............................................. [  5%]
...........................s.......                                      [  7%]
testing\test_assertrewrite.py ......................................s..s [  9%]
sss...........                                                           [ 10%]

The output can be changed back to its former style by configuring your pytest.ini:

[pytest]
console_output_style=classic

New capfdbinary and capsysbinary fixtures

Those two new fixtures return their contents as bytes instead of str, making them suitable to interact with programs that write binary data to stdout/stderr.

pytest.skip() at module level

pytest.skip in 3.0 was changed so it could not be used at the module level; it was a common mistake to use it as a decorator in a test function, when the user should have used pytest.mark.skip instead, which would cause the whole module to be skipped.

Some users missed this functionality, so allow_module_level can be passed to pytest.skip to skip an entire module:

import pytest

pytest.skip('this whole module is TODO', allow_module_level=True)

New dependencies

Pytest now depends on the following modules:

This change should not affect users.

Improvements and bugfixes

As usual, this release includes a lot of small improvements and bug fixes. Read the full CHANGELOG for the complete list.

What's new in pytest 3.0

nicoddemus — Tue, 12 Jul 2016 12:00:00 +0200

This blog post provides a short overview of some of the major features and changes in pytest 3.0. See the CHANGELOG for the complete list.

Additional command alias: `pytest`

pytest came into existence as part of the py library, providing a tool called py.test.

Even after pytest was moved to a separate project, the py.test name for the command-line tool was kept to preserve backward compatibility with existing scripts and tools.

In pytest-3.0 the command-line tool is now called pytest, which is easier to type and prevents some common misunderstandings when tools with the same name are installed in the system.

Note that py.test still works.

`approx()`: function for comparing floating-point numbers

The approx function makes it easy to perform floating-point comparisons using a syntax that's as intuitive and close to pytest's philosophy:

from pytest import approx

def test_similar():
    v = 0.1
    assert (v + 0.2) == approx(0.3)

`yield` statements in `@pytest.fixture`

Fixtures marked with @pytest.fixture can now use yield statements to provide values for test functions exactly like fixtures marked with @pytest.yield_fixture decorator:

import smtplib
import pytest

@pytest.fixture(scope="module")
def smtp():
    smtp = smtplib.SMTP("smtp.gmail.com")
    yield smtp
    smtp.close()

This makes it easier to change an existing fixture that uses return to use yield syntax if teardown code is needed later. Also, many users consider this style more clearer and natural than the previous method of registering a finalizer function using request.addfinalizer because the flow of the test is more explicit. Consider:

@pytest.fixture(scope="module")
def smtp(request):
    smtp = smtplib.SMTP("smtp.gmail.com")
    request.addfinalizer(smtp.close)
    return smtp

This change renders @pytest.yield_fixture deprecated and makes @pytest.fixture with yield statements the recommended way to write fixtures which require teardown code.

Note that registering finalizers in request is still supported and there's no intention of removing it in future pytest versions.

`doctest_namespace` fixture

The doctest_namespace fixture can be used to inject items into the namespace in which doctests run. It is intended to be used by auto-use fixtures to provide names to doctests.

doctest_namespace is a standard dict object:

# content of conftest.py
import numpy
@pytest.fixture(autouse=True)
def add_np(doctest_namespace):
    doctest_namespace['np'] = numpy

Doctests below the conftest.py file in the directory hierarchy can now use the numpy module directly:

# content of numpy.py
def arange():
    """
    >>> a = np.arange(10)
    >>> len(a)
    10
    """
    pass

Fixture `name` parameter

Fixtures now support a name parameter which allow them to be accessed by a different name than the function which defines it:

@pytest.fixture(name='value')
def fixture_value():
    return 10

def test_foo(value):
    assert value == 10

This solves the problem where the function argument shadows the argument name, which annoys pylint and might cause bugs if one forgets to pull a fixture into a test function.

Disable capture within a test

capsys and capfd fixtures now support a disabled context-manager method which can be used to disable output capture temporarily within a test or fixture:

def test_disabling_capturing(capsys):
    print('this output is captured')
    with capsys.disabled():
        print('output not captured, going directly to sys.stdout')
    print('this output is also captured')

`pytest.raises`: regular expressions and custom messages

The ExceptionInfo.match method can be used to check exception messages using regular expressions, similar to TestCase.assertRaisesRegexp method from unittest:

import pytest

def myfunc():
    raise ValueError("Exception 123 raised")

def test_match():
    with pytest.raises(ValueError) as excinfo:
        myfunc()
    excinfo.match(r'.* 123 .*')

The regexp parameter is matched with the re.search function.

Also, the context manager form accepts a message keyword parameter to raise a custom message when the expected exception does not occur:

def check_input(val):
    if input == "b":
        raise KeyError

@pytest.mark.parametrize('val', ["a", "b", "c"])
def test_inputs(val):
    with pytest.raises(KeyError, message="Key error not raised for {}".format(val)):
        check_input(val)

New hooks

pytest-3.0 adds new hooks, useful both for plugin authors and local conftest.py plugins:

pytest_fixture_setup(fixturedef, request): executes the fixture setup;
pytest_fixture_post_finalizer(fixturedef): called after the fixture's finalizer and has access to the fixture's result cache;
pytest_make_parametrize_id(config, val): allow plugins to provide user-friendly strings for custom types to be used by @pytest.mark.parametrize calls;

Complete documentation for all pytest hooks can be found in the hooks documentation.

Parametrize changes

Parametrize ids can now be set to None, in which case the automatically generated id will be used:

import pytest
@pytest.mark.parametrize(("a,b"), [(1,1), (1,1), (1,2)],
                         ids=["basic", None, "advanced"])  
def test_function(a, b):
    assert a == b

This will result in the following tests:

test_function[basic]
test_function[1-1]
test_function[advanced]

New command line options

--fixtures-per-test: Shows which fixtures are being used for each selected test item. Features doc strings of fixtures by default. Can also show where fixtures are defined if combined with -v.
--setup-show: Performs normal test execution and additionally shows the setup and teardown of fixtures.
--setup-plan: Performs normal collection and reports the potential setups and teardowns, but does not execute fixtures and tests.
--setup-only: Performs normal collection, executes setup and teardown of fixtures and reports them.
--override-ini/-o: Overrides values from the configuration file (pytest.ini). For example, "-o xfail_strict=True" will make all xfail markers act as strict, failing the test suite if they exit with XPASS status.
--pdbcls: Allow passing a custom debugger class that should be used together with the --pdb option. Syntax is in the form <module>:<class:, for example --pdbcls=IPython.core.debugger:Pdb.
--doctest-report: Changes the diff output format for doctests, can be one of: none, udiff, cdiff, ndiff or only_first_failure.

Assert reinterpretation is gone

Assertion reinterpretation was the preferred method to display better assertion messages before assertion rewriting was implemented. Assertion rewriting is safer because there is no risk of introducing subtle errors because of assertions with side-effects. The --assert=reinterp option has also been removed.

See this excellent blog post by Benjamin Peterson for a great overview of the assertion-rewriting mechanism.

In addition to this change, assertion rewriting is now also applied automatically in conftest.py files and plugins installed using setuptools.

Feature Deprecation

The pytest team took the opportunity to discuss how to handle feature deprecation during the pytest 2016 sprint, intending to minimize impact on the users as well as slowly fade-out some outdated features. It was decided:

Internal pytest-warnings now are displayed by default

Not showing them by default is confusing to users, as most people don't know how to display them in the first place. Also, it will help wider adoption of new ways of using pytest's features.

Deadline for deprecated features: new major version

Following semantic versioning, the pytest team will add deprecation warnings to features which are considered obsolete because there are better alternatives or usage is low while maintenance burden on the team is high. Deprecated features will only be removed on the next major release, for example pytest-4.0, giving users and plugin authors ample time to update.

Deprecated features

The following features have been considered officially deprecated in pytest-3.0, emitting pytest-warnings when used and scheduled for removal in 4.0:

yield-based tests; use @pytest.mark.parametrize instead;
pytest_funcarg__ prefix to declare fixtures; use @pytest.fixture decorator instead;
--resultlog: this option was rarely used and there are more modern alternatives;

Small improvements and bug-fixes

As usual, this release includes a lot of small improvements and bug fixes. Make sure to checkout the full CHANGELOG for the complete list.

pytest development sprint

pfctdayelise — Thu, 07 Jul 2016 09:00:00 +0200

We have just wrapped up the pytest development sprint, which was held 20-25 June 2016 in Freiburg, Germany. Other sprints have taken place at Python conferences, but this was the first dedicated standalone event. We had 27 participants from five continents!

It was funded by an indiegogo campaign which raised over US$12,000 with nearly 100 backers. Several companies generously donated $300 or more:

Participants travelled from around Europe and the UK, the US, Brazil, Australia and China.

Work started or finished included the following.

Discussions:

what to include in pytest 3.0
advanced intro sessions on plugin mechanics (pluggy) and the fixture system implementation
tox roadmap
Started to conceptualize generalized package builds and virtualenv creation
process for removing deprecated features
proposal for combining parametrize with fixtures
new 'fromcontext'/'invocation' scope for fixtures and some initial prototyping
lightning talks

Major features:

dropping pytest assert reinterpret code
rename binary from 'py.test' to 'pytest' (but don't worry, py.test will still work)
documentation restructuring
--setup-only and --setup-plan flags which show how fixtures are initialised without actually running the tests
Adding -o commandline option to override .ini config values

Other features:

pytest-qt features - improvements towards a 2.0 release
pytest-bdd enhancements
pytest-django: triaged issues in general and made some progress especially with regard to better handle DB access/setup.
pytest-html enhancements
pytest-factoryboy enhancements
pytest-selenium enhancements
pytest-repeat enhancements
pytest-environment research
cookiecutter-pytest-plugin improvements
4 projects submitted to pytest-dev organisation

Bugs fixed/in progress:

In total 35 pull requests were merged to pytest, and at least 12 to tox, not to mention many others to plugins.

(It is worth noting that this is not an exhaustive list of all features that will go into pytest 3.0; a separate post will the full release notes will be made once the release is done.)

We also spent some time designing the stickers and t-shirts. Surveys went out to those who backed the fundraising campaign last week and the goods will be shipped very soon.

Thank you so much to everyone who supported the sprint. We have greatly improved our knowledge of pytest internals, expanded the pytest contributor community, and we're looking forward to bringing you pytest 3.0 very soon (aiming for before EuroPython)!

Blog 1.0

pfctdayelise — Fri, 24 Jun 2016 12:00:00 +0200

Welcome to the launch of the official pytest blog!

Pytest has grown from strength to strength over the past few years, and it's overdue that we have an official blog for releases and other project announcements.

There is now a pytest planet, which is a blog aggregator. If you have your own blog and can tag some entries as 'pytest', we invite you to add your RSS feed there.

There is also the pytest tricks blog, so if you have a small demonstration of how to use a cool pytest feature, that might be the place for it. If you don't have your own blog, you can submit a guest post to either pytest-tricks or the main pytest blog.

It might be obvious that we're not the most gifted web designers, so we definitely welcome improvements at the Github repository. This blog was first talked about at EuroPython 2015, so 11 months later, we thought it was more important to get it out there than to perfect the theme. Thanks to Javier Gonel and Andreas Pelme for their work to get it started.

In upcoming months there are lots of things happening, with wrapping up the pytest development sprint and preparing for pytest 3.0. Stay tuned!

We have a blog

pelme — Sat, 25 Jul 2015 22:00:00 +0200

And we have a blog.

This was a nice hacking day! A photo from the EuroPython 2015 sprints in Bilbao.