pytest introduction

I think of pytest as the run-anything, no boilerplate, no required api, use-this-unless-you-have-a-reason-not-to test framework.
This is really where testing gets fun.
As with previous intro’s on this site, I’ll run through an overview, then a simple example, then throw pytest at my markdown.py project. I’ll also cover fixtures, test discovery, and running unittests with pytest.

Contents

No boilerplate, no required api

The doctest
and unittest both come with Python.
They are pretty powerful on their own, and I think you should at least know about those frameworks, and learn how to run them at least on some toy examples, as it gives you a mental framework to view other test frameworks.

With unittest, you a very basic test file might look like this:

The style of deriving from unittest.TestCase is something unittest shares with it’s xUnit counterparts like JUnit.

I don’t want to get into the history of xUnit style frameworks. However, it’s informative to know that inheritance is quite important in some languages to get the test framework to work right.

But this is Python. We have very powerful introspection and runtime capabilities, and very little information hiding. Pytest takes advantage of this.

An identical test as above could look like this if we remove the boilerplate:

Yep, three lines of code. (Four, if you include the blank line.)
There is no need to import unnittest.
There is no need to derive from TestCase.
There is no need to for special self.assertEqual(), since we can use Python’s built in assert statement.

This works in pytest. Once you start writing tests like this, you won’t want to go back.

However, you may have a bunch of tests already written for doctest or unittest.
Pytest can be used to run doctests and unittests.
It also claims to support some twisted trial tests (although I haven’t tried this).

You can extend pytest using plugins you pull from the web, or write yourself.
I’m not going to cover plugins in this article, but I’m sure I’ll get into it in a future article.

You will sometimes see pytest referred to as py.test.
I use this convention:
pytest : the project
py.test : the command line tool that runs pytest
I’m not sure if that’s 100% accurate according to how the folks at pytest.org use the terms.

pytest example

Using the same unnecessary_math.py module that I wrote in the
doctest intro,
this is some example test code to test the ‘multiply’ function.

Running pytest

To run pytest, the following two calls are identical:

And with verbose:

I’ll use py.test, as it’s shorter to type.

Here’s an example run both with and without verbose:

pytest fixtures

Although unittest does allow us to have setup and teardown, pytest extends this quite a bit.
We can add specific code to run:

  • at the beginning and end of a module of test code (setup_module/teardown_module)
  • at the beginning and end of a class of test methods (setup_class/teardown_class)
  • alternate style of the class level fixtures (setup/teardown)
  • before and after a test function call (setup_function/teardown_function)
  • before and after a test method call (setup_method/teardown_method)

We can also use pytest style fixtures, which are covered in pytest fixtures nuts and bolts.

I’ve modified our simple test code with some fixture calls, and added some print statements so that we can see what’s going on.
Here’s the code:

To see it in action, I’ll use the -s option, which turns off output capture.
This will show the order of the different fixture calls.

Testing markdown.py

The test code to test markdown.py is going to look a lot like the unittest version, but without the boilerplate.
I’m also using an API adapter introduced in a previous post.
Here’s the code to use pytest to test markdown.py:

And here’s the output:

You’ll notice that all of them are failing. This is on purpose, since I haven’t implemented any real markdown code yet.
However, the formatting of the output is quite nice.
It’s quite easy to see why the test is failing.

Test discovery

The unittest module comes with a ‘discovery’ option.
Discovery is just built in to pytest.
Test discovery was used in my examples to find tests within a specified module.
However, pytest can find tests residing in multiple modules, and multiple packages, and even find unittests and doctests.
To be honest, I haven’t memorized the discovery rules.
I just try to do this, and at seems to work nicely:

  • Name my test modules/files starting with ‘test_’.
  • Name my test functions starting with ‘test_’.
  • Name my test classes starting with ‘Test’.
  • Name my test methods starting with ‘test_’.
  • Make sure all packages with test code have an ‘init.py’ file.

If I do all of that, pytest seems to find all my code nicely.
If you are doing something else, and are having trouble getting pytest to see your test code,
then take a look at the pytest discovery documentation.

Running unittests from pytest

To show how pytest handles unittests, here’s a sample run of pytest on the simple unittests I wrote in the unittest introduction:

As you can see, I didn’t provide any extra options, pytest finds unittests automatically.

Running doctests from pytest

You can run some doctests from pytest, according to the documentation.
However, with my examples of putting doctests in text files, I can’t figure out a way to get pytest to run them.

I’ve tried several attempts, and keep getting into import error problems:

If anyone out there knows what I’m doing wrong, please let me know.
Thanks in advance.

More pytest info (links)

Examples on github

All of the examples here are available in the markdown.py project on github.

Next

In the next post, I’ll throw nose at the sampe problems.

Comments

  1. says

    I find pytest very opinionated compared with nose tests. To break my print statement in code habits I’ve been shifting to test driven hacking which means shifting print statements and set_trace out of my code and into tests, but pytest appears to consume much more output by default (and all test output) and I assume threads/processes are consuming my break points. Maybe I’m *doing it wrong* but it makes nose the more flexible testing tool.

    • says

      Hmmm. Interesting point.
      I will have to take a look at the issue sometime and see if I can reproduce the difficulty.
      I’ll be taking a look at nose soon.
      Admittedly, the code and tests I’m presenting so far are trivial examples.
      Also, I haven’t yet delved into the details of either framework.

      But you definitely bring up a good topic, a reasonable workflow, and a valid concern. Thanks.

      • says

        I fired off a little quickly I think. Reading your article made me look back at the docs after commenting, and both those problems appear to be in the RTFM category. Nose provides less initial resistance but pytest looks like it rewards after a very modest investment of time and I’ll definitely give it another shot.

        One thing I’m interested in exploring is how well testing frameworks adjust to a more semantic testing workflow. I think there’s a bit of a gulf between doctests and testsuites that others are trying to fill with very formal Behaviour driven development test frameworks which I’m a little sceptical of but I understand the benefit of a readable test suite.

        Anyway, good work. There’s plenty of threads to mine in testing – I look forward to your coverage ;).

  2. says

    Good starting doc, thanks! I guess i’d like to link to it from pytest.org :)

    As to doctests not finding your app module: indeed, pytest does no do any syspath-inserting magic for doctests. If you make sure that your module is importable (e.g. via a “python setup.py develop” or via adding the directory to PYTHONPATH) then the doctest should work by default. The “–doctest-modules” flag is only neccessary if you want to run run docstrings in your python modules.

    @bbhaydon pytest by default captures on the “fd” file descriptor level. This means that if you start a subprocess without overridding stdout, you will see its output captured as well. If you want to restrict pytest to only do sys.stdout/stderr capturing, you can pass “–capture=sys”. If you want to run like this always you can add a line “addopts = –capture=sys” in a pytest.ini or setup.cfg [pytest] section. If you want to have no output capturing use “–capture=no” accordingly or the shortcut “-s”.

    • says

      Holger,
      Danke.
      A link from pytest.org would be pretty great. :)
      Thanks for the doctest information. I’ll take a look at the path suggestions when I get some time and try to get it to work. I figured it was some type of pilot error.

  3. Jeff Hinrichs says

    I am a big py.test fan. The mental load to write unittests negates its effectiveness in my workflow. I am forced into it when I use django (I believe you can run django tests with py.test, but I haven’t went for a look see yet.) Everything else that I write, I write in the succinct way of py.test.

    When it is easier to start writing tests, you tend to do more of it. *wink* The more testing you do, the more solid your code ends up being. py.test is more pythonic, imho, because it keeps the simple stuff, simple and makes the hard stuff possible.

    For instance, quick how do you test exceptions in UnitTests? Low tech way with py.test

    try:
    thing_that_rasises_typeerror()
    assert False
    except TypeError:
    assert True

    py.test has classier ways of doing this, and this is not an indictment of py.test, merely to show how fast it is to get a test written. No imports, Just straight forward python – 0 mental load thinking about your testing tool. Thought process stays on what your are testing, not time-sharing with thinking about your testing tool.

  4. recher says

    In your class TestUM :

    This function is defined twice :

    def teardown_class(cls):
    print (“teardown_class class:%s” % cls.__name__)

    Is it made on purpose, or is it a copypaste typo ?

  5. says

    Dude, I haven’t tried this, but I think that if you have your doctest files that will import your own modules do the following steps first (which actually count as 2 more “tests”):

    >>> import sys, os
    >>> sys.path.append(os.getcwd())

    or if your module is within some folder, say, folder x

    >>> sys.path.append(os.path.join(os.getcwd(), x))
    a
    Thanks for this blog, by the way. Helped me quickly get into the existing python testing frameworks. =)

Leave a Reply