I think of pytest as the run-anything, no boilerplate, no required api, use-this-unless-you-have-a-reason-not-to test framework.
This is really where testing gets fun.
As with previous intro’s on this site, I’ll run through an overview, then a simple example, then throw pytest at my markdown.py project. I’ll also cover fixtures, test discovery, and running unittests with pytest.

Contents

• No boilerplate, no required api
• pytest example
• Running pytest
• pytest fixtures
• Testing markdown.py
• Test discovery
• Running unittests from pytest
• Running doctests from pytest
• More pytest info (links)
• Examples on github
• Next

No boilerplate, no required api

The doctest
and unittest both come with Python.
They are pretty powerful on their own, and I think you should at least know about those frameworks, and learn how to run them at least on some toy examples, as it gives you a mental framework to view other test frameworks.

With unittest, you a very basic test file might look like this:

import unittest
from unnecessary_math import multiply

class TestUM(unittest.TestCase):

    def test_numbers_3_4(self):
        self.assertEqual( multiply(3,4), 12)

The style of deriving from unittest.TestCase is something unittest shares with it’s xUnit counterparts like JUnit.

I don’t want to get into the history of xUnit style frameworks. However, it’s informative to know that inheritance is quite important in some languages to get the test framework to work right.

But this is Python. We have very powerful introspection and runtime capabilities, and very little information hiding. Pytest takes advantage of this.

An identical test as above could look like this if we remove the boilerplate:

from unnecessary_math import multiply

def test_numbers_3_4():
    assert( multiply(3,4) == 12 )

Yep, three lines of code. (Four, if you include the blank line.)
There is no need to import unnittest.
There is no need to derive from TestCase.
There is no need to for special self.assertEqual(), since we can use Python’s built in assert statement.

This works in pytest. Once you start writing tests like this, you won’t want to go back.

However, you may have a bunch of tests already written for doctest or unittest.
Pytest can be used to run doctests and unittests.
It also claims to support some twisted trial tests (although I haven’t tried this).

You can extend pytest using plugins you pull from the web, or write yourself.
I’m not going to cover plugins in this article, but I’m sure I’ll get into it in a future article.

You will sometimes see pytest referred to as py.test.
I use this convention:
pytest : the project
py.test : the command line tool that runs pytest
I’m not sure if that’s 100% accurate according to how the folks at pytest.org use the terms.

pytest example

Using the same unnecessary_math.py module that I wrote in the
doctest intro,
this is some example test code to test the ‘multiply’ function.

from unnecessary_math import multiply

def test_numbers_3_4():
    assert multiply(3,4) == 12 

def test_strings_a_3():
    assert multiply('a',3) == 'aaa' 

Running pytest

To run pytest, the following two calls are identical:

python -m pytest test_um_pytest.py
py.test test_um_pytest.py

And with verbose:

python -m pytest -v test_um_pytest.py
py.test -v test_um_pytest.py

I’ll use py.test, as it’s shorter to type.

Here’s an example run both with and without verbose:

> py.test test_um_pytest.py
============================= test session starts ==============================
platform win32 -- Python 2.7.3 -- pytest-2.2.4
collecting ... collected 2 items

test_um_pytest.py ..

=========================== 2 passed in 0.05 seconds ===========================


> py.test -v test_um_pytest.py
============================= test session starts ==============================
platform win32 -- Python 2.7.3 -- pytest-2.2.4 -- C:\python27\python.exe
collecting ... collected 2 items

test_um_pytest.py:12: test_numbers_3_4 PASSED
test_um_pytest.py:15: test_strings_a_3 PASSED

=========================== 2 passed in 0.02 seconds ===========================

pytest fixtures

Although unittest does allow us to have setup and teardown, pytest extends this quite a bit.
We can add specific code to run:

• at the beginning and end of a module of test code (setup\_module/teardown\_module)
• at the beginning and end of a class of test methods (setup\_class/teardown\_class)
• alternate style of the class level fixtures (setup/teardown)
• before and after a test function call (setup\_function/teardown\_function)
• before and after a test method call (setup\_method/teardown\_method)

I’ve modified our simple test code with some fixture calls, and added some print statements so that we can see what’s going on.
Here’s the code:

from unnecessary_math import multiply

def setup_module(module):
    print ("setup_module      module:%s" % module.__name__)

def teardown_module(module):
    print ("teardown_module   module:%s" % module.__name__)

def setup_function(function):
    print ("setup_function    function:%s" % function.__name__)

def teardown_function(function):
    print ("teardown_function function:%s" % function.__name__)

def test_numbers_3_4():
    print 'test_numbers_3_4  <============================ actual test code'
    assert multiply(3,4) == 12 

def test_strings_a_3():
    print 'test_strings_a_3  <============================ actual test code'
    assert multiply('a',3) == 'aaa' 


class TestUM:

    def setup(self):
        print ("setup             class:TestStuff")

    def teardown(self):
        print ("teardown          class:TestStuff")

    def teardown_class(cls):
        print ("teardown_class    class:%s" % cls.__name__)

    def setup_class(cls):
        print ("setup_class       class:%s" % cls.__name__)

    def teardown_class(cls):
        print ("teardown_class    class:%s" % cls.__name__)

    def setup_method(self, method):
        print ("setup_method      method:%s" % method.__name__)

    def teardown_method(self, method):
        print ("teardown_method   method:%s" % method.__name__)

    def test_numbers_5_6(self):
        print 'test_numbers_5_6  <============================ actual test code'
        assert multiply(5,6) == 30 

    def test_strings_b_2(self):
        print 'test_strings_b_2  <============================ actual test code'
        assert multiply('b',2) == 'bb'

To see it in action, I’ll use the -s option, which turns off output capture.
This will show the order of the different fixture calls.

> py.test -s test_um_pytest_fixtures.py
============================= test session starts ==============================
platform win32 -- Python 2.7.3 -- pytest-2.2.4
collecting ... collected 4 items

test_um_pytest_fixtures.py ....

=========================== 4 passed in 0.07 seconds ===========================
setup_module      module:test_um_pytest_fixtures
setup_function    function:test_numbers_3_4
test_numbers_3_4  <============================ actual test code
teardown_function function:test_numbers_3_4
setup_function    function:test_strings_a_3
test_strings_a_3  <============================ actual test code
teardown_function function:test_strings_a_3
setup_class       class:TestUM
setup_method      method:test_numbers_5_6
setup             class:TestStuff
test_numbers_5_6  <============================ actual test code
teardown          class:TestStuff
teardown_method   method:test_numbers_5_6
setup_method      method:test_strings_b_2
setup             class:TestStuff
test_strings_b_2  <============================ actual test code
teardown          class:TestStuff
teardown_method   method:test_strings_b_2
teardown_class    class:TestUM
teardown_module   module:test_um_pytest_fixtures

Testing markdown.py

The test code to test markdown.py is going to look a lot like the unittest version, but without the boilerplate.
I’m also using an API adapter introduced in a previous post.
Here’s the code to use pytest to test markdown.py:

from markdown_adapter import run_markdown

def test_non_marked_lines():
    print ('in test_non_marked_lines')
    assert run_markdown('this line has no special handling') == \
            '<p>this line has no special handling</p>'

def test_em():
    print ('in test_em')
    assert run_markdown('*this should be wrapped in em tags*') == \
            '<p><em>this should be wrapped in em tags</em></p>'

def test_strong():
    print ('in test_strong')
    assert run_markdown('**this should be wrapped in strong tags**') == \
            '<p><strong>this should be wrapped in strong tags</strong></p>'

And here’s the output:

> py.test test_markdown_pytest.py
============================= test session starts ==============================
platform win32 -- Python 2.7.3 -- pytest-2.2.4
collecting ... collected 3 items

test_markdown_pytest.py F.F

=================================== FAILURES ===================================
____________________________ test_non_marked_lines _____________________________

    def test_non_marked_lines():
        print ('in test_non_marked_lines')
>       assert run_markdown('this line has no special handling') ==
                '<p>this line has no special handling</p>'
E       assert 'this line ha...cial handling' == '<p>this line ... handling</p>'
E         - this line has no special handling
E         + <p>this line has no special handling</p>
E         ? +++                                 ++++

test_markdown_pytest.py:14: AssertionError
------------------------------- Captured stdout --------------------------------
in test_non_marked_lines
_________________________________ test_strong __________________________________

    def test_strong():
        print ('in test_strong')
>       assert run_markdown('**this should be wrapped in strong tags**') ==
                '<p><strong>this should be wrapped in strong tags</strong></p>'
E       assert '**this shoul...strong tags**' == '<p><strong>th...</strong></p>'
E         - **this should be wrapped in strong tags**
E         + <p><strong>this should be wrapped in strong tags</strong></p>

test_markdown_pytest.py:24: AssertionError
------------------------------- Captured stdout --------------------------------
in test_strong
====================== 2 failed, 1 passed in 0.30 seconds ======================

You’ll notice that all of them are failing. This is on purpose, since I haven’t implemented any real markdown code yet.
However, the formatting of the output is quite nice.
It’s quite easy to see why the test is failing.

Test discovery

The unittest module comes with a ‘discovery’ option.
Discovery is just built in to pytest.
Test discovery was used in my examples to find tests within a specified module.
However, pytest can find tests residing in multiple modules, and multiple packages, and even find unittests and doctests.
To be honest, I haven’t memorized the discovery rules.
I just try to do this, and at seems to work nicely:

• Name my test modules/files starting with ‘test_’.
• Name my test functions starting with ‘test_’.
• Name my test classes starting with ‘Test’.
• Name my test methods starting with ‘test_’.
• Make sure all packages with test code have an ‘init.py’ file.

If I do all of that, pytest seems to find all my code nicely.
If you are doing something else, and are having trouble getting pytest to see your test code,
then take a look at the pytest discovery documentation.

Running unittests from pytest

To show how pytest handles unittests, here’s a sample run of pytest on the simple unittests I wrote in the unittest introduction:

> py.test test_um_unittest.py
============================= test session starts ==============================
platform win32 -- Python 2.7.3 -- pytest-2.2.4
collecting ... collected 2 items

test_um_unittest.py ..

=========================== 2 passed in 0.07 seconds ===========================
> py.test -v test_um_unittest.py
============================= test session starts ==============================
platform win32 -- Python 2.7.3 -- pytest-2.2.4 -- C:\python27\python.exe
collecting ... collected 2 items

test_um_unittest.py:15: TestUM.test_numbers_3_4 PASSED
test_um_unittest.py:18: TestUM.test_strings_a_3 PASSED

=========================== 2 passed in 0.06 seconds ===========================

As you can see, I didn’t provide any extra options, pytest finds unittests automatically.

Running doctests from pytest

You can run some doctests from pytest, according to the documentation.
However, with my examples of putting doctests in text files, I can’t figure out a way to get pytest to run them.

I’ve tried several attempts, and keep getting into import error problems:

> py.test --doctest-modules test_unnecessary_math.txt
============================= test session starts ==============================
platform win32 -- Python 2.7.3 -- pytest-2.2.4
collecting ... collected 1 items

test_unnecessary_math.txt F

=================================== FAILURES ===================================
__________________________________ [doctest] ___________________________________
001 This is a doctest based regression suite for unnecessary_math.py
002 Each '>>>' line is run as if in a python shell, and counts as a test.
003 The next line, if not '>>>' is the expected output of the previous line.
004 If anything doesn't match exactly (including trailing spaces), the test fails.
005
006 >>> from unnecessary_math import multiply
UNEXPECTED EXCEPTION: ImportError('No module named unnecessary_math',)
Traceback (most recent call last):

  File "C:\python27\lib\doctest.py", line 1289, in __run
    compileflags, 1) in test.globs

  File "<doctest test_unnecessary_math.txt[0]>", line 1, in <module>

ImportError: No module named unnecessary_math

E:\python_notes\repo\markdown.py-dev\simple_example\test_unnecessary_math.txt:6: UnexpectedException
=========================== 1 failed in 0.06 seconds ===========================

If anyone out there knows what I’m doing wrong, please let me know.
Thanks in advance.

More pytest info (links)

• pytest.org – main site, great info
• pytest talks and tutorials

Examples on github

All of the examples here are available in the markdown.py project on github.

Next

In the next post, I’ll throw nose at the sampe problems.