doctest introduction

The doctest test framework is a python module that comes prepackaged with Python. This post covers the basics of how to put doctests in your code, and outside of your code, in a separate file. Then I’ll show how I’m using it to test markdown.py.

conceptual model of python doctest

This is from python.org:

The doctest module searches for pieces of text that look like interactive Python sessions,
and then executes those sessions to verify that they work exactly as shown. …

I like to think of doctest as if I’m actually sitting at a python interactive prompt, and typing stuff in. The doctest is a script that says “My session should look exactly like this. If it doesn’t something is wrong.”

Actually, I think some people do use it that way. They write some module, and then demonstrate how it works in an interactive shell, and copy/paste the session into a docstring as their doctests.

However, that doesn’t work so well in TDD, where I’ve not got the code working before I write the test. With TDD, I’ve really got to think about the exact output of something before it works.

When using doctest and TDD, it can end up getting rather iterative:

  1. Write some doctests
  2. Run the doctests to see that they fail
  3. Write some code that should make it pass
  4. If it still fails, examine the failure.
  5. If it is a false failure, and the doctest is just being to picky, then modify the doctest, possibly
    with doctest flags, then go to 2.
  6. If it is a real failure, fix the code, then go to 2.

I have found that some of the nitpicky aspects of doctest can be minimized with the use of an api adapter. I’ll be using an adapter in the markdown.py example in this post.

doctest example

Here is a simple module with one function in it, along with two doctests embedded in the docstring.

unnecessary_math.py:

running doctest

You run doctest like this:

The ‘-v’ means verbose. Verbose is real handy when testing your doctests, since doctest doesn’t output anything if all of the tests pass.

You can see in the first run that nothing prints out, since all tests pass.

doctests in a separate file from the code

One of the really cool features of doctest is the ability to put your doctests in a text file. This is especially useful for functional testing, since that allows you to use doctest to test even non-python interfaces.

For our simple math example, I can just put the same code from the docstring into a text file.

test_unnecessary_math.txt:

running doctests in separate file

Running doctest on a file is the same as running it on a module.

example with markdown.py

For markdown.py, I don’t want to include doctests in the code. Since I’m only testing the external CLI (through an adapter), I’m will be using the ‘doctests in a text file’ method.

I’m not going to write tests for the entire syntax right away. My first three tests will be for paragraphs, single asterisk em tags, and double asterisk strong tags.

test_markdown_doctest.txt:

Well, that’s simple enough. I’ve imported ‘run_markdown’ from my api adapter. Then I throw some example strings into the script and show what I expect to come out.

testing markdown.py

Here’s the output of running doctest on my text file.

And with verbose.

As you can see. Once you’ve convinced yourself that your tests are correct, the verbose setting doesn’t add much. You will get plenty of output without verbose if there are errors.

In my case, everything FAILED!!!. But that’s good, because I haven’t implemented anything real yet, I just have a stub.

more doctest info

All of the examples in this post are available in the github markdown.py project.The math example is in a folder called ‘simple_doctest_example’.

The python.org site has pretty good information about using doctest.
On that same page is the writeup on how to use text files for your doctests.

Doug Hellmann has a great writeup on doctest that I highly recommend.It’s called Testing through documentation and it covers many of the problems that you may run into including dealing with multiple lines, whitespace, unpredictable output, etc.

I will cover some of these aspects as I get further into the implementation and testing of markdown.py.

next

Next up, I’ll take a look at implementing the same tests using unittest, also sometimes referred to as PyUnit.

If you like this…

Get notified of new posts, delivered to your inbox. No spam. Not very frequent.

Comments

Leave a Reply