«
»

,

Python Package Structure with Unit Testing, Injection and Mocking

Saturday, February 6th, 2010 - 8:25 pm - Python

Python is a language that some people like and use.  I recently was given a small Python application and had to refactor it so that I could implement unit and functional tests.  It took me a while to find the resources and learn how to implement them, so I thought I’d write it up here.

* Please note that this is not a unit testing tutorial.  Those are plentiful and can be found elsewhere.  This describes setting up a professional Python project with a real package structure, injection and tests.

Python Project Structure

Python was conceived as a scripting language.  Unlike compiled languages like C or Java, Python files are designed to be directly runnable.

  • RunMe.java -p someargument < Not ok!
  • RunMe.py -p someargument < Ok!

This is convenient for small scripts, but doesn’t help applications.  Since you are intended to run .py files directly, Python projects should not have source (src) or binary (bin) folders.  Having a source folder makes a program difficult to run.  The root of the Python package structure should be the root of the project.

For the same reason, the test folder should not be separate.  It should be a package in the main project.  Since the root folder is the root package, you won’t have much choice.

Example

  • MyProject
    • __init__.py
    • model
      • __init__.py
      • user.py
    • service
      • __init__.py
      • user_service.py
      • ldap_user_service.py
    • test
      • __init__.py
      • unit
        • __init__.py
        • service
          • __init__.py
          • user_service_test.py
      • functional
        • __init__.py
        • service
          • __init__.py
          • user_service_test.py

If you’re new to Python, please note that those __init__.pys are required.  They tell Python which folders contain Python files.  This is Python’s way of allowing you to have other folders, such as config, that don’t contain source files.

Note

You should specify exactly one class per file.  It makes your code easier to find and read.

Injection with snake-guice

Injection is essential for unit tests.  In the example above, LDAPUserService may connect to an LDAP server to retrieve information, something we don’t want to happen in unit tests.

I’ve been using snake-guice and highly recommend it.

Install snake-guice, Mock and nose

If you don’t have it, grab a copy of easy install.  Then, $ sudo easy_install.py snake-guice Mock nose.  That installs packages from PyPI, the Python Package Index.  For those used to non-scripting languages, this may not seem like a great way to use external libraries.  Since Python is a scripting language, however, it’s the only way that makes sense.

Create a module and use it to instantiate your application

class ExampleModule:
  def configure(self, binder)
    binder.bind(UserService, to=LDAPUserService)
class ExampleRunner:
  user_service = None
 
  @inject(user_service=UserService)
  def __init__(self, user_service):
    self.user_service = user_service
 
  def main(user_service)
    self.user_service.login("name", "password")
injector = Injector(ExampleModule)
runner = injector.get_instance(ExampleRunner)

When get_instance is called, injector will use the ExampleModule to discover bindings.  Then, it will create an ExampleRunner, passing in arguments specified by @inject.

The setUp method of PyUnit tests should create an injector with a TestExampleModule, which should be configured to return mocks.

class TestExampleModule:
  def user_service = Mock()
 
  def configure(self, binder)
    binder.bind(UserService, to=user_service)
class ExampleTest(TestCase):
  application = None
 
  def setUp(self):
    Injector = Injector(TestExampleModule)
    self.application = injector.get_instance(ExampleApplication)
 
  def test_something(self):
    application.user_service.search.return_value = "expected"
    actual = application.user_service.search("any")
    self.assertEquals(expected, actual)

Since setUp is called before each test, each test method gets a fresh mock that it can configure any way it likes.

Similarly, a functional test module can be created that connects to a local or testing server.

Mocking with Mock

I recommend using Mock for mocking, which is why I had you install it earlier.  Be careful when searching for it because there appear to be two projects called Mock.

The Mock package is well documented.  Please go to the Mock site for more information.

Running Tests

For running tests, I recommend Nose.  To run your tests, open a command line and change to your root project folder.  Then, nosetests.  Nose will take care of adding the packages you installed earlier, snake-guice and Mock, to the Python path.  It will also find all tests, run them and report the results.  Handy!

$ cd ~/exampleproject
$ nosetests

To run only unit or functional tests, use the -w argument.

$ nosetests -w ./ ./test/unit
$ nosetests -w ./ ./test/functional

Is that all?

To those who have not used injection or mocks for testing, this may seem like a lot of work.  I hope you’ll try it, though, because it’s a one-time setup.  Once these good practices are in place, you’ll find that all of your code organization and testing becomes simple and almost automatic.  If I missed something or if you have any suggestions, please post below.


  1. James says:
    January 21, 2012 at 9:47 am

    Thanks for the great write-up. I’ve been curious about how to do injection/mocking with Python after doing a bit of Grails.

    Reply
  2. Arthur says:
    July 5, 2011 at 1:54 pm

    What about multi layered packaging? What structure would you recommend. For example

    + MyProject
    – some_script.py
    – __init__.py
    + model
    – __init__.py
    – some_other_script.py
    + model_specific_package
    – __init__.py
    – one_more_script
    + service
    – __init__.py
    – service_script.py

    how would you structure your test directory?

    Reply
    • Arthur says:
      July 5, 2011 at 1:57 pm

      ack, it got rid of my whitespace, but essentially:
      MyProject – Root
      model – in My Project (dir level 1)
      model_specific_package – in model (dir level 2)

      I’ve been having trouble with directories that have the same name making it impossible to import because the python interpreter gets confused

    • MrSqueezles says:
      July 7, 2011 at 5:29 am

      Hi Arthur.

      You are correct. Python has trouble with name conflicts. That’s caused by Python’s runtime and its coding standards. The problem you described would never happen in Java because classes are camel case and packages are lowercase and also because Java doesn’t allow imports of packages. In Python, it’s possible to “import model” where model is a package. Python allows package imports because of those annoying __init__.py files. The only workable solution is to not have name conflicts. I suggest simplifying the package structure by removing the model-specific package.

  3. Clarence Gutherie says:
    February 1, 2011 at 10:31 am

    Howdy would you mind letting me know which hosting company you’re utilizing? I’ve loaded your blog in 3 completely different web browsers and I must say this blog loads a lot faster then most. Can you suggest a good web hosting provider at a honest price? Kudos, I appreciate it!

    Reply
  4. David Stanek says:
    May 26, 2010 at 7:25 pm

    I enjoyed this post and just recommended it to a few friends. I’d be interesting in hearing more about how you use snake-guice.

    Reply
    • MrSqueezles says:
      May 28, 2010 at 1:16 pm

      Hi David.
      All of that business about modules is the snake-guice part. It’s very similar to Guice because it uses modules, bindings and code configuration. In this project, we only used it for testing support. We had a unit test module that mocked all HTTP calls. We had another for functional testing that connected to a local test server. The third is used in production to connect to real servers.

Leave a Reply