Unit testing

 Unit tests are not strictly a Plone 2.1 feature, but they are a very important "best practice", and are definitely something you should be familiar with. Plone uses unit testing extensively (don't try to check in code to Plone itself without unit tests, you will be burned at the stake), and we will introduce them briefly here. It's time to get religious.
 
The idea of unit testing is that as the complexity of a piece of software increases, it becomes harder to test. It is difficult to know if you cover all the cases when you simply test your product in the Plone UI, and as you continue working on your product, you will invariably break something you thought were working before.

The golden rules of unit testing are:

o Write at least one test for every feature

o Write the interface and/or stub methods first (if applicable), *then* write the test, make sure the test fails (because the code isn't written yet!)

o Only when you have a failing test are you allowed to implement your feature. The goal of every line of code you write should be to make a failing test pass. 

o When you find a bug, *don't fix it*...

o ...instead, write a failing test to demonstrate that the bug is there...

o ...and *then* you're allowed to fix the bug

No, we're not kidding. This may all seem cumbersome and unintuitive to you. You're wrong. Unit tests are:

1. The only way of even remotely convincing your customers and friends your code doesn't completely suck.

2. The only way of making sure (or at least being more confident) you don't break things without realising it.

3. The only way of making sure (or at least being more confident) you don't re-introduce bugs you thought you'd fixed.

4. Usually a way of saving time in the long run, because you know immediately when you break something, and you spend less time chasing down obscure bugs in code you wrote six months ago.

5. A useful way of writing and testing code in the same environment - you don't have to switch context to a browser and click around to test your newest feature - just run the tests!

Much like a good Scotch Whiskey, making unit tests pass is a good feeling, and it only gets better as time goes by and your test coverage increases. Plone has, at the time of writing, over 1500 passing tests. In fact, we need perhaps three times that, even for the existing code base.

Zope/Plone unit tests

  Unit testing works by setting up a sandbox (also called a test fixture) in which the test is run. Using PloneTestCase, this is basically an empty Zope instance with a single Plone site, containing a single member, with a default member folder. All tests are run in the same environment - that is, each time a test method is finished, the Zope transaction is aborted so that no matter how that test changed the state of the portal, the next test will be completely unaffected. You are free to do obscene things in tests - look at testMigrations.py in Plone. We delete things like 'portal_types' without so much as a twitch. Tests are executed in arbitrary (actually, alphabetical) order, and you should not rely on any interaction between tests at all.

Some basic rules of thumb for writing unit tests with PloneTestCase should be aware of:

o Write test first, don't put it off, and don't be lazy :)
 
o Write one test (i.e. one method) for each thing you want to test
 
o Keep related tests together (i.e. in the same class)
 
o Be pragmatic. If you want to test every combination of inputs and outputs you will probably go blue in the face, and the additional tests are unlikely to be of much value. Similarly, if a method is complicated, don't just test the basic case. This comes with experience, but in general, you should test common cases, edge cases and preferably cases in which the method or component is expected to fail (i.e. test that it fails as expected).
 
o Keep them simple. Don't try to be clever, don't over-generalise. When a test fails, you need to easily determine whether it is because the *test itself* is wrong, or the thing it is testing has a bug.
 
o Always run all tests before you check in your code (*especially* if you're not the original author/maintainer) and make sure you didn't break anything. Not doing this is a cardinal sin for the Plone core, and should be in your own environment, too.

Setting up your test environment

 Adding unit tests to your product is fairly simple. You must:
 
o Install PloneTestCase and its dependencies (see its INSTALL.txt. Note that ZopeTestCase, upon which PloneTestCase depends, ships with Zope 2.8 and later)

o Add a 'tests/' directory and copy the files 'runalltests.py' and 'framework.py' into it. You can find these in the folder 'RichDocument/tests', any many other packages.

o Add some test case classes, with test methods. To make this easier, you will normally define a test case base class to set up your test environment in a consistent manner. For RichDocument, you can find this in 'rdtc.py'. This is then used by 'testSetup.py', which contains the actual test methods. 

We're going to re-visit the RichDocument tests in a moment, but first let's see how to run unit tests.

Running tests
 
 There are three different ways in which you can run unit tests. The easiest one is to use 'zopectl' from your Zope instance root, with a command like::

  ./bin/zopectl test --libdir Products/RichDocument
  
Alternatively, you can go to the tests directory (e.g. 'Products/RichDocument/tests') and run a test or all tests directly with::

    python testSetup.py
    
or::

    python runalltests.py
    
In both cases, you may have to set the environment variables 'INSTANCE_HOME' and 'SOFTWARE_HOME'. The former should point to your Zope instance (the parent of your 'Products' folder), the latter should point to the python library directory where Zope is installed, e.g. '/usr/local/zope-2.8.4/lib/python'.
    
The second of the two commands above, 'runalltests.py', is equivalent to the 'zopectl' method above, but may be more convenient. The first, calling a test file directly, is quite useful - it allows you to run only a subset of tests. Since test execution can take quite a long time on larger projects, this is often a good way of testing only what you think is relevant. When the tests you think are relevant all pass, it's time to run all tests and make sure nothing else broke. (No, we don't care that you are writing your code in a totally different python module than what those other tests are supposed to test, and that they were all fine and good and all you changed was a docstring. Run the tests when you think you're done.)
  
When tests finish running, you will see a report like::

    ...
    Ran 18 tests in 6.463s

    OK

Rehearse a satisfied sigh as you read the line "OK", as opposed to seeing a count of failed tests. With time, this will be the little notifier that lets you go to bed, see your friends again or generally get back to real life with an 'svn commit'. 

If you're not so lucky, you may see::

    Ran 18 tests in 7.009s
    
    FAILED (failures=1, errors=1)
    
This means that there were 1 python error and 1 failed test during test execution. 

A python error means that some of your test code, or some code that was called by a test, raised an exception. This is bad, and you should fix it right away. 

A failed test means that your test was trying to assert something that turned out not to be true. This could be OK. It could mean you haven't written the code the test is testing yet (well done, you wrote the test first!), or that you don't yet know why it's failing. Sometimes you may be radically refactoring or rewriting parts of your code, and the tests will keep on failing until you're done. Incidentally, this is part of the reason why unit tests are so good - you can do that kind of stuff.

It's sometimes (not always - don't try this on Plone core unless you've been told it's OK by the release manager) acceptable to go to bed and check in a failing test if you are not in a position to know how to fix it. At least other developers will be aware of the problem and may be able to fix it. 

Writing your own unit tests

  Now that you know how to set up and run your tests, let's get back to how you actually write some. Unit testing is an art form that is best learnt by example. You are encouraged to look at the RichDocument unit tests, the Plone unit tests and any other unit tests you think may be relevant to see how they work and what sort of patterns they use.

Unit testing in Zope/Plone relies on a few naming introspections to reduce the burden on you the programmer. Basically:

o All test .py files must begin with the word 'test', as in 'testSetup.py'

o Inside the test files, you define test case classes. In each class, you can have any number of testing methods, which must also begin with the word 'test', as in 'testSkinLayersInstalled()'.

Let us look at at example. In 'rdtc.py', the base test case for RichDocument, we have::
  
  # These install (or fail) quietly
  ZopeTestCase.installProduct('CMFCore', quiet=1)
  
  ...
  
  ZopeTestCase.installProduct('kupu', quiet=1)
  
  # These must install cleanly
  ZopeTestCase.installProduct('RichDocument')
  
  PRODUCTS = ['RichDocument']
  PloneTestCase.setupPloneSite(products=PRODUCTS)
  
  class RichDocumentTestCase(PloneTestCase.PloneTestCase):
  
    ....
  
These lines register the standard products with ZopeTestCase, and sets up a Plone site with 'RichDocument' installed. Then, the base class for all other test case classes is defined. In this case, it does nothing but some boilerplate. In other cases, you may find it useful to put utility methods here that may be called by the tests.

In 'testSetup.py', the first (and to date only) unit test file for RichDocument, we then have::

  import os, sys
  if __name__ == '__main__':
      execfile(os.path.join(sys.path[0], 'framework.py'))

  from Products.RichDocument.tests import rdtc

  class TestInstallation(rdtc.RichDocumentTestCase):
      """Ensure product is properly installed"""

      def afterSetUp(self):
          self.css        = self.portal.portal_css
          self.kupu       = self.portal.kupu_library_tool
          self.skins      = self.portal.portal_skins
          self.types      = self.portal.portal_types
          self.factory    = self.portal.portal_factory
          self.workflow   = self.portal.portal_workflow
          self.properties = self.portal.portal_properties

          self.metaTypes = ('RichDocument', 'ImageAttachment', 'FileAttachment')

      def testSkinLayersInstalled(self):
          self.failUnless('RichDocument' in self.skins.objectIds())
          self.failUnless('attachment_widgets' in self.skins.objectIds())
        
      ...

  def test_suite():
      from unittest import TestSuite, makeSuite
      suite = TestSuite()
      suite.addTest(makeSuite(TestInstallation))
      ...
      return suite
      
  if  __name__ = '__main__':
      framework()

At the top, some python magic is included to make sure you can run this test in isolation by running 'python testSetup.py'. Then, we define the test class. Finally, the test suite is defined. You can add several test classes to each test suite. When you run a test suite (e.g. with 'python testSetup.py') all test methods in all test classes in the test suite in this file will be included.
  
Notice the line 'from unittest import ...'. This is importing test machinery from the standard Python unit-testing module, which ZopeTestCase uses. You can read more about the 'unittest' module "in the Python documentation":http://docs.python.org/lib/module-unittest.html.

Inside the 'TestInstallation' test class, the 'afterSetUp()' method is called immediately before *each* test, and can be used to set up some test data. This is frequently used to fetch tools, create dummy content etc. After a test is executed, the transaction will be rolled back, so it's normally not necessary to do much cleanup, but if you are interacting with something outside of Zope and need to clean up after each test, you can do this in a method called 'beforeTearDown()'.

You are free to add whatever helper methods you wish to your unit test class, but any method with a name starting with 'test...' will be executed as a test. Tests are usually written to be as concise (not to be confused with "obfuscated") as possible. You'll see several calls to methods like 'self.assertEqual()' or 'self.failUnless()'. These are the assertion methods that do the actual testing. If any of these fail, that test is counted as a failure and you'll get an ugly F in your test output.

Assertion and utility methods in the unit testing framework

  There are quite a few assertion methods, most of which do basically the same thing - check if something is True or False. Having a variety of names allows you to make your tests read the way you want. In fact, tests should be viewed as stories of how your code is expected to behave. It's often useful for other developers to read your tests to see what your actual code is capable of. The list of assertion methods can be found in the "Python documentation for unittest.TestCase":http://docs.python.org/lib/testcase-objects.html. The most common ones are:

failUnless(expr) -- Ensure 'expr' is true
 
assertEqual(expr1, expr2) -- Ensure 'expr1' is equal to 'expr2'

assertRaises(exception, callable, ...) -- Make sure 'exception' is raised by the callable. Note that 'callable' here should be the name of a method or callable object, not an actual call, so you write e.g. 'self.assertRaises(AttributeError, myMethod, someParameter)'. Note lack of '()' after 'myMethod'. If you included it, you'd get the exception raised in your test method, which is probably not what you want. Instead, the statement above will cause the unit testing framework to call 'myMethod(someParameter)' (you can pass along any parameters you want after the calalble) and check for an AttributeError.

fail() -- Simply fail. This is useful if a test has not yet been completed, or in an 'if' statement inside a test where you know the test has failed.

In addition to the unit testing framework assertion methods, ZopeTestCase and PloneTestCase include some helper methods and variables to help you interact with Zope. It's instructive to read the source code for these two products, but briefly, the key variables you can use in unit tests are:

self.portal -- The Plone portal the test is executing in

self.folder -- The member folder of the member you are executing as

And the key methods are:

self.logout() -- Log out, i.e. become anonymous

self.login() -- Log in again. Pass a username to log in as a different user.

self.setRoles(roles) -- Pass in a list of roles you want to have. For example, self.setRoles(('Manager',)) lets you be manager for a while. How nice.

self.setPermissions(permissions) -- Similarly, set a number of permissions for the test user in self.folder

self.setGroups(groups) -- Set a list of groups for the current user. 

Tips & Tricks

  Good unit testing comes with experience. It's always useful to read the unit tests of code with which you are fairly familiar, to see how other people unit test. We'll cover a few hints here to get you thinking about how you approach your own tests:
 
o Don't be timid! Python, being a dynamic, weakly typed scripting language, lets you do all kinds of crazy things. You can rip a function right out from the Plone core and replace it with your own implementation in 'afterSetUp()' or a test if that serves your testing purposes.

o Similarly, replacing things like the MailHost with dummy implementations may be the only way to test certain features. Look at 'CMFPlone/tests/dummy.py' for some examples of dummy objects.

o Use tests to try things out. They are a safe environment. If you need to try something a bit out of the ordinary, writing them in a test is often the easiest way of seeing how something works.

o During debugging, you can insert 'print' statements in tests to get traces in your terminal when you execute the tests. Don't check in code with printing tests, though. :)

o Similarly, the python debugger is very valuable inside tests. Putting 'import pdb; pdb.set_trace()' inside your test methods lets you step through testing code and step into the code it calls. If you're not familiar with the python debugger, your life is incomplete. Read more in the "python documentation":http://docs.python.org/lib/module-pdb.html.


    