Testing and Django
by Matthew Snider · June 11, 2011 @ 2:24 p.m.
This article continues last weeks series on building a continuous deployment environment using Python and Django.
- Starting Your First Django Project
- Testing and Django
- Mock and Coverage
- Using Fabric for Painless Scripting
- Using Celery to handle Asynchronous Processes
- Deployment/Monitoring Strategies
Before we discuss testing in Django, lets define what makes up good testing infrastructure.
There are many schools of thoughts on testing, but that's another article. Ignoring the different testing philosophies, I've always broken my tests into 3 basic types: unit, functional, and non-functional testing. At the very least one should be able to test every function, testing the response of a function when different variable combinations are provided (unit testing). When building websites, one also needs a client unit test that evaluates site pages by executing the controller code for a specific request and evaluating the view response (treat these like unit tests). An extension of unit testing is functional testing, where you test a business objective from end-to-end, such as BAT (basic acceptance test), user sign in, or purchasing goods. The rest falls into non-functional testing, which covers events that are not related to business objectives, but may be equally important, such as scalability and security.
From the above you can make a test plan that covers every function and view in your code, and any critical business objectives or support processes. However, in addition to a basic testing harness, we will need at least three more tools: mock data, test coverage, and a tool for mocking objects/functions. Mock data is a collection of known data to load before a test suite/case, and usually unload afterwards. Test coverage will measure how much of your code is executed by the tests. This will help evaluate the completeness of your test harness. A mocking tool will allow testing around services outside of the app being tested, without actually calling those services. The mocking tool will replace the service call and return specified data instead, allowing for testing of known good and bad responses.
We will be discussing all concepts mentioned above today, except coverage and the mocking tool, which will be covered in the next article. With these concepts in mind, lets look at what Django and Python provide for building your testing system.
Getting ready
Django looks for tests.py
files in each of your apps registered in settings.py
and executes the TestCase classes in them. Django runs tests in the order it loads them, but unloads any data written by one test before executing another (except when modifying initial test data), so test cases should always be written independent of each other. This ensures that tests will pass, regardless of the order, while allowing for multithreading, as your test suite grows.
If you were following the instructions from Tips for Starting Your First Django Project then you already have everything setup. Otherwise, follow the steps to create a project and start the myapp1 app.
How to do it…
Django will have generated a tests.py
file that contains:
"""
This file demonstrates writing tests using the unittest module. These will pass
when you run "manage.py test".
Replace this with more appropriate tests for your application.
"""
from django.test import TestCase
class SimpleTest(TestCase):
def test_basic_addition(self):
"""
Tests that 1 + 1 always equals 2.
"""
self.assertEqual(1 + 1, 2)I recommend writing a TestCase per file in an app, so something like:
class BaseTestCase(TestCase): … class ModelsTestCase(TestCase): … class ViewsTestCase(TestCase): … etc…
The Django test runner will look at each TestCase instance and execute the functions starting with test
. Before each test function runs, it will load any mock data and run the setUp function (if defined). After each test, it will drop any data loaded before and during then test function, then run the tearDown function (if defined).
Test Data
Test data in Django is called a fixture and can be used to load static data, such as geographical locations, or dynamic data, such as test users. Fixtures are useful when you need data in your app for the tests to run or you setup users with specific conditions that will target different parts of the code.
Fixtures should be stored in the myapp1/fixtures
directory. If the fixture is called initial_data.json
then it will be automatically when the test runner starts. You may also define fixtures that need to be loaded before each test in a TestCase by defining the fixtures attribute:
class BaseTest(TestCase): fixtures = ['myapp1_fixture1', 'myapp1_fixture2'] …
Django will look for
myapp1_fixture1.jsonand
myapp1_fixture2.jsonin the
fixturesdirectory of all apps, so make sure the names are unique accross all apps (I do this my prefixing file names with the name of the app).
When creating fixtures, use the command line to create a fixture from the database:
python manage.py dumpdata myapp1 > myapp1/fixtures/myapp1_testdata.json
Here is an example fixture for the Django auth user:
[
{
"pk": 1,
"model": "auth.user",
"fields": {
"username": "completed_reg",
"first_name": "Matt",
"last_name": "Snider",
"email": "[email protected]",
"password": "sha1$e66d8$c994203a282b2ebff8330a690378908b71c8c1c2",
"is_staff": 0,
"is_active": 1,
"is_superuser": 0,
"last_login": "2010-09-07 19:52:39",
"date_joined": "2010-09-07 19:52:39"
}
}
]
All JSON fixtures are an array of objects, with at least the pk
, model
, and some fields
defined. Whenever possible, I try to use one of the fields to make the fixture data descriptive as to its purpose, such as the username
in this example.
For more information on fixtures and different data formats, see Initial Data.
Assertions
The core to any testing infrastructure is the ability to make assertions about the code. The Django TestCase
class extends the python ut2.TestCase class, which has all the basic assertions:
from django.test import TestCase class BaseTest(TestCase): def test_some_things(self): # these assertions have been in python for a while self.assertTrue(1) # evaluates truthy-ness self.assertFalse(0) # evaluates falsy-ness self.assertEquals(1, 1) # evaluates second (dynamic) value equals the first (known) value; self.assertNotEquals(1, 2) # some of the new assertions added in python 2.7 o = 1 self.assertIs(o, o) # assert evaluate to the same object self.assertIn(1, [1, 2, 3]) # assert first value in second self.assertIsInstance(self, TestCase) # first value is instance of second
For the full documentation see, the Python Documentation 25.3. unittest – Unit testing framework.
Django also adds new assertions for evaluating views and other Django specific tasks:
from django.test import TestCase class BaseTest(TestCase): def test_some_things(self): self.assertRedirects(response, expected_url, status_code, target_status_code) self.assertContains(response, text, count, status_code) self.assertNotContains(response, text, count, status_code) self.assertFormError(response, form, field, errors) self.assertTemplateUsed(response, template_name) self.assertTemplateNotUsed(response, template_name) self.assertQuerysetEqual(qs, values)
Most of these evaluate responses from the
Clientclass, which emulates requests so you can test views. Additionally, you can use
assertQuerysetEqualto examine Django query sets. See the Django Assertion, for more information on how these tests work.
Testing Views
While basic testing is straight forward, lets dive into view testing, as it is a little more complex. If we setup the following crud urls, views, and tests for a simple comment app:
/urls.py
…
(r'^comments/', include('comments.urls', namespace='comments')),
…
comments/urls.py
from django.conf.urls.defaults import *
urlpatterns = patterns('comments.views',
url(r'^create/(?P\d+)/$', 'create', name='create'),
url(r'^delete/(?P\d+)/$', 'delete', name='delete'),
url(r'^(?P\d+)/$', 'read', name='read'),
url(r'^update/(?P\d+)/$', 'update', name='update'),
)
comments/views.py
# imports not included, comments/models.py and comments/forms.py assumed
def create(request, comment_id=None):
if 'POST' == request.method:
form = CommentForm(request.POST)
if form.is_valid():
cleaned_data = form.cleaned_data
comment = Comment.objects.create(user=request.user, comment=cleaned_data.get('comment', ''))
return redirect('comments:read', comment_id=comment.id)
if comment_id: # update
comment = get_object_or_404(Comment, id=comment_id)
form = CommentForm(comment=comment.comment)
else: # create
form = CommentForm()
ctx = {
'comment': comment,
'form': form,
}
return render_to_response('comments_create.html', ctx, context_instance=RequestContext(request))
@login_required
def delete(request, comment_id):
comment = get_object_or_404(Comment, id=comment_id)
user = request.user
# assert logged in user is comment owner
if not (user and user.is_authenticated() and user.pk = comment.user_id):
raise Http404
return redirect('comments:add')
def read(request, comment_id):
comment = get_object_or_404(Comment, id=comment_id)
ctx = {
'comment': comment,
}
return render_to_response('comments_read.html', ctx, context_instance=RequestContext(request))
comments/tests.py
from django.test import TestCase
from django.test.client import Client
class CommentViewTestCase(TestCase):
def setUp(self):
super(CommentViewTestCase, self).setUp()
self.client = Client()
self._i = 1
def _create_user(self) {
'''Create simple, throw-away users'''
user = User.objects.create(
first_name='first_name%d' % self._i,
last_name='last_name%d' % self._i,
email='test%[email protected]' % self._i,
username='test%d' % self._i,
)
self._i += 1
return user
}
def test_create_and_update(self):
# too complex for this example
pass
def test_delete(self):
view_name = 'contacts:delete'
# comment doesn't exist, should raise 404
url = reverse(view_name, kwargs={'comment_id': 1})
response = self.client.get(url)
self.assertEquals(response.status_code, 404)
# create two test users and a comment
user1 = self._create_user()
user2 = self._create_user()
comment_text = 'test'
comment = Comment.objects.create(comment=comment_text, user=user1)
# non-logged in user cannot delete a comment
url = reverse(view_name, kwargs={'comment_id': comment_id})
response = self.client.get(url)
self.assertRedirects(response, reverse('signin'), 302, 200)
# non-owning user cannot delete a comment
self.client.login_user(user2)
url = reverse(view_name, kwargs={'comment_id': comment_id})
response = self.client.get(url)
self.assertEqual(response.status_code, 404)
# owning user can delete a comment
self.client.login_user(user1)
url = reverse(view_name, kwargs={'comment_id': comment_id})
response = self.client.get(url)
self.assertContains(response, comment_text, 1)
def test_page(self):
view_name = 'contacts:read'
# comment doesn't exist, should raise 404
url = reverse(view_name, kwargs={'comment_id': 1})
response = self.client.get(url)
self.assertEqual(response.status_code, 404)
# create test user and comment
user = self._create_user()
comment_text = 'test'
comment = Comment.objects.create(comment=comment_text, user=user)
# non-logged in user can view page
url = reverse(view_name, kwargs={'comment_id': comment_id})
response = self.client.get(url)
self.assertContains(response, comment_text, 1)
# logged in user can view page
self.client.login_user(user)
url = reverse(view_name, kwargs={'comment_id': comment_id})
response = self.client.get(url)
self.assertContains(response, comment_text, 1)
This example shows how to setup the urls, build some simple crud views, and test them. It is important to test against which users can access and edit content. Additionally, we assert that the comment is in the response text when testing the comment page, and we assert that the delete redirects to the sign in (assumed to be defined) page when the user is not logged in.
Running Tests
Tests are simple to run from the command line. To test a single app use:
python manage.py test myapp1
To test several apps use:
python manage.py test myapp1 myapp2 myapp3 …
To test all apps, including Django apps, use:
python manage.py test
How it works…
The Django TestRunner does the following: sets up the Django environment, builds the test suite from all the test cases in the included apps, connects to the database and creates test tables using syncdb, runs the test suite, tears down the database and environment, then reports the results. The TestRunner will report the number of tests run, which ones fail or raise exceptions, and will show the exceptions. When running via a script for continuous deployment, check the stderr output for failures.
Fixtures can drastically affect the performance of your tests, as fixtures defined on a test case are loaded and unloaded between each test. Use initial_data.json (can have 1 per app) to load static data that should be shared between all apps. However, one should generally try to keep fixtures as small as possible.
There’s more…
For more general information on tests, see Testing Django applications.
One way you can increase the performance of your tests is to use the SQLite DB for testing, instead of an SQL DB. Use the following setting to detect if you are running a test or not, and to setup your DB options:
from sys import argv
if argv and 1 < len(argv):
IS_TEST = 'test' == argv[1]
else:
IS_TEST = False
if IS_TEST:
DB_ENGINE = 'django.db.backends.sqlite3'
ELSE:
DB_ENGINE = 'django.db.backends.mysql'
DATABASES = {
'default': {
'NAME': DB_NAME,
'ENGINE': DB_ENGINE,
'USER': DB_USER,
'PASSWORD': DB_PASSWORD,
'HOST': DB_HOST,
'OPTIONS': DB_OPTIONS,
},
}
As you write more tests, common patterns begin to arise, such as a helper function for creating disposable users (shown in the view example). I recommend defining a common TestCase class as a basis for your other tests:
from django.test import TestCase
class TestCaseBase(TestCase):
def setUp(self):
super(TestCaseBase, self).setUp()
self._user_counter = 1
def _create_user(self):
first_name = 'delete'
last_name = 'me'
i = self._user_counter
username = '%s%s%[email protected]' % (first_name, last_name, i)
self._user_counter += 1
user = User.objects.create(
email='%[email protected]' % username,
username=username,
first_name=first_name,
last_name=last_name,
)
self._created_users.append(user)
return user
Lastly, if you use signals and myapp1
listens for model changes in myapp2
, then when you are testing myapp2
those listeners will fire. This tends to be undesired as you want to test each app in isolation of each other. In this case myapp1
should be testing those listeners, not myapp2
, which shouldn’t need to know what other apps listen for its model changes.
Fortunately, Django allows you to temporarily turn off some or all signal listeners. At Votizen.com, we have abstracted this logic away for testing purposes, so we can off all or just some signals:
class NoSignalTestCase(TestCaseBase):
def _receiver_in_lookup_keys(self, receiver, lookup_keys):
"""
Evaluate if the receiver is in the provided lookup_keys; instantly terminates when found.
"""
for key in lookup_keys:
# print '%s - %s - %s' % (key[0], receiver[0][1], key[1])
if (receiver[0][0] == key[0] or key[0] is None) and receiver[0][1] == key[1]:
return True
return False
def _find_allowed_receivers(self, receivers, lookup_keys):
"""
Searches the receivers, keeping any that have a lookup_key in the lookup_keys list
"""
kept_receivers = []
for receiver in receivers:
if self._receiver_in_lookup_keys(receiver, lookup_keys):
kept_receivers.append(receiver)
return kept_receivers
def _create_lookup_keys(self, sender_receivers_tuple_list):
"""
Creates a signal lookup keys from the provided array of tuples.
"""
lookup_keys = []
for keep in sender_receivers_tuple_list:
receiver = keep[0]
sender = keep[1]
lookup_key = (_make_id(receiver) if receiver else receiver, _make_id(sender))
lookup_keys.append(lookup_key)
return lookup_keys
def _remove_disallowed_receivers(self, receivers, lookup_keys):
"""
Searches the receivers, discarding any that have a lookup_key in the lookup_keys list
"""
kept_receivers = []
for receiver in receivers:
if not self._receiver_in_lookup_keys(receiver, lookup_keys):
kept_receivers.append(receiver)
return kept_receivers
def setUp(self, sender_receivers_to_keep=None, sender_receivers_to_discard=None):
"""
Turns off signals from other apps
The `sender_receivers_to_keep` can be set to an array of tuples (reciever, sender,), preserving matching signals.
The `sender_receivers_to_discard` can be set to an array of tuples (reciever, sender,), discarding matching signals.
with both, you can set the `receiver` to None if you want to target all signals for a model
"""
super(NoSignalTestCase, self).setUp()
self.m2m_changed_receivers = m2m_changed.receivers
self.pre_delete_receivers = pre_delete.receivers
self.pre_save_receivers = pre_save.receivers
self.post_delete_receivers = post_delete.receivers
self.post_save_receivers = post_save.receivers
new_m2m_changed_receivers = []
new_pre_delete_receivers = []
new_pre_save_receivers = []
new_post_delete_receivers = []
new_post_save_receivers = []
if sender_receivers_to_keep:
lookup_keys = self._create_lookup_keys(sender_receivers_to_keep)
new_m2m_changed_receivers = self._find_allowed_receivers(self.m2m_changed_receivers, lookup_keys)
new_pre_delete_receivers = self._find_allowed_receivers(self.pre_delete_receivers, lookup_keys)
new_pre_save_receivers = self._find_allowed_receivers(self.pre_save_receivers, lookup_keys)
new_post_delete_receivers = self._find_allowed_receivers(self.post_delete_receivers, lookup_keys)
new_post_save_receivers = self._find_allowed_receivers(self.post_save_receivers, lookup_keys)
if sender_receivers_to_discard:
lookup_keys = self._create_lookup_keys(sender_receivers_to_discard)
new_m2m_changed_receivers = self._remove_disallowed_receivers(new_m2m_changed_receivers or self.m2m_changed_receivers, lookup_keys)
new_pre_delete_receivers = self._remove_disallowed_receivers(new_pre_delete_receivers or self.pre_delete_receivers, lookup_keys)
new_pre_save_receivers = self._remove_disallowed_receivers(new_pre_save_receivers or self.pre_save_receivers, lookup_keys)
new_post_delete_receivers = self._remove_disallowed_receivers(new_post_delete_receivers or self.post_delete_receivers, lookup_keys)
new_post_save_receivers = self._remove_disallowed_receivers(new_post_save_receivers or self.post_save_receivers, lookup_keys)
m2m_changed.receivers = new_m2m_changed_receivers
pre_delete.receivers = new_pre_delete_receivers
pre_save.receivers = new_pre_save_receivers
post_delete.receivers = new_post_delete_receivers
post_save.receivers = new_post_save_receivers
def tearDown(self):
"""
Restores the signals that were turned off.
"""
super(NoSignalTestCase, self).tearDown()
m2m_changed.receivers = self.m2m_changed_receivers
pre_delete.receivers = self.pre_delete_receivers
pre_save.receivers = self.pre_save_receivers
post_delete.receivers = self.post_delete_receivers
post_save.receivers = self.post_save_receiversBy default this will turn off all signals, but you can use the special
setUp function to specify signals to keep or signals to discard. For example, if I wanted to turn off all signals to only the Comment model:
class CommentTestCase(NoSignalTestCase):
def setUp(self):
super(MentionMentionTestCase, self).setUp(sender_receivers_to_discard=[(None, Comment),])Or only keep signals for the
Comment model:
class CommentTestCase(NoSignalTestCase):
def setUp(self):
super(MentionMentionTestCase, self).setUp(sender_receivers_to_keep=[(None, Comment),])And lastly, if you want to target a specific signal callback function (known as the
receiver), pass that in instead of
None, as the first value of the tuple:
def comment_pre_save_listener(sender, instance, **kwargs):
# do something
pass
pre_save.connect(comment_pre_save_listener, sender=Comment)
class CommentTestCase(NoSignalTestCase):
def setUp(self):
super(MentionMentionTestCase, self).setUp(sender_receivers_to_discard=[(comment_pre_save_listener, Comment),])