Peter Marklund's Home

Introduction

This is an article that talks about the motivation for testing and how I went about writing automated tests for my personal website. It is best suited for someone with at least some familiarity with Rails but who hasn't yet explored its testing capabilities. There is also a section on web testing that is independent of Rails.

I learned my lesson in Berlin

I still remember clearly the somewhat embarrassing mistake that sparked my interest in software testing. I was working as an intern for a research institute in Berlin and my task was to program an image processing algorithm in C. Once the program was finished I was stalled for a long time, maybe weeks, puzzled over the unexpected outputs of my program. My supervisor was starting to lose patience with my lack of progress. Eventually it occurred to me that the underlying algorithm that I had implemented was broken due to a trivial mistake. All along I had simply assumed that it was working without conducting any thorough tests to prove my assumption. That's when it dawned on me what a challenging and important problem software testing is.

The miserable past of web testing

A lot of developers see testing as a dull or even unnecessary activity. The level of testing has been particularly poor in web development, a sector of the software industry notorious for being hackish and low in quality. Granted, the cost of a broken web page is typically not as big as a bug in banking software. However, a broken page on a high volume public site or an internet banking site can be very costly indeed.

With Agile methodologies such as Extreme Programming the idea is to make changes, refactor, and release often. Good automated test coverage is almost a prerequisite for this style of development. This is because for each change that we make we need to make sure that we didn't break existing functionality. In a system with high coupling the bug caused by a change can arise in a seemingly unrelated part of the system. Of course, manual testing is not really an option in agile development as it would be too expensive and slow down the development and release cycle too much.

There are benefits to testing other than just finding bugs that are worth mentioning. One is that testing forces you to focus on and define the interface (aka contract) of your application or API. This is the reason why test driven development can be so valuable since defining the contract of your code early on provides a good direction and goal for your programming - it serves as a specification. Also, when you are writing tests you will often find yourself reviewing your code and this review in and of itself can reveal valuable improvements.

Rails shows the way

Most of the hype surrounding Rails has to do with its productivity and the surprisingly few lines of code needed to build a typical website. A nice side effect of having a small code base is that you have fewer lines of code to debug. Testing in Rails is also helped by its strict MVC (Model-View-Controller) structure. Last but not least, Rails comes shipped with a tightly integrated testing framework that allows testing both on the controller and model levels.

Rails has an interesting code statistics tool invoked by "rake stats" that counts the lines of code (LOC) and the ratio of production to test code. Here is the output for my website:

+----------------------+-------+-------+---------+---------+-----+-------+
| Name                 | Lines |   LOC | Classes | Methods | M/C | LOC/M |
+----------------------+-------+-------+---------+---------+-----+-------+
| Helpers              |    19 |    18 |       0 |       1 |   0 |    16 |
| Controllers          |   260 |   210 |       6 |      37 |   6 |     3 |
| Components           |     0 |     0 |       0 |       0 |   0 |     0 |
|   Functional tests   |   340 |   253 |      10 |      45 |   4 |     3 |
| Models               |   122 |    91 |       4 |      12 |   3 |     5 |
|   Unit tests         |   191 |   150 |       3 |      17 |   5 |     6 |
| Libraries            |     0 |     0 |       0 |       0 |   0 |     0 |
+----------------------+-------+-------+---------+---------+-----+-------+
| Total                |   932 |   722 |      23 |     112 |   4 |     4 |
+----------------------+-------+-------+---------+---------+-----+-------+
  Code LOC: 319     Test LOC: 403     Code to Test Ratio: 1:1.3

Unit testing

A unit test in Rails tests a single model class. For my website I have three model classes - Post, Category, and User. There is a one-to-one mapping in Rails between model classes and database tables. The model classes have methods for the usual CRUD (create, read, update, delete) operations and all of these operations interact with the database.

When you write tests that depend on database data it will simplify your life a lot if the tests have control over that data. This is one of the areas where the Rails testing framework shines. All Rails tests are executed against a separate test database into which data is loaded with the help of fixtures. Each database table has a corresponding fixture YAML file. YAML is a concise and yet readable text data format that is quite handy to work with. The YAML files can contain embedded ruby code and so can be dynamic with loops and variables.

Here are some examples of what my unit tests cover:

• The method used to list posts on the front page only lists public posts and lists them in the right order. The contents of the posts is right and the category mappings are correct.
• The Post.save(), Post.update(), and Post.destroy() methods work. Since this is default Rails functionality one could argue that those tests could safely be left out.
• The method used to list all categories for the right menu finds all categories and lists them in the right order.
• Users cannot be created without proper name and password.
• Correct logins credentials are accepted and bad login attempts are rejected.
• No other user attributes other than name and password can be set from a user registration form. This means for example that a hacker could not force a user to become an admin.

Functional testing

In Rails functional tests are applied at the controller level. Each controller class has a corresponding controller test class. The controller test executes actions on its controller, makes assertions about the values of the member variables in the controller, about the type of response (success, redirect etc.), and about the HTML output generated by the view. The controller test also has access to the request and response objects, to the session, and to the flash object used to display status messages to the user. All in all this makes controller tests very powerful.

Here are some functional tests I wrote for my website:

• The actions listing posts on the homepage uses the right layout, initializes the posts variable as it should and has the public posts in its HTML output.
• Trying to view a private post through URL manipulation yields a 404 response.
• The category and year pages list the posts that they should in the right order.
• Invoking a non-existent action such as peter/foobar yields a 404 response.
• Anonymous users or non-admin users cannot access the admin interface.
• The admin interface with CRUD operations for posts and categories works.

One minor limitation of controller tests to be aware of is that they don't include the URL-action mappings (the routes). One could probably test those separately somehow but I haven't seen an example of that yet. The URL mappings are covered by the web tests though, see below.

There is potential for overlap between unit and functional testing. Thorough testing at the model level decreases (but does not eliminate) the need for testing at the controller level. It is clearly an advantage if the controller tests can rely on the underlying model to be working correctly so that they can focus on the controller specific logic.

Web testing and Monitoring

I use web testing for pre-launch and post-launch acceptance testing and for monitoring of my website. This means I run the web tests just before a site upgrade, immediately after the upgrade, and then regularly every day against the production site. The web tests execute the entire system including the production database. The fact that the production database is used is important since it is not uncommon for bugs to depend on database data. The Rails unit and functional tests can provide good test coverage but the web tests are a necessary complement and are the ultimate check that the system is working at the point of release and that it keeps working in production.

I have divided my web tests into two parts. I have one "ping" test that I run frequently and that just makes sure that the front page works and the site is up. The other test is a smoke test that I run twice a day. For all pages on my site it checks that:

• The HTTP response code is 200 OK
• The response time is reasonable, i.e. below a certain threshold like 3 seconds.
• The page is valid XHTML. This check is useful not only for accessibility and browser compatibility but also for finding bugs, broken markup, and lack of quoting.
• There are no broken links or broken images

If any of the web tests fail I am alerted by email. I am also alerted by email if an error occurs in the code (i.e. an exception is thrown). With this setup I can rest assured that if major functionality on the site breaks down I will know about it and have a good chance of taking action quickly.

Web testing is not provided by the Rails framework and I have chosen to rely on Perl scripts for those tests. My web tests currently don't do much HTML parsing, but if there was a need for that Perl has its simple yet powerful look_down method. However, if there were similar APIs available in Ruby I would probably switch to Ruby as I am more comfortable with that language.

What's Missing?

There should be some kind of automatic test that verifies that site backups are working. A way to do that would be to have a mirror site in a different location that is updated nightly with the database backup from the production site. One could then run the web tests against the mirror site as well. One would also need to verify, through a web test or otherwise, that no data is missing in the backup. If the production site is not changed during the recreation of the mirror site a sufficient test would be to check if the two sites are identical.

If a website makes heavy use of JavaScript and Ajax then an in-browser testing tool such as Selenium may be useful for acceptance testing and browser compatibility testing.

Last, but not least, if you are expecting a lot of traffic you should look at load testing with tools such as Apache Bench (ab), or siege. Basic usage of these tools is shown in the Rails book. Rails also has the concept of performance tests that make it easy to load huge quantities of data and then repeatedly execute some critical action and measure its average execution time.

Conclusion

The level of testing that I have described here isn't comprehensive and really ought to serve as a baseline for pretty much any serious website. Unfortunately, this is far from the case in the industry today. With the advent of Rails, testing is more readily available to web developers than ever before. Let's remind ourselves to take full advantage of that.