El Tramo | Unit Testing

Beautiful (XMPP) Testing

2009-11-02T00:00:00+01:00

O’Reilly recently released the book Beautiful Testing, a collection of essays about testing and QA in general. As I mentioned earlier, I wrote an article in that book on (unit) testing XMPP protocols, using Swift as a motivating example. Since the book’s scope may (oddly enough) not always be as interesting for developers in general, I released my article under a Creative Commons Attribution license (thanks to the good folks from O’Reilly for encouraging us to do this), which you can find here (or directly from my Git repository). The original excerpt from the book (including the index, list of biographies, and all the fancy artwork) is also available for download under the same license.

I of course encourage you to buy either the PDF or dead tree version of the book, as all the proceeds of the book go to charity.

“Beautiful Testing” XMPP Chapter

2009-05-03T00:00:00+02:00

Adam Goucher and Tim Riley (Director of QA at Mozilla) announced a few months ago that they are putting together a Beautiful Testing book for O’Reilly. I took the opportunity to write a chapter about testing in the context of XMPP (more specifically, about testing protocol implementations in Swift), and just submitted the final draft for technical review. The book is expected to be released this August.

Although there are many types of testing being done in the XMPP world, the chapter focuses on the beauty of testing the functionality of XMPP protocol implementations. After a brief introduction on XMPP, it starts out with a description of unit testing simple IQ request/response protocols, and then gradually moves on to higher-level testing of more complex, multi-stage protocols such as session initialization. As you might expect from a developer like me, the chapter is quite heavy on the (C++) code, but I’m told it compensates for the rest of the book ;–)

As with all other books in the O’Reilly “Beautiful” series (which started with Beautiful Code, but has since been followed up by Beautiful Architecture, Beautiful Teams, Beautiful Security, and Beautiful Data), all proceeds of the book go to charity, in this case “Nothing But Nets” (which provides mosquito netting to malaria infested areas of Africa). This means that I can plug this book as much as I want, and still have the feeling I’m actually doing a noble, unselfish thing. (contrary to when I casually mention that you can buy our book XMPP: The Definitive Guide at very sharp prices these days). Some time after the book’s release this summer, I will even make a free version of the chapter available here, so check back soon!

Improving QtTest usability with QtTestUtil

2008-11-06T00:00:00+01:00

As much as I like CppUnit for writing C++ unit tests, I still prefer using Qt’s built-in QtTest module for Qt-based projects. This avoids a dependency on an external library, lowering the threshold for running and writing unit tests. Unfortunately, QtTest is very basic, and lacks some useful features such as automatic test registration and running multiple test suites in one test binary. In order to improve QtTest’s usability, I started creating some macros and classes that fill in some of the gaps, and bundled them into QtTestUtil.

QtTest’s recommended way to write unit tests is to compile and link one test suite per class separately, and using the QTEST_MAIN macro to generate a main() that runs this particular test suite. However, this introduces quite a bit of overhead in writing (creating a project file for every test/class), building (linking a binary for every test/class), and running (running a separate binary for every test/class, usually using a custom script) all unit tests. This overhead is especially painful in a project with many classes. All this conflicts with the general rule that both creating and running unit tests should be very efficient, and makes QtTest not well suited for unit testing as it is.

A first attempt at solving this problem is by manually creating a main() function that runs all tests. For example:

#include "MyFirstTest.h"
#include "MySecondTest.h"

int main(int argc, char* argv) {
  int result = 0;
  MyFirstTest test1;
  result |= QTest::qExec(&test1, argc, argv);
  MySecondTest test2;
  result |= QTest::qExec(&test2, argc, argv);
  return result;
}

The resulting binary runs all listed test suites. The downside of this approach is that you have to write quite a bit of boilerplate code: every test binary needs to have its own main() function that explicitly runs all the tests in that test suite, and which needs to be put in a file that has to include all the tests separately. In order remedy this, I took the idea from CppUnit to provide a macro which auto-registers a test suite, and makes it easy to run all registered test suites at once. Creating a test suite is now as simple as creating a .cpp file for every unit test, adding a call to QTTESTUTIL_REGISTER_TEST to that file, and compiling that together with a shared main() that does nothing but call runTests() on the test registry. Since this main() does not explicitly depend on any test, QtTestUtil comes with a SimpleChecker.cpp containing this call, and which can be used by all unit test modules. As a result, no module needs to provide its own main() anymore.

An example of a unit test module using the macro and the shared checker described above can be found in the Example subdir of the QtTestUtil repository. It is worth noticing that these QtTest-based unit tests are more compact than the same unit test written in CppUnit (because the latter requires you to call a macro for every test in a fixture).

This is just a first set of utilities that QtTestUtil provides on top of QtTest. More utility classes and macros will be added as they are needed.

Going Agile with Google Summer of Code

2008-06-26T00:00:00+02:00

Although Psi has had a fair number of succesful Google Summer of Code projects so far, we have experienced some failures as well: the summer before last, 3 out of 6 projects didn’t make the final deadline. A project’s failure was typically due to not having anything really usable at the end of the summer, regardless of the good work that was done during the past months. To reduce the risk of such surprises, I decided to take an Agile Development approach for this year's ‘Roster improvement’ project.

Here are some of the principles of Agile software development (as written down in the Agile Manifesto), and how we apply them to Adam’s GSoC project:

The developer frequently delivers working software to the customer. In our case, Adam sends me a working roster every monday, which I can play around with. I then tell him what I do and don't like about it, and he processes my feedback during the next iteration (i.e. the next week). Of course, since the roster is made from scratch, the backend of the roster is completely mocked up during the first weeks. As the project progresses, the backend will become more 'real'.
Working software is the primary measure of progress. This avoids the kind of surprises we had before, where both student and mentor believed everything was going fine during the project, but where nothing could be delivered at the end of the project.
Code is kept as simple as possible: if you don't need something, leave it out. By initially focusing on the roster frontend (while keeping the backend framework mocked up), we ensure that our backend will only be as complex as we actually need it to be. Moreover, we can constantly change our mind about the requirements throughout the whole project without having to throw away the complete backend. This contrasts with the previous attempts we had at reworking the Psi roster, where we started by working out a complex roster backend, but never got to the point where we had something working with this (over-)designed framework, and had to start over all again.
Code is written with testability in mind. In principle, all new code needs to be tested (and preferably even unit tested before it's written), which allows you to quickly refactor while remaining sure that your code still works as before. However, given the very strict time constraints, I'm not sure yet that we're going to apply this Test-driven Development approach (although some claim that the extra time you spend in tests will even pay off in the short term).
Good designs make sure that you can easily adapt your code to changing requirements. For example, a Model-View-Controller-based design seems to lend itself well to the roster, making the (rather complex) roster logic both testable, extensible, and easy to understand.

I’m convinced that these principles will ensure that both Adam and I (and of course all Psi users) will get a succesful result at the end of the project, and that we’ll have had a fun time during these 3 months.

Improving Psi's roster

2008-05-01T00:00:00+02:00

For a while now, Psi users have been requesting several changes and additions to the roster (or `contact list‘). These requests include grouping contacts into meta-contacts, nested roster groups, and displaying user avatars in the roster. We have been postponing all these changes to the roster as much as possible, because none of us wanted to touch the roster code, for reasons I’ll explain below. This year, Psi is fortunate enough to have Adam Czachorowski (aka Gislan), a student from the Google Summer of Code, to work on roster improvements.

So far, the major cause of all Psi developers staying clear from doing substantial roster changes is probably the fact that all roster-related classes are very tightly coupled to each other. This has several consequences:

It's hard to get a good understanding of which piece of code does what, since there is no separation of logic between the different classes, and none of them work without the other one.
The code is very fragile: if you change one tiny piece, you might break something completely different. Moreover, it's hard to tell what you broke, since the functionality is spread out across the different classes.
The code is untestable: Since there is no decent separation of the roster logic (i.e. the structure of the roster) and the roster user interface (i.e. how it is shown), it has been impossible so far to create some form of automated tests for the roster code. Because of the tight coupling between the various classes, it is impossible to test each part of the roster functionality in isolation, making unit testing impossible. Given that the roster is the most central part of the Psi user interface, it is actually unacceptable that we have no form of testing whether it (still) functions correctly.
There can be no reuse of any of the roster code in other parts of the UI (such as the list of participants in the MUC dialog), or even for other IM clients that are based on the back-end of Psi.

Because of these fundamental issues, a complete makeover of the roster code is in order. More specifically, we want to have a clear separation of anything that has to do with UI, and the actual logic of the roster. Additionally, we want the (untestable) UI layer to be as thin as possible, pushing as much down to the logic layer as we can. Finally, we want to achieve a full coverage of the logic layer using only unit tests.

What Gislan will exactly do in his Google Summer of Code project still has to be worked out in detail. He will be responsible for a major part (if not all) of the roster rewrite, and get some new functionality in there as well (since that will be a breeze with the new code). You can follow his progress on his blog, and get more detailed technical information on the project page for his GSoC project.

Unit testing method overrides

2008-03-07T00:00:00+01:00

It probably happened to most of us developers before: while refactoring, you change the name of a virtual method, but forget to change the name of the overriding method in one of your derived classes. Compilation works fine, all unit tests pass, but your program doesn’t work: the overriding method is never called. Java (and C#) programmers can avoid this problem by putting @Override (and override) in front of their methods, which causes the compiler to print out an error message if the method is not overriding anything. However, most other languages leave you hanging with this problem. Luckily, with statically typed languages like C++, you can avoid these bugs by slightly adapting your unit tests.

Suppose you have a base class Oracle, and a derived class GeekyOracle, defined as follows:



  class Oracle {
    virtual int getLckyNmbr() {
      return 13;
    }
  };

  class GeekyOracle : public Oracle {
    virtual int getLckyNmbr() {
      return 42;
    }
  };

Being a good developer, you unit test GeekyOracle’s getLckyNmbr():



  void GeekyOracleTest::testGetLckyNmbr() {
    GeekyOracle* oracle = createGeekyOracle();
    assertEquals(42, oracle->getLckyNmbr());
  }

After a while, you realize that using abbreviations in method names is actually not very good practice, so you change your method name to something more readable:



  class Oracle {
    virtual int getLuckyNumber() {
      return 13;
    }
  };

However, when you compile your program now, it is behaving completely different than before, although all unit tests passed. It requires manual debugging to find out that you forgot to rename GeekyOracle’s method as well, causing Oracle’s getLuckyNumber() to be executed, no matter what oracle you instantiate. Your unit tests didn’t catch this, because they test every method of a class separately, but don’t test the class being used in your application.

A way to check for this in your unit tests is to make sure that, when testing a virtual method, you always call it on an object that is statically typed with its base class. In our example, our unit test would call getLckyNmbr() on an Oracle instead of on a GeekyOracle:



  void GeekyOracleTest::testGetLckyNmbr() {
    Oracle* oracle = createGeekyOracle();
    assertEquals(42, oracle->getLckyNmbr());
  }

This unit test will fail to compile when you change the method name of the base class, which avoids you end up with the silly bug in your program.

If your test allocates your object under test on the stack (e.g. for simplicity), you could use the following alternative:



  void GeekyOracleTest::testGetLckyNmbr() {
    GeekyOracle oracle;
    assertEquals(42, ((Oracle*) &oracle)->getLckyNmbr());
  }

However, since this degrades the readability of the test, I prefer to add a static helper method to my unit test, and write the test as follows:



  static inline Oracle* p(Oracle& foo) {
    return &foo;
  }

  void GeekyOracleTest::testGetLckyNmbr() {
    GeekyOracle oracle;
    assertEquals(42, p(oracle)->getLckyNmbr());
  }

Although I used C++ in my example, this technique should work for all statically typed languages. Advocates of dynamically typed languages claim they can compensate the lack of types by unit testing their classes properly; I wonder how they avoid this kind of problems, though.

Testing Psi

2007-10-01T00:00:00+02:00

While the last bugs are being squeezed out of Psi 0.11’s release candidates, work on 0.12 has already begun. One thing I’m excited about as a developer is the fact that we’re making the Psi codebase `testable', which has some nice consequences.

The first part of our effort to make Psi testable is creating unit tests for most of our classes (using CppUnit), which should provide quick checks that our classes behave (and keep behaving) the way they should. However, the fact that Psi was not designed to be testable from the ground up makes this a non-trivial job, requiring quite some refactoring. This is something that will happen incrementally over time. A side effect of this refactoring will be an increase in modularity, allowing us to use different frontends (e.g. Cocoa) for the same code.

The second part we’re working on is creating UI tests for parts that are not automatically testable. For this, we are decoupling all our dialogs from the rest of the application, such that we can create a standalone application that allows you to toy with the dialog. This decoupling not only means that it becomes very easy to test dialogs with your own scenarios (without having to start Psi, connect to a server, …), but also means that the dialogs become completely reusable for other applications.

We still have a long way to go until every part of Psi is testable (either automatically or not), but I am very much looking forward to the day where all of our code is nicely decoupled, and we can run make check on every commit to ensure that we didn’t break anything silly.