Programmer Musings

Review of Perl Testing: A Developer's Notebook

Perl Testing: A Developer's Notebook
Ian Langworth and chromatic
O'Reilly, 2005

This is the first book I have read from the new O'Reilly Developer's Notebook series. Although the style of these books probably won't work for every topic, it works particularly well for a book on testing. Each chapter is organized as a set of labs. These labs walk through some portion of the topic with the goal of developing practical skills.

Each lab revolves around a task. They don't delve into theory, they focus on describing how you perform the task. Unlike O'Reilly's Cookbook series, this is a directed set of tasks. Much like labs in a class, the labs in this book start with basic testing skills and work to build more advanced testing skills. The goal of this set of labs is to help you become proficient in the skills needed for doing quality unit testing of Perl code.

I have been doing testing with Perl for some years now, so the first few labs were review. But, the labs swiftly escalated to more advanced topics that I was either light on or had not explored at all. In addition to labs about Test::Simple, Test::More, and Test::Harness, the book covers:

• testing the POD documentation in your modules
• coverage testing
• testing exceptions generated by code
• testing warning generated by code
• building testing libraries to simplify testing
• mocking objects
• overriding system calls
• testing databases
• testing websites

and many more topics related to testing. This little book is packed with practical information and labs that help you improve the skills you need to build quality testing for your Perl code.

I really don't expect this book to become a well-worn reference book that stays near my desk at all times. However, I don't expect to let it get far away. Since few of us need to use all of our testing skills on every project, some skills become rusty over time. Anytime I find some of my testing skills have gotten rusty, I expect to grab this book and run through the appropriate labs to sharpen the skills I need.

If you want to know why you should be unit testing your Perl code, there are many books out there that will explain. If you are looking for an explanation of Test Driven Development, this is not the book to read. Once you have decided that testing is important to you and your projects, this is the book you need. I would not say that there are no other resources for learning this material, but this may be the fastest way to build the appropriate skills that I've seen.

It also helps that the authors have a very relaxed style that helps you feel more like they are working along side you than lecturing at you. The titles of the sections are very casual (How do I do that?, What just happened?, etc.) This makes it easy to read the next part and get back to working on the meat of the lab.

I would recommend this book to anyone writing Perl code. We all need to be doing testing to improve our code and our designs. The key difference between good testing and bad testing is the skills to make every test count. This book teaches those skills.

Version Control and Test Driven Development

Almost fifteen years ago, I stumbled across the concept of version control in the form of RCS. I wasn't real sure that this tool was going to be useful for me. After all, it required more work that did not contribute directly to producing code. In order to try it out, I began using RCS to keep up with the source on a few projects I was working on at home. In a few days, I was hooked. I immediately began pushing version control at work despite opposition from the other developer (my boss).

The key to version control for me was a change in my work habits that version control allowed. I have recently come to the realization that Test Driven Development has had a similar effect on my work habits.

The realization that drove my use of version control was actually pretty pedestrian. I could experiment with new ideas or designs with almost no risk. I I wanted to see if a new algorithm would work better, I would try it out. If the new idea didn't work out, I could always go back to an earlier version of the code. If I wanted to try out new data structures, I could do the same thing. This was incredibly liberating! I could experiment without worry about messing up code that I had slaved over.

Before I had used RCS, I could do something similar by backing up the old code before trying out my idea. Then, if things didn't work out, I could throw everything away and go back to the old code. But with RCS, I could do backups on small changes and move forward and back through changes with relatively minor effort. Later, I found CVS and my reliance on version control became even stronger.

For the last couple of years, I've been experimenting with unit-test frameworks and Test Driven Development. A lot of the time, the tests really seem to put the fun back in programming. I seem to be moving almost as fast as I did without the tests, but my designs and code appear to be cleaner and more solid.

Although several people have said it different ways before, this is the first time I realized that TDD is giving me the same benefit I first recognized with version control. I can experiment (or refactor) again with almost no cost. If my unit tests are good enough, I can completely refactor my code or re-implement any part of the algorithm with confidence. If I broke anything, the tests will show me. If the tests don't give me the confidence to refactor, my tests aren't complete enough.

This has given me more incentive to work on the TDD way of doing things. It took a little while for me to reach the point of using version control for every project no matter how small. I guess it will probably take a while for me to get to that point with testing as well.

Unit tests that should fail

I was doing a little research on the Java JUnit test framework and ran across the article The Third State of your Binary JUnit Tests.

The author points out that in many test sets there are ignored tests as well as the passing and failing tests. As the author says, you may want to ignore tests that show bugs that you can't fix at this time. He makes a pretty good case for this concept.

The Perl Test::More framework takes a more flexible approach. In this framework you can also have skipped tests and todo tests in addition to tests that actually need to pass. These two different types of tests have very different meanings.

Skipped tests are tests that should not be run for some reason. Many times tests will be skipped that don't apply to a particular platform, or rely on an optional module for functionality. This allows the tests to be run if the conditions are right, but skipped if they would just generate spurious test failures.

Todo tests have a very different meaning. These tests describe the way functionaly should work, even if it doesn't at this time. The test is still executed. But, if the test fails, it is not treated as a failure. More interestingly, if a todo test passes, it is reported as a failure because the test was not expected to pass. This allows bugs and unfinished features to be tracked in the test suite with a reminder to update the tests when they are completed.

Unlike the idea in the referenced article, these two separate mechanisms don't ignore tests that cannot or should not pass. Instead, we can document two different types of non-passing tests and still monitor them for changes.