Testing on High: Bottom-up versus Top-down Test-driven Development 40

Posted by ryan Mon, 19 Nov 2007 02:13:21 GMT

I recently talked to a number of Rails developers about their general approach to testing some new functionality they're about to code. I asked these developers if they found it to be more useful to start testing from the bottom-up or top-down. I suggested to them that, since Rails uses the MVC pattern, it's easy to think of the view, or user interface, as the "top", and the model as the "bottom". Surprisingly, nearly every developer that I asked this question of answered that they prefer to start from the bottom, or model, and test upwards. Nearly every one! I expected that I'd get a much more mixed response than I have. In fact, I think that the correct place to start testing is precisely at the highest level possible, to reduce the risk of building software based on incorrect assumptions of how best to solve a user requirement.

Bottom-up Testing

Bottom-up testing implies bottom-up design in TDD. In bottom-up design, a developer would probably consider the high-level objectives and break them up into manageable components that interact with each other to provide the desired functionality. The developer thinks about how each component will be used by its client components, and tests accordingly.

The problem with the bottom-up approach is that it's difficult to really know how a component needs to be used by its clients until the clients are implemented. To consider how the clients will be implemented, the developer must also think about how those clients will be used by their clients. This thought process continues until we reach the summit of our mighty design! Hopefully, when the developer is done pondering, they can write a suite of tests for a component which directly solves the needs of its client components. In my experience, however, this is rarely the case. What really happens is that the lower-level components tend either to do too much, too little, or the right amount in a way that is awkward or complicated to make use of.

The advantage of bottom-up testing is that, since we're starting with the most basic, fundamental components, we guarantee that we'll have some working software fairly quickly. However, since the software being written may not be closely associated with the high-level user requirements, it may not produce results that are necessarily valuable to the user. A simple client could quickly be written which demonstrates how the components work to the user, but that's besides the point unless the application being developed is a simple application. In such a case, the bottom-level of components are probably close enough to the top-level ones that there is little risk involved in choosing either the bottom-up or top-down approach.

Unless you're writing a small application, the code is probably going to have to support unforeseen use cases. When this comes as a result of ungrounded assumptions about the software that's already been written, this can mean a lot of rework. I can tell you from experience, once you realize that your lower-level components don't fit the bill for the higher levels in the system, it can be quite a chore to go back and fix, remove, or replace all of that unnecessary or incorrect code.

Top-down Testing

Top-down testing implies top-down design in TDD. Following the top-down approach, the developer will pick the highest level of the system to be tested; that is to say, the part of the system that has the closest correlation to the user requirements. This approach is sometimes referred to as Behavior Driven Development. Whatever it's called, the point is that you test the most critical parts of the application first.

Since software is often written for human users, the most critical parts usually involve the front-end as it relates to the value being provided by the system being developed. When testing from the top-down, the effort is the inverse of bottom-up testing: Instead of spending a lot of time thinking about how the components to be developed will be used by other components to be developed, the focus is on how the user needs to interact with the system. Testing involves proving that the system supports the required usability. For an application with a graphical front-end, this might involve testing for a minimal version of that front-end.

The disadvantage of top-down testing is that you can end up with a lot of stubbed or mocked code that you then have to go back and implement. This means it might take longer before you have software that actually does something besides pass tests. However, there are ways that you can minimize this sort of recursive development problem.

One way to minimize the time between starting development of a feature and demonstrating functionality that is valuable to the user is to focus on a thin slice of the overall architectural pie of the application. For example, there may be a number of views that need to be implemented before the system provides some major piece of functionality. However, the developer can focus on one view at a time, or one part of the view. That way, the number of components that need to be implemented before the system does something useful is small; ideally, one component in each architectural layer that I need build out, and often times only a part of the overall functionality of each component.

Another way to minimize the amount of time before the system does something useful is to code a small bit of functionality without worrying about breaking the problem up into classes until you have some tested, working code to analyze. You can then use established methods for refactoring to bring the code to an acceptable level of quality.

The advantage of top-down testing is that you write functionality that solves the most critical functionality first. This generally means starting development at a high level. When the system eventually does something besides pass tests, what it does will provide value to its users. Additionally, because development starts at a high level, the code that is written is based on the current understanding of the problem, and not on assumptions. This guarantees that the tests and code that are written are not superfluous.


The challenge with top-down testing is that you must be highly disciplined to ensure that the code you write is being refactored and is properly evolving into a cohesive domain model for the application. This is compared with bottom-up testing, where you start with the domain model and build your system around it. Either way, you're going to be refactoring code. The difference is in where the time in refactoring is spent. In my experience, when doing bottom-up testing, more time is spent correcting incorrect assumptions about how the domain model will be used than on actually improving code that already works to solve the user requirements. In order to avoid making assumptions about the code being written, it must be written at the level that is closest to providing actual value to the end-user. In so doing, the developer focuses on continuous refinement of code that already provides value, as opposed to speculative design and development.

Never Raise StandardError Directly

Posted by ryan Tue, 21 Nov 2006 06:40:00 GMT

It's not a good idea to raise StandardError directly from your Rails application, particularly when using an exception as a way to indicate a recoverable error from an ActiveRecord transaction. In such cases, it's better to subclass from StandardError and raise the subclass instead. If you use StandardError directly, you may find that your system is catching and recovering from errors that are related to faulty logic rather than broken business rules.

For example, consider the following bad code:

def some_method(x)
  x[x.length] + 1
  raise StandardError if x[0] < 2

  a = [1, 2]
  some_method( a )
rescue StandardError
  puts "Bad value for index 0: #{a[0]}"
This code does nothing useful, but assuming that it did, and assuming that you're catching StandardError for the case when the first index of the given array has an invalid value, you're in for a surprise. In fact, the error that would be caught by the rescue clause above is thrown by the code on line 2. In fact, the actual error thrown by this line is an instance of NoMethodError, which is a subclass of NameError, which is a subclass of StandardError.

If you wrote a unit test for the some_method method, and asserted that StandardError was raised, it would pass incorrectly. This is especially misleading if you are writing a test for a controller action and asserting that, upon some error condition, the application redirects the user to a page displaying that error. I find that I don't explicitly raise many errors from my application code, except when I wrap the code in a controller action inside an ActiveRecord transaction. The only way to roll back the transaction is to raise an exception from within the transaction and rescue from it outside, such as:

class SomeController < ApplicationController
  def some_action
      SomeRecordClass.transaction do
        a = [1, 2]
        some_method( a )
    rescue StandardError
      @error_message = "Bad value for index 0: #{a[0]}"
      render 'some_template'
    redirect_to some_url
In the above code, you can't avoid raising an exception if you want to roll back the transaction. In this case, rather than rescuing from StandardError, rescue from an application-specific exception that inherits from StandardError. This way you know that you're recovering from an application error, and not faulty logic in the code.

Evolutionary Development in XP 1

Posted by ryan Tue, 14 Nov 2006 22:42:59 GMT

This has probably been said before, but it's extremely important: Evolutionary development is all about coding what you know at the time you've picked the story card off the wall. More often than not, the harder you think about what the "proper" design might be for the solution, the further away from the target you'll actually be. When presented with a problem, don't stop and think about what might be the most elegant way to design the solution - Just Code It!

Presumably, if you're picking the story off of the wall, some time has been spent mulling over what the story means for the system under development with the necessary stakeholders. You should also have a pretty good idea of the existing pieces of the system that will be affected, as you've considered this when giving your point estimate. So, start coding for the functionality that you think you need.

As you write tests, and build functionality to make them pass, you'll probably run into pieces of the system that somehow don't jive with the new functionality that you're developing. Stop development of the new feature. Comment out the currently-failing test that you're trying to get passing. Start refactoring the existing code to make sense in light of the new requirements, but only as they concern your currently-failing test! Don't stop until you've got the existing system singing the sweet song of cohesion. Then, uncomment out the currently-failing test and make that baby pass!

The important thing to keep in mind when coding like this is that you should never consciously degrade the quality of the existing system in the name of making the new functionality "just work". Each time you finish a story, the system under development should be a cleaner, more complete facilitator of client value, and no more difficult to further-extend and enhance than it was when you started working on the card.