Unit Testing is a methodology that allows you to test your designs, to confirm that your design still works after changing it, and to quickly isolate the cause of any failures. Test Driven Development (TDD), on the other hand, is a methodology that puts testing at the centre of the design process. They're not the same thing, but there is often confusion about the boundary between the two: where one stops, and the other starts.
There are, of course, many different definitions of what 'unit testing' actually is, and what constitutes a 'unit test'. However, there are a number of common elements:
- unit tests are low-level. They focus on a small part of the design: a single module, or a small number of modules with a well-defined function
- unit tests are normally written by the designers themselves, using a framework that automates the execution of large numbers of tests
- unit tests should be relatively fast
Unit testing isolates each part of the system and confirms that those individual parts behave correctly. It has a number of major advantages over more traditional development flows:
- A unit testing framework allows tests to be automated, and repeatable. The tests should be run frequently, and should complete quickly, to give a continuous indication of the state of the code base as development progresses
- When used appropriately, a unit test is the specification for the module being developed. In a TDD flow, for example, the test is written before the module itself, and the module is known to be correct when the test passes. During development, the module can be refactored (in other words, rewritten), and the new code can be tested against the original test
- In a hardware context, parts of the system can be written as behavioural models before the detailed RTL is available, and the same tests can be used to verify both the behavioural and RTL models
- By testing individual units, unit testing finds problems early in the development cycle. The complexity of the system will increase exponentially as modules are combined, making it much more difficult to detect and correct errors
The ideas behind unit testing have been around since at least the late 1980s, with Smalltalk's SUnit test framework. This was the basis of many later frameworks (collectively known as the 'xUnit' tools). The unit testing philosophy quickly led on to Extreme Programming (XP) and TDD, and is a fundamental part of agile development in general. However, while these methodologies have been widely used in software development since the late 1990s, they have been essentially unused in hardware development (with the occasional exception of Continuous Integration tools such as Jenkins). One issue with the adoption of these techniques in hardware development is that some of these ideas are difficult to translate to a hardware environment. There are some areas which are particularly relevant, and some which are less so:
- The test language. Unit tests are normally written in the same language that the developer used for the original coding. However, this isn't a requirement, and isn't practical for HDL development, for a number of reasons
- The unit size and complexity. There is a common view that the units to be tested should be the 'smallest testable unit', or the smallest part of the design which can be tested. This, again, is not likely to be practical in HDL development, and you'll need to take a more pragmatic approach to the definition of your 'units'
- One issue that comes up in software testing is the distinction between 'solitary' and 'sociable' units. This is a distinction that makes little sense in a hardware environment, and can be ignored
The test language
VHDL and Verilog are not high-level general-pupose languages, and are not suitable for building test frameworks, and are not popular (11). A more fundamental problem is that the limited synthesisable subset of VHDL or Verilog generally used and understood by EEs is not suitable for testing hardware. This is significant, given that unit tests should be written by the original developer (more on this below).
However, there is no fundamental requirement that a unit test should be written in the same language as the unit itself, as long as the language allows direct access to the unit being tested (unit tests are 'white box' tests). This is the case for Maia, which allows direct instantiation of the UUT, and direct access to the ports and internal signals of the UUT.
The smallest testable unit
When developing software, there is a lot of latitude in the definition of what the 'smallest testable unit' might be. It might be an entire class, for example, or just an individual method within that class. However, the smallest testable part of a hardware design is always going to be very much smaller than the smallest testable part of a software design: it could be a register, or even a gate, for example. It is not going to be practical or constructive to test at this granularity, so some thought is required here.
In a hardware context, the 'smallest testable unit' will, practically speaking, be a module with a well-defined interface that can be written in a short period of time (as a rule of thumb, perhaps a day or so). This may be a single leaf module, or one which instantiates a small number of other modules. All that actually matters is that any error reported by the automated test system should be easy to pinpoint and easy to fix.
This necessarily means that the unit being tested must be relatively small. Identifying and fixing an error in 300 lines of code, for example, should be straightforward. However, if your test framework reports an error in a 3,000-line module, it might take a week and a rewrite before you can even identify the cause of the error. This is, fundamentally, what unit testing is all about: if a part of your design fails after some change in the system, then the unit tests should be able to immediately identify the smallest part of the system that needs to be fixed, and that can be fixed quickly.
Who writes the tests?
This is an argument that was settled in the software development community over 20 years ago: you write your own unit tests. As Martin Fowler writes, when discussing unit testing and XP:
Unit tests are usually written these days by the programmers themselves using their regular tools - the only difference being the use of some sort of unit testing framework [...] I say "these days" because this is certainly something that has changed due to XP. In the turn-of-the-century debates, XPers were strongly criticized for this [view] as the common view was that programmers should never test their own code. Some shops had specialized unit testers whose entire job would be to write unit tests for code written earlier by developers. The reasons for this included: people having a conceptual blindness to testing their own code, programmers not being good testers, and it was good to have a adversarial relationship between developers and testers. The XPer view was that programmers could learn to be effective testers, at least at the unit level, and that if you involved a separate group the feedback loop that tests gave you would be hopelessly slow. Xunit played an essential role here, it was designed specifically to minimize the friction for programmers writing tests.
Can you test with only unit tests?
No, for at least two reasons.
First, you will, at some point, have to start testing at higher levels in the hierarchy, to test communication between your lower-level modules. This is 'Integration testing'. This is not fundamentally different to unit testing, and can be done with the same tools, but the objects you are testing are no longer 'units', so the process has a different name. More fundamentally, however, testing modules at a higher level in the hierarchy will take much longer than testing leaf modules. Your should therefore run separate sets of tests: lower-level unit tests which can be run quickly and frequently, and higher level integration tests which could be run overnight, for example.
Second, unit and integration testing should be seen as part of an overall test framework. You write tests for your own code to convince yourself that your code works, with a mindset of how do I make this work? During later testing you, or your verification or QA department, will have a completely different mindset: how do I break this?