Once the test inputs have been selected and the expected outputs determined, test execution can begin. The cost of test execution is heavily influenced by two factors: controllability and observability. Controllability refers to the ease with which inputs may be supplied to the device-under-test (DUT). Observability refers to the ease with which outputs from the DUT may be observed.
Controllability and observability are useful in two important ways:
- Controllability and observability help testers to predict test automation problems. Consider a test configuration using a GUI-based HMI to test a device. Here the controllability is poor: the inputs are supplied with mouse clicks and typed text. The observability is also poor: outputs are displayed on screen, as text and images. With this configuration, it is easy to run a few simple tests manually. It is costly, however, to run a large number of tests, especially if the tests must be repeated for multiple device versions.
- Controllability and observability help testers identify opportunities for cost reduction. For example, a device could be tested with custom software rather than a GUI-based HMI. The custom software will generate packets to send to the device similar to those sent by the HMI. Similarly, the custom software will monitor packets returned from the device, to extract the outputs which were displayed by the HMI. While the cost of developing the custom software must be considered, the tradeoff is clear: will the improvements in controllability and observability justify the development cost?
Controllability and observability have long been understood by VLSI testers and are commonly discussed in undergraduate textbooks. The ideas are largely unknown to device testers, but can be used to reduce the cost and improve the effectiveness of device testing.