Automated testing is inherently difficult. Worse, there is spotty coverage of the topic in university courses and experienced personnel are scarce in industry. As a result, there is a lot of confusion about the most important concepts and techniques.
Suppose that we want to perform automated testing of a networked SCADA device: the device-under-test or DUT. Typically, the following six steps are required:
- Build the test harness. Configure a PC to send packets to, and receive packets from the DUT. The PC may use a commercial HMI or custom software to send and receive the packets. The PC may also need specialized hardware to connect to the DUT’s control I/O lines. More than one PC may be involved.
- Generate the test inputs. There are always a vast number of possible inputs. The tester must choose intelligently among them.
- Determine the expected output for each of the test inputs. Testing always involves a means for determining the correct or expected output for each test input. This mechanism is known as the test oracle. Because automated testing usually involves many tests, an automated test oracle is desireable.
- Execute the test cases, capturing the outputs. The outputs may captured in a file or in program variables in the test program in the PC.
- Compare the actual and expected outputs. If the output is a single number, then the comparison is easy to carry out. If the output is a screen image, then the comparison can be quite challenging.
- Evaluate the test results. If some tests fail, the tester must determine whether they are severe enough to delay product release. Even if there are no errors, the tester must consider whether the testing was sufficiently thorough.
In some cases, steps 4 and 5 are sequential: all the test cases are run and all the outputs captured. Only then is the comparison begun. In other cases, steps 4 and 5 are interleaved: a test case is run and its output is immediately compared to the expected output.
Note that steps 4, 5, and 6 are carried out every time the tests are run. As a result, these steps are often the best target for reducing test costs.