This case study outlines a design proving project where the specification for a new complex IT system was modelled in order to improve the specification and build a test harness for the final system. The model implemented the full specification but on a simplified platform with stubs for the external actors interacting with the planned system. The model mocked communications with those stubs, interactions with users and the data in the system.
Enhancing the Specification
Improvements in the specification were expected to come from the following areas:
- Removal of ambiguity: clarifications from the business analysts during implementation.
- Demonstration of feasibility: building a working model demonstrated that the specification could be implemented.
- System and use case testing: testing of the model enabled demonstration that the specification met the objectives of the system.
- End user interaction: a user interface to the test harness allowed end users to interact directly with the model and give feedback on the speciation.
- What-if analysis: by allowing different rules and values to be investigated within the model.
The Model and Test Harness
The diagram shown highlights how the test harness interacts with the model and how the users access the test harness and model. The model contains a database which is set up with initial values ‘pre-test’ via the test harness and is then queried ‘post-test’ to help establish Pass/Fail.
The Advantages of Using Accellera SystemC
The Accellera SystemC standard was used to allow modelling of multiple parallel processes within the model, interacting with multiple stubs and the test harness in parallel. The user interface allowed the user to run pre-defined tests in the test harness, or to write new tests, and to analyse the results. In this way the test harness allowed business analysts and end users to fully investigate the specification via the model. The model had a set of variables and rules (the ‘configuration’ in the diagram) configurable from the user interface that allowed the business analysts to perform a number of what-if analyses to improve the specification.
The test harness is also used to generate a set of user acceptance tests for the final system. Working with the business analysts, a set of use case scenarios are generated which are necessary and sufficient to ensure a correct implementation. The test harness can then generate the expected outputs and end state for the system for each test in turn, thus forming a full test specification.
Finally, it is worth noting that the test harness can potentially be used in two additional ways:
- As a test oracle for the eventual full implementation of the specified system
- As an integration test for the external actors by replacing the stub with the real actor.
Project Deliverables / Benefits
The project delivered the following:
- A working, executable model of the complete specification.
- A test harness which demonstrated that the specification fully met the required use cases.
- A user interface that allowed the end users and business analysts to run pre-defined tests, run their own tests and perform what-if analyses.
- A test specification defining inputs and expected outputs.
The modelling activity identified a large number of holes and ambiguities in the specification and the what-if analyses allowed a number1 of specification improvements to be made. This should result in both time and cost savings in the actual implementation phase that easily justify the modelling costs.
The test harness is expected to be reused with the actual implementation and to enable independent integration testing for external actors.
Note 1: The exact numbers are commercially confidential.