Oracle (software testing)
In computing, software testers and software engineers can use an oracle as a mechanism for determining whether a test has passed or failed.[1] The use of oracles involves comparing the output(s) of the system under test, for a given test-case input, to the output(s) that the oracle determines that product should have. William E. Howden first defined and used the term in this sense in his article "Introduction to the Theory of Testing".[2] Additional work on different kinds of oracles was explored by Elaine Weyuker.[3]
Oracles often operate separately from the system under test.[4] Method postconditions are commonly used as automated oracles in automated class testing.[5] The oracle problem is often much harder than it seems, and involves solving problems related to controllability and observability.[6]
Common oracles include:
- specifications and documentation[7][8]
- other products (for instance, an oracle for a software program might be a second program that uses a different algorithm to evaluate the same mathematical expression as the product under test)
- a heuristic oracle that provides approximate results or exact results for a set of a few test inputs[9]
- a statistical oracle that uses statistical characteristics[10]
- a consistency oracle that compares the results of one test execution to another for similarity[11]
- a model-based oracle that uses the same model to generate and verify system behavior[12]
- a human oracle (i.e. the correctness of the system under test is determined by manual analysis)[6]
References
- ↑ Kaner, Cem; A Course in Black Box Software Testing, 2004
- ↑ In: Miller, Edward; and Howden, William E.; Software Testing and Validation Techniques, Long Beach, CA: IEEE Computer Society Press, 1978 (first edition), pp. 16-19
- ↑ Weyuker, Elaine J.; "The Oracle Assumption of Program Testing", in Proceedings of the 13th International Conference on System Sciences (ICSS), Honolulu, HI, January 1980, pp. 44-49
- ↑ Jalote, Pankaj; An Integrated Approach to Software Engineering, Springer/Birkhäuser, 2005, ISBN 0-387-20881-X
- ↑ Meyer, Bertrand; et al.; "Programs that Test Themselves", IEEE Computer, 42:9 (September 2009), pp. 46-55
- 1 2 Ammann, Paul; and Offutt, Jeff; "Introduction to Software Testing", Cambridge University Press, 2008, ISBN 978-0-521-88038-1
- ↑ Peters, Dennis K. (1995). Generating a Test Oracle from Program Documentation (M. Eng. thesis). McMaster University. CiteSeerX 10.1.1.69.4331
. - ↑ Peters, Dennis K.; Parnas, David L. "Generating a Test Oracle from Program Documentation" (PDF). Proceedings of the 1994 International Symposium on Software Testing and Analysis. ISSTA. ACM Press. pp. 58–65.
- ↑ Hoffman, Douglas; Heuristic Test Oracles, Software Testing & Quality Engineering Magazine, 1999
- ↑ Mayer, Johannes; and Guderlei, Ralph; Test Oracles Using Statistical Methods
- ↑ Hoffman, Douglas; Analysis of a Taxonomy for Test Oracles, Quality Week, 1998
- ↑ Robinson, Harry; Finite State Model-Based Testing on a Shoestring, STAR West 1999
Bibliography
- Binder, Robert V. (1999). "Chapter 18 - Oracles" in Testing Object-Oriented Systems: Models, Patterns, and Tools, Addison-Wesley Professional, 7 November 1999, ISBN 978-0-201-80938-1