CRII: SHF: Toward Sustainable Software for Science - Implementing and Assessing Systematic Testing Approaches for Scientific Software

Scientific software is widely used in science and engineering. In

addition, results obtained from scientific software are used as evidence in

research publications. Despite the critical usage of such software, many

studies have pointed out a lack of systematic testing of scientific software.

As a result, subtle program errors can remain undetected. There are numerous

reports of subtle faults in scientific software causing losses of billions of

dollars and the withdrawal of scientific publications. This research aims to develop

automated techniques for test oracle creation, test case selection, and develop

methods for test oracle prioritization targeting scientific software. The intellectual

merits of this research are the following: (1) It advances the understanding of

the scientific software development process and investigates methods to

incorporate systematic software testing activities into scientific software

development without interfering with the scientific inquiry, (2) It forges new

approaches to develop automated test oracles for programs that produce complex

outputs and for programs that produce outputs that are previously unknown, (3)

It develops new metrics to measure the effectiveness of partial test oracles

and uses them for test oracle prioritization, (4) It extends the boundaries of

current test case selection to effectively work with partial or approximate

test oracles. The project's broader significance and importance are (1)

produces a publicly available, easy to use testing tool that can be incorporated

into the scientific software development culture such that the testing

activities will not interfere with ?doing science,? (2) recruits Native Americans

and women into computer science research, (3) develops a new higher level

undergraduate course titled ?Software development methods for Scientists?

targeting senior level undergraduate students in non-CS disciplines.

This project develops METtester: an automated testing framework that can effectively

test scientific software. This testing framework analyzes the source code of

the program under test and utilizes machine learning techniques in order to

identify suitable test oracles called metamorphic relations (MRs). Then, it automatically

generates effective test cases to conduct automated testing based on the

identified MRs using a mutation based approach. After that, it creates a

prioritized order of MRs to be used with testing in order to identify faults as

early as possible during the testing process. Finally, METtester conducts

testing on the program under test using the prioritized order of MRs with the generated

test cases.