Chandra Interactive Analysis of Observations (CIAO) consists of a
suite of tools and applications developed for the analysis of
Chandra X-ray Observatory data. With its unprecedented technical
capabilities and science potential, Chandra requires a novel,
flexible, multi-dimensional set of tools for data analysis. This
mission provides 4-dimensional data (2 spatial-dimensions, time,
& energy), each containing many independent elements. CIAO
tools are able to filter and project the 4-dimensional event
data to manageable sizes and convenient arrays in an easy and
flexible manner. CIAO is a mission independent system and can
be used for data analysis of other X-ray and non- X-ray
Since launch there have been several major CIAO releases. A major step
in the preparation of each CIAO release is scientific and
functional testing of the software. This is a joint effort by
the Chandra X-ray Center Data Systems (DS) and Science Data
Systems (SDS) teams. We highlight here the stages of CIAO
testing and the procedures we follow at each stage.
CIAO Tool Development
The CIAO tool development process begins many months before a planned release. SDS scientists identify and evaluate science and calibration milestones for Chandra data analysis, and outline the priorities and goals for tool and application development for a CIAO release.
During the software development process scientists and developers
interact closely to assure CIAO tools are designed and built to
the specified requirements and that they follow CIAO system
Once implemented, unit tests are developed and run to assure proper implementation. We recognize that more rigid tests are required though, and science and development teams combine efforts to ensure this goal is met.
CIAO Release Testing
CIAO releases involve 4 testing stages. These are: (1) unit testing, where detailed tests of CIAO tools verify and validate the correct implementation of scientific algorithms; (2) mini-testing, where the new functionality of the system is bundled and tested across ported platforms; (3) regression testing, where baseline system tests are run on all ported platforms to validate the system; and (4) package testing, where tar files are downloaded from the public web page to validate web links, instructions, and the software installation.
Science Unit Testing
Unit science tests are developed for each new or updated tool as they are completed by the software team. It is in this testing phase that scientists run tools they have specified for the first time. Unit tool testers verify and validate that the tool was implemented as they envisioned with correct and robust algorithms to meet the science goals. They also run the tools in science data analysis scenarios where the output file of one tool is provided as the input to the next in a series of tool steps organized to meet specific science goals. It is these science scenarios that really separate science testing from the software team testing.
A tool that is run standalone (as in development unit test) is less
rigorously tested than one that is run as part of a data
analysis thread where tool's functionality is judged, not only
the tool execution and creation of output data, but whether the
data product provides all of the input required to do further
analysis with other tools. The output products of one tool are
often the input to many tools that utilize those data to do
further analysis. It is sometimes found that there are keywords
missing in a file header, or table columns written by a tool
have missing information or are problematic with other programs
that read those data products. Sometimes the software upgrades
are in software libraries. In those cases we identify tools
that explicitly utilize the enhancement or bug fix for this
level of testing.
A science tester first outlines a plan based on the requirements for
the tool, their expectations of the tools flexibility, and the
range of parameters and data the tool was specified to work
with. They have often already provided baseline datasets and
use-cases for the developers unit test. At this stage the goal
is to catch missing details in the specification or
implementation of the tool.
Once there is a plan, the scientist runs the commands or builds a script to test the tool. Often iteration with the development team starts at this point. E-mails are sent to an internal alias where they are being stored; They are accessible from an e-mail archive. During this process questions are asked, issues are uncovered, and interpretation of requirements is discussed. Developers address reported problems and make bug fixes and enhancements to the tool. E-mail messages also seed bug reports that track problems.
The unit testing stage is completed either with successful completion
of the unit test, or by identifying that the uncovered problem
would require a longer time-frame for solving. Each unit tester
captures the testing steps along with the input and output data,
and prepares a unit test worksheet.
The test worksheet is a means to communicate the statusof unit testing, and when success is indicated, it is the trigger to have the test added to the regression suite. Tests are run across all platforms of a CIAO release to verify that there are no porting issues.
Each test worksheet contains information on the tool status, level of
completion of unit testing, and a log of the test session and/or
a pointer to a test script. Furthermore, it contains pointers to
input data, and to datasets created as output from the tool. The
test worksheets are then used as input to build automated
regression test scripts. Scientists work to validate the tool
on their computer platform of choice. The work to validate the
tools across ported platforms is for most part performed
When unit testing is completed, and the issues related to a given tool
are resolved, the tool lead indicates that testing was
successful and the tool is ready for release. Sometimes it is
at this point that a tool is rejected from the upcoming release,
or if the tool is required for a successful release, the
schedule is extended to allow time to address the problem that
must be solved.
Over the few weeks of testing following a code freeze, theCIAO system is built bi-weekly to bring tools that have had problems addressed back to the release configuration. They are re-tested by unit testers until they are ready to complete a new test worksheet. At this time, a targeted regression test is built that contains a sample of previously released CIAO tools, the new or modified tools, and tools that exploit the functions added to libraries. This test set is small so it runs quickly and is aimed at exercising the highest risk areas of the software. Since the test script consists of a subset of what would become the full regression test, we refer to this as the mini-test. The mini-test is run once the first build of the system is completed after a code freeze, and almost weekly during the period of rebuild and science unit testing to ensure software stability over all platforms.
Later in the release cycle, the software tests outlined in the test worksheets are added to the regression test script. The output results of the current execution are compared to data saved from previous runs. Some of these data are from the last CIAO release, while new tests use data provided with the test worksheet. When the mini-test is upgraded with the worksheet tests, we refer to it as the mini+ test.
Full regression testing
Late in the release schedule, a full regression test is run and results are verified on all platforms. This test is the culmination of all test scripts written since the start of CIAO development. New tests outlined in the test worksheets are added for each release. This test is usually run once, late in the release cycle, across all ported platforms, and the results verified against previous runs of the tools.
An important part of this step is the overall test evaluation by the SDS testing lead. The unit tests, mini tests, and full regression tests are reviewed and evaluated, and upon successful completion, the test lead signs off that science testing is complete and the tools are ready for release. With sign-off of science testing, we move into download testing which is the last step in the release testing
In download testing, the system as packaged for release is posted to a hidden public web page. Scientists assigned to verify downloads on various platforms supported by CIAO, practice a download and installation of the system. Installation instructions are reviewed, web links and tar files validated, and the system is spot checked. The development team provides a download test script which runs a sample set of CIAO tools. The script reports whether the test is successful. At this stage we evaluate success based on the execution of a tool and not a comparison to previous results.
Both science and development leads evaluate the readiness for release, and upon successful completion of download testing give their concurrence to make the system available for public download.
The test procedures described here, have been developed over the 6 years of the Chandra mission. This framework has assisted us in making 3 major releases and 6 minor releases of CIAO. We generally support ports to Sun, several flavors of Linux, and Mac systems. CIAO consists of approximately 80 command line tools and several large applications. Our release schedule is well defined.
Our process is sometimes adapted to specific circumstances but well within the framework described here. We have found these steps very helpful to achieve stable and successful CIAO releases.