Advances in CIAO Interactive Analysis

Over the past decade there has been a resurgence of interest in interpreted programming languages, and with the release of CIAO 2.0 last December this trend has at last infiltrated the Chandra data analysis system. Scripting languages like Python, Tcl, and Perl have become enormously popular, due in large part to their promise for increasing programmer and end-user productivity [1]. While each of these languages has strengths, and some actually find use elsewhere in CIAO, after careful evaluation it was apparent that the less widely-known S-Lang language ([2], [3]) was more suitable to both our software architecture and the types of analysis prevalent in the user community. The integration of S-Lang presents a range of exciting new features to CIAO users:

$\bullet$ global and local interpreted variables, and multidimensional arrays (up to 7D)

$\bullet$ branching and looping, and programmability with user-defined functions

$\bullet$ string and numeric datatypes, structures, and a limited form of pointers

$\bullet$ built-in arithmetic and mathematical functions, which operate transparently on arrays

Within CIAO tools S-Lang is accessed through a supplementary library layer, varmm, which gives the user additional capability to define structured variables directly from disk files, as well as enabling existing CIAO applications to access S-Lang variables and other features with a bare minimum of new development effort. While in release CIAO 2.0 only Sherpa, ChIPS, and the beta version of GUIDE utilize the S-Lang and varmm libraries, plans are being laid to employ them within other applications (e.g., table calculator in Prism) and hook in important numerical algorithms (e.g, FFT and convolution) so that they may be utilized interactively with S-Lang variables.

Together these new features are comparable to those typically found only in expensive commercial packages like IDL and MatLab (though in fairness it must be noted that since CIAO is considerably younger it cannot hope to match their breadth of functionality), and dramatically increase the analytical power and extensibility of CIAO.

Examples

To give a better feel for these new features let's look at a few examples. First we'll read a file into an S-Lang structure and display its contents with the print command:

chips> cd /my/data/files

chips> pldata = readfile("phas.dat")

chips> print(pldata)

filename = phas.dat

path = /my/data/files

filter = NULL

ncols = 2 nrows = 124

col1 = Float_Type[124]

col2 = Float_Type[124]

Here we used ASCII, but FITS tables and images, IRAF/IMH, and QPOE files are also supported, with the full range of CXC DataModel filtering and binning.

Here's an example where we skip the first 350,000 rows of a binary table and use the DM syntax to select only 3 columns:

sherpa> btab = readbintab

("acisf00459N002_evt2.fits.gz[cols

pha,pi,time]", "",350000);

sherpa> print(btab)

filename = acis.fits

path = /my/data/files

filter = [cols pha,pi,time]

ncols = 3

nrows = 241093

pha = Integer_Type[241093]

pi = Short_Type[241093]

time = Double_Type[241093]

To plot the ASCII data specify its column fields to the plot command:

chips> plot x pldata.col1 y pldata.col2

One might smooth the Y axis, for example, and replot the result as follows:

chips> pldata.col2 = log(pldata.col2 + 1.0) + 10

chips> redraw

As a simple example of the extensibility and customization now possible in CIAO consider ignoring points below a certain y-value. One way to do this would be to write a new function

define ignoreLowPoints(data, floor) $\{$

variable points = where(data.col2 > floor);

data.col1 = data.col1[points];

data.col2 = data.col2[points];

$\}$

that could then be used to extend the functionality of ChIPS (or Sherpa):

chips> evalfile("ignoreLowPoints.sl")

The new function can now be used wherever appropriate:

chips> p2 = readfile("phas.dat") chips> ignoreLowPoints(p2,30) chips> plot x p2.col1 y p2.col2

A less trivial example of the extensibility of CIAO is the Grating User Interactive Data Extension, or GUIDE, which is a package of S-Lang/varmm scripts -- glued to Sherpa with a small amount of C/C++ code -- written to assist in the analysis of high-resolution spectra. While GUIDE is beta software, we already see that embedding S-Lang in CIAO enabled us to write this package much more rapidly than we could have otherwise, and to involve scientists more directly in the process. By making CIAO extensible we have removed a major development bottleneck: users need no longer wait for the Chandra development staff to code new releases of CIAO, as new analysis capability may be added to CIAO, with relative ease, by virtually anyone. We believe that this approach will prove valuable for future internal CIAO development and for users who may want to create their own CIAO extensions.

References and Web pages:

[1] J.Ousterhout, ``Scripting: Higher Level Programming for the 21st Century", IEEE Computer, March 1998

[2] J. Davis, http://www.s-lang.org

[3] S.Doe, M. Noble, R.Smith, ``Interactive Analysis and Scripting in CIAO 2.0", Astronomical Data Analysis Software and Systems X, November 2000

- Michael S. Noble