CHANGES IN CIAO 3.3

The statistics calculated by dmstat are now written to the parameter file as well as the screen. The parameter tools, such as pget, can be used to access these values. This means that it is no longer necessary to try to extract the calculated values from the screen output. For instance:

  unix% set mean = `pget dmstat out_mean`
  unix% set max = `pget dmstat out_max`
  unix% set maxloc = `pget dmstat out_max_loc`

get the mean, max value, and location of the max value from the parameter file. Further information is given in the "ACCESSING STATISTICS FROM THE PARAMETER FILE" section below.

CHANGES IN CIAO 3.2

Minimum and Maximum location for vector and array columns

The row numbers of the minimum and maximum value of a column are now correctly reported for vector columns (such as "chip", "sky" and "eqpos") and array columns (such as "phas" in a level 1 event file). For example:

  unix% dmstat "evt2.fits[cols chip]"
  chip(chipx, chipy)[pixel]
      min:        ( 2 2 )       @:        ( 959 489 )
      max:        ( 1023 1023 )         @:        ( 73 212 )
  ...

and

  unix% dmstat "evt1.fits[cols phas]"
  phas[adu]
      min:        [ -3477 -3329 -3476 -3476 39 -3475 -3478 -3426 -3477 ]   @:  [ 1689380 324809 85470 1689380 15 477886 36473 171914 1870446 ] 
      max:        [ 3373 1861 3434 1866 3749 1871 3578 1856 3224 ]         @:  [ 500028 1401773 706989 1859999 98456 1236583 593975 600216 207539 ] 
  ...

Image support is limited to two-dimensional data

The support for images with more than two dimensions has been removed in this release.

CHANGES IN CIAO 3.0

Improved handling of spatial filters

Normally when a spatial filter is applied to an image using the Data Model (see "ahelp dmimfiltering"), pixels excluded by the filter are set to 0. Prior to CIAO 3.0, the only way to get dmstat to ignore such filtered pixels was to set them to the null/NaN value for the image (see "ahelp dmopt"). This is no longer needed since dmstat now uses only those pixels that lie within the data subspace (see "ahelp subspace") of the image.

In CIAO 3.0 one can therefore say:

  unix% dmstat "img.fits[sky=circle(100,100,5)]"

whereas you used to have to use something like:

  unix% dmstat "img.fits[sky=circle(100,100,5)][opt null=-999]"

This new behavior is particularly useful since it recognizes spatial filters that have previously been applied to the data (assuming the coordinate system matches). For instance, after the following set of commands:

  unix% dmcopy "evt.fits[sky=region(source.reg)]" evt.source
  unix% dmcopy "evt.source[bin sky=1]" img.source
  unix% dmstat img.source

the calculation will ignore those pixels in img.source that are not within the region defined by the file source.reg.

Output precision

The maximum number of significant digits used to display floating-point numbers has been increased from 9 to 11. For example, dmstat will now display 0.0057441547943 rather than 0.00574415479.

Minimum and Maximum row location for tables

The row number of the the minimum and maximum value of a column is now included in the output to match the behavior with images. For example:

  unix% dmstat "evt2.fits[cols x]"
  x[pixel]
      min:        2479.8139648          @:        102340
      max:        6018.9042969          @:        126933
  ...

USING VIRTUAL COLUMNS

If the table contains a column with a World Coordinate System attached to it - for instance, the SKY column of a Chandra event list - then dmstat can calculate statistics in both the original and transformed coordinate frames. All you have to do is include the names of the columns you require - e.g. "evt2.fits[cols ra]" or "evt2.fits[cols eqpos]" - as part of the infile parameter. For example:

  unix% dmstat "evt2.fits[cols eqpos]"
  EQPOS(RA, DEC)[deg]
      min:        ( 258.57348385 67.020579406 )         @:        ( 31812 29626 )
      max:        ( 259.82637267 67.421050984 )         @:        ( 206 7825 )
     mean:        ( 259.26622974 67.217694774 )
    sigma:        ( 0.30743608783 0.094642855434 )
      sum:        ( 48892684.871 12675980.098 )
     good:        ( 188581 188581 )
     null:        ( 0 0 )

HANDLING 'INVALID' DATA

Tables and images can include elements which do not contain valid data; for instance, the result of a division by 0 or an event in a grating observation which does not have a grating order. Such items are generally flagged by a "special" number (see the "null" option in "ahelp dmopt") and dmstat will ignore them during the calculation. As part of its output, dmstat reports the number of valid (good) and invalid (null) rows/pixels it processed.

Pixels in images can also be ignored if they lie outside the region of interest - i.e. a spatial filter which has been applied to the image. Prior to CIAO 3.0, you had to explicitly set these pixels to a "null" value using the "opt null" option of the Data Model; this is no longer necessary since dmstat uses the subspace information of the image (see "ahelp subspace") to find out which pixels to exclude from the calculation.

ACCESSING STATISTICS FROM THE PARAMETER FILE

New to CIAO 3.3 is the ability of dmstat to write the calculated values to the parameter file (using the "out_*" parameters). This means that you no longer need to parse the screen output to access these values. The following csh/tcsh expression will set the variable m to the mean value calculated by dmstat.

  unix% set m = `pget dmstat out_mean`

Not all the fields are written to the parameter file; the output depends on what options are selected and the type of the input file. For example, out_median is only filled in if the median option is set to true and either the input file is not an image or the centroid option is set to false. The contents of some of the fields also depends on the options chosen: e.g. when analyzing an image, the out_columns parameter is set to the block name when the centroid option is set to no, but it is set to a comma-separated list of the names of the components of the physical coordinate system when centroid is set to "yes".

Using multiple columns

When given a scalar column, such as "evt2.fits[cols x]", the output values will also be scalar values. For more complicated situations, such as "evt2.fits[cols sky]" and "evt2.fits[cols time,energy]", the output values will be written out as a comma-separated list. As these values can be treaded as stacks (see "ahelp stack"), the stack tools and module ("ahelp /stk_/") can be used to access individual elements of the list. The contents of the out_columns parameter can be used to map between position and column. So, after

  unix% dmstat "evt2.fits[cols time,sky]" sigma-
  time[s]
      min:        68010810.424          @:        1 
      max:        68010862.403          @:        1973 
     mean:        68010837.09 
      sum:        1.3602167418e+11 
     good:        2000 
     null:        0 

  sky(x, y)[pixel]
      min:        ( 2495.1342773 2870.4226074 )         @:        ( 206 940 )
      max:        ( 5968.3662109 5709.3413086 )         @:        ( 1610 1384 )
     mean:        ( 3969.3936443 4225.1935254 )
      sum:        ( 7938787.2886 8450387.0508 )
     good:        ( 2000 2000 )
     null:        ( 0 0 )

we can say

  unix% set cols = `pget dmstat out_columns`
  unix% echo $cols
  time,x,y
  unix% set means = `pget dmstat out_mean`
  unix% echo $means
  68010837.09,3969.3936443,4225.1935254
  unix% stk_where $cols x echo+
  2
  unix% stk_read_num $means 2 echo+
  3969.3936443

Note that although only two columns - namely "time" and "sky" - were given to dmstat, the output contains three values, since the "sky" column has been broken up into its constituent columns ("x" and "y"). Columns that are unsupported by dmstat - i.e. bit arrays and character strings - are not written out to the parameter file.

For images, the number of values written out to each parameter depends on the dimensionality of the image (although dmstat currently only accepts two-dimensional images).

Setting the infile parameter to a stack

When the infile parameter is a stack - such as "img1.fits,img2.fits" or "@file.lis" - only the statistics for the last item in the stack are written to the parameter file. The filename can be found from the parameter file using something like the following:

  unix% set files = `pget dmstat infile`
  unix% set n = `stk_count "$files" echo+`
  unix% set file = `stk_read_num "$files" $n echo+`