Last modified: 24 October 2023


HRC Subspace Caveat

Caveat: Merging some HRC observations

A change was made to HRC Level 0 data in standard data processing version DS8.5 (December 5, 2012) which corrected the allowed range values of some engineering columns in the Level 0 event file. These ranges are part of the file's data subspace and are propagated from Level 0 through Level 2.

This can cause problems when files created with different releases are merged. If any part of the subspace is different, the Good Time Intervals will not merge together which can result in the ONTIME (and related keywords such as LIVETIME and EXPOSURE) being incorrectly computed. This affects all CIAO tools and scripts when applied to merging datasets. The merge_obs and reproject_obs scripts will have a fix coming soon.

Users can work around the issue by removing the subspace associated with the columns as in the example below:

unix% dmcopy "hrc_evt2.fits[subspace -av1,-au1,-mjf,-mnf,-endmnf,-sub_mjf,-clkticks]" evt2.fits

The columns listed above are not typically used when analyzing Chandra data so removing the subspace associated with them is acceptable. Removing the subspace of other arbitrary columns should not be done unless the user fully understands the consequences of doing so.

Are my data affected?

The easiest way to illustrate the problem is with an example. Below we are going to attempt to merge two observation of M31.

unix% download_chandra_obsid 6202,13231 evt2
unix% mv {6202,13231}/primary/*evt2.fits.gz ./
unix% dmlist "*evt2.fits.gz" header,clean | egrep "ASCDSVER"
ASCDSVER             8.5.1                          Processing system revision
ASCDSVER             8.4.5                          Processing system revision

This shows that indeed the two observations have been processed with two different processing versions: one before the change and one after. We can check for other difference using the dmdiff tool.

unix% dmdiff *evt2.fits.gz data- key- 
Infile 1:  hrcf06202N004_evt2.fits.gz
Infile 2:  hrcf13231N003_evt2.fits.gz


Message:                         Column:       Row              Value(s):                    Diff:
--------                         -------       ---              ---------                    -----
Range maxs are not equal        CLKTICKS         1        262143 1091567608    +1.09131e+09 (+4.16e+05 %)
# dmdiff (CIAO 4.5): WARNING: subspace column "status" type unknown or not supported.
# dmdiff (CIAO 4.5): WARNING: subspace column "SUB_MJF" type unknown or not supported.

This shows us that the CLKTICKS (clock ticks) column has a different range. Note that SUB_MJF is not checked since dmdiff is currently unable to check BYTE subspace data types. Looking at the raw data (see dmlist subspace example below), we can see though that it too is different.

Despite these differences, we continue by reprojecting the observations to a common tangent point and using dmmerge to merge the files together.

unix% reproject_events hrcf06202N004_evt2.fits.gz 6202_reprj.fits match=hrcf13231N003_evt2.fits.gz clob+
unix% dmmerge 6202_reprj.fits,hrcf13231N003_evt2.fits.gz merge.fits clob+
BTIMDRFT values are different...FAIL...
BTIMNULL values are different...FAIL...
BTIMRATE values are different...FAIL...
warning: DS_IDENT has different value...Merged...
warning: OBJECT has different value...Merged...
warning: OBSERVER has different value...Merged...
warning: OBS_ID has different value...Merged...
omit - RA_PNT values different more than 0.000300
omit - ROLL_NOM values different more than 1.000000
omit - ROLL_PNT values different more than 1.000000
warning: SEQ_NUM has different value...Merged...
warning: TITLE has different value...Merged...

The various keyword warnings are normal and expected when merging different observations, see merging_rules. If we now look the structure of the merged file we will see an anomaly

unix% dmlist merge.fits blocks 
Dataset: merge.fits
     Block Name                          Type         Dimensions
Block    1: PRIMARY                        Null        
Block    2: EVENTS                         Table         9 cols x 2071847  rows
Block    3: GTI                            Table         2 cols x 1        rows
Block    4: GTI_CPT2                       Table         2 cols x 1        rows

For an HRC event file there is usually only 1 GTI block that describes the good time. If things merged cleanly this should still be the case but as we suspected, we are now seeing two. If we ask the datamodel to display the subspace information we see

% dmlist merge.fits subspace
Data subspace for block EVENTS: Components: 2 Descriptors: 26 
 --- Component 1 --- 
   1 time                 Real8               TABLE GTI
  25 SUB_MJF              Byte                0:7 
  26 CLKTICKS             Int4                0:262143 
 --- Component 2 --- 
   1 time                 Real8               TABLE GTI_CPT2
  25 SUB_MJF              Byte                0:64 
  26 CLKTICKS             Int4                0:1091567608 

Since SUB_MJF and CLKTICKS are different, the GTIs cannot be combined.

Now, what effect does this have on the data? Initially, not much. If we check the ONTIME keywords in the event files we see

unix% dmlist "*evt2.fits.gz,merge.fits" header,clean | grep ONTIME 
ONTIME                   18170.4320714770 [s]       
ONTIME                   19681.0260393620 [s]       
ONTIME                   37851.4581108390 [s]       

Where the last value listed above is associated with the merge.fits and looks to be as expected: the sum of the earlier two. The dmmerge keyword merging rule, calcGTI correctly unions the two GTIs to compute the ONTIME value.

However, this file is no longer self consistent. Any further filtering will trigger the ONTIME to be recomputed and following the CXC conventions will be computed from the first GTI.

unix% dmcopy "merge.fits[sky=circle(16135,16544,25)]" mysrc.fits
unix% dmlist mysrc.fits header,clean | grep ONTIME
ONTIME                   18170.4320714770 [s]       

The example above shows that applying a simple spatial filter has resulted in the output file's ONTIME value being incorrectly recomputed. If one were to use this ONTIME value to compute fluxes it would be incorrect.