Last modified: 19 Jan 2022

URL: https://cxc.cfa.harvard.edu/ciao/threads/filter/

Filtering Data

CIAO 4.16 Science Threads


Overview

Synopsis:

The CIAO Data Model allows powerful filtering of datafiles beyond the standard onboard event filtering. A file may be filtered on any of its columns, e.g. energy, time, position.

Related Links:

Last Update: 19 Jan 2022 - Reviewed for CIAO 4.14. No updates needed.


Contents


Get Started

Download the sample data: 1843 (ACIS-I, G21.5-0.9)

unix% download_chandra_obsid 1843 evt2

Filtering is not restricted to the level=2 event file; in fact, filters are usually applied to the evt1.fits file when working with grating data.


Restrict The Energy Range

For most ACIS analyses, you will want to include at most the 0.3 kev to 10.0 kev energy range, as explained in the Choosing an Energy Filter why topic.

unix% punlearn dmcopy
unix% dmcopy "acisf01843N003_evt2.fits[energy=300:10000]" acis_1843_evt2_0.3-10.fits 

This command creates a new event file that only includes the data within the specified energy range.


Using Exclude Filters

It is also possible to create a filter based on what you do not want in the final file by using the exclude syntax:

unix% dmcopy "acisf01843N003_evt2.fits[exclude ccd_id=0:4,6,8:9]" acis_1843_evt2_bi.fits 

This filter only keeps the back-illuminated chips (ccd_id=5,7).

Care must be taken when combining filters; this syntax WILL NOT WORK:

unix% dmcopy "acisf01843N003_evt2.fits[exclude sky=region(reg.fits)][energy=300:10000]" \
      acis_1843_evt2_out.fits
Failed to open virtual file acisf01843N003_evt2.fits[exclude sky=region(reg.fits)][energy=300:10000]
DM Parse error: cannot mix EXCLUDE and FILTER

The second filter (on energy) must be rewritten in terms of an exclude for dmcopy to produce the desired output:

unix% dmcopy "acisf01843N003_evt2.fits[exclude sky=region(reg.fits),energy=:299,10000:]" \
      acis_1843_evt2_out.fits 

Applying Time Filters

For simple time filters, e.g. selecting a particular interval, the basic DM syntax may be used:

unix% dmcopy "acisf01843N003_evt2.fits[time=60413209:60414209]" acis_1843_evt2_1000s.fits 

For more involved filters, use the CIAO tool dmgti. This tool allows the user to set constraints on the other (i.e. non-TIME) columns in the file and extract the set of times for which those constraints are true. A general outline of this process is given first, followed by a common filtering scenario in detail (Eliminate High Background Times).

Time Filtering in General

Applying time filters to an event file involves four primary steps:

  1. Define the filter

    Determine a filtering criterion that can be expressed as a valid dmgti userlimit parameter.

  2. Create a Good Time Interval

    A good time interval (GTI) indicates which data are "good", based on the chosen filter definition. The tool dmgti is used to create a GTI table:

    unix% punlearn dmgti
    unix% pset dmgti infile=filename_mtl2.fits
    unix% pset dmgti outfile=myfilter_gti.fits
    unix% pset dmgti userlimit="<filtering criterion>"
    unix% dmgti
    
  3. Apply the GTI table

    Apply the GTI table to the event fits file with dmcopy:

    unix% punlearn dmcopy
    unix% dmcopy "filename_evt2.fits[@myfilter_gti.fits]" filename_evt2_flt.fits 
    
  4. Verify filtering

    Looking at the EXPOSURE keyword(s) is one way to see the results of filtering; this will show the amount of time that was eliminated in creating the filtered event file:

    unix% dmlist event_filename.fits header | grep EXPO
    unix% dmlist event_filename_flt.fits header | grep EXPO
    

    The tool dmkeypar may also be used to examine specific header keywords:

    unix% dmkeypar event_filename.fits EXPOSUR7 echo+
    

Eliminate High Background Times

The Chandra Calibration team notes the following in its study of the ACIS background available from the Calibration page: "A phenomenon not anticipated prior to launch that can seriously affect the scientific value of an observation is background flares, when the count rate can increase by a factor of up to 100. Such flares have been observed anywhere in the orbit, including near the apogee." Therefore, eliminating high-background periods is another type of filtering that might be required for your data:

Here is one example of how to eliminate periods of high background. The ACIS background and Filtering lightcurves threads also discuss ways of filtering out flares.

  1. View the image

    An event file may be viewed directly by ds9:

    unix% ds9 acis_1843_evt2_0.3-10.fits &
    

    A log scale (Scale → log) and some adjustment to the color table show the object and the background. It may also be necessary to adjust the blocking factor to view the whole image at once; try "Bin → block 8" for a reasonable view.

  2. Select the background area

    From this image, regions may be created to define the background area. In this example, the background will be defined as the entired dataset minus the selected source region.

    Select "Region → Shape → Ellipse" from the ds9 toolbar, then hold down the left mouse button at the center of the field, and drag until the ellipse covers the desired area. Clicking on the resulting shape will select the region, allowing you to reposition or resize it. This region (Figure 1) is used for the example.

    Figure 1: Selecting a region of interest

    [Thumbnail image: The source region on chip ACIS-S3 is displayed in red on the event data in ds9.]

    [Version: full-size]

    [Print media version: The source region on chip ACIS-S3 is displayed in red on the event data in ds9.]

    Figure 1: Selecting a region of interest

    This source region will be excluded from the data in order to create a background lightcurve.

    The defined region is saved using the following steps:

    1. Region → Save Regions... → Save As "obj.reg".
    2. After choosing "OK" in the region filename dialog, a format dialog is opened. Set the format to "CIAO" and the coordinate system to "Physical".

    The resulting file is:

    unix% cat obj.reg 
    # Region file format: CIAO version 1.0
    ellipse(1628,4116,100,140,0)
    
  3. Extract a background-only lightcurve

    Now it is possible to create a background lightcurve by using the "virtual file" syntax. We do this by specifying the whole field, minus the source object(s). In this case, the ellipse that is defined in obj.reg is omitted; note that we could have also used the exclude syntax to ignore the source (i.e. "exclude sky=region(obj.reg)" in the infile parameter:

    unix% punlearn dmextract
    unix% pset dmextract \
          infile="acis_1843_evt2_0.3-10.fits[sky=field()-ellipse(1628,4116,100,140,0)][bin time=::3.24104]"
    unix% pset dmextract outfile=background_lc.fits
    unix% pset dmextract opt=ltc1
    unix% dmextract
    Input event file  (acis_1843_evt2_0.3-10.fits[sky=field()-ellipse(1628,4116,100,140,0)][bin time=::3.24104]): 
    Enter output file name (background_lc.fits): 
    

    You can check the resulting parameter file with plist dmextract. Here the shorthand "sky" was used to specify x,y coordinates. Since "sky" is an alias of "(x,y)", both work in this context.

  4. Examine the lightcurve and define a filter

    The lightcurve that was just created can be plotted in python

    unix% python
    
    >>> from pycrates import read_file
    >>> import matplotlib.pylab as plt
    >>> 
    >>> tab = read_file("background_lc.fits")
    >>> xx = tab.get_column("time").values
    >>> yy = tab.get_column("count_rate").values
    >>> 
    >>> plt.plot(xx,yy,marker="o",linestyle="")
    >>> plt.xlabel("Time (s)")
    >>> plt.ylabel("Count Rate (c/s)")
    >>> plt.title("G21.5-0.9 [Chip S3, T=120, Offsets=-1,20,-4]")
    >>> plt.show()
    

    The plot is shown in Figure 2.

    Figure 2: Lightcurve of the background

    [A lightcurve of time vs count rate]
    [Print media version: A lightcurve of time vs count rate]

    Figure 2: Lightcurve of the background

    The lightcurve may also be examined with dmstat:

    unix% dmstat "background_lc.fits[cols count_rate]" 
    COUNT_RATE[count/s]
        min:        0             @:        1
        max:        120.31249977          @:        1037
       mean:        28.919604201
      sigma:        19.953609176
        sum:        86961.249832
       good:        3007
       null:        0
    

    Use dmlist to locate the region of maximum count rate:

    unix% dmlist "background_lc.fits[cols time,count_rate]" data
    ...
      1028  84274232.3370999992        78.7499998474
      1029  84274235.5781400204        97.4999998111
      1030  84274238.8191800117        91.8749998220
      1031  84274242.0602200031        91.8749998220
      1032  84274245.3012599945       109.6874997875
      1033  84274248.5423000157       104.6874997972
      1034  84274251.7833400071        91.8749998220
      1035  84274255.0243799984        95.3124998153
      1036  84274258.2654200196       100.3124998056
      1037  84274261.5064600110       120.3124997669
      1038  84274264.7475000024       107.4999997917
      1039  84274267.9885400236        66.5624998710
      1040  84274271.2295800149        69.3749998656
      1041  84274274.4706200063        67.4999998692
    ...
    

    The filter could also include an estimate for the lower limit of the count rate column:

    unix% dmlist "background_lc.fits[cols time,count_rate][count_rate>65]" data
    
    --------------------------------------------------------------------------------
    Data for Table Block LIGHTCURVE
    --------------------------------------------------------------------------------
    
    ROW    TIME                 COUNT_RATE
    
         1  84273308.6407000124        77.1874998504
         2  84273742.9400600195        70.3124998638
         3  84273746.1811000109        68.7499998668
         4  84274229.0960600078        70.6249998632
         5  84274232.3370999992        78.7499998474
         6  84274235.5781400204        97.4999998111
         7  84274238.8191800117        91.8749998220
         8  84274242.0602200031        91.8749998220
         9  84274245.3012599945       109.6874997875
        10  84274248.5423000157       104.6874997972
        11  84274251.7833400071        91.8749998220
        12  84274255.0243799984        95.3124998153
        13  84274258.2654200196       100.3124998056
        14  84274261.5064600110       120.3124997669
        15  84274264.7475000024       107.4999997917
        16  84274267.9885400236        66.5624998710
        17  84274271.2295800149        69.3749998656
        18  84274274.4706200063        67.4999998692
        19  84274277.7116599977        76.8749998510
    ...
    

    The spike occurs in the lightcurve between 84274232 and 84274267. To remove this feature from the dataset, we need to define a filter that excludes rates greater than ~80.0 counts/sec.

  5. Create a Good Time Interval table

    As mentioned before, the Good Time Interval (GTI) table contains the times during which the data are good, based on the filter definition. For this example, it will be the times when the count rate is below 80.0 counts/sec, as determined in the previous step:

    unix% punlearn dmgti
    unix% pset dmgti infile=background_lc.fits
    unix% pset dmgti outfile=bkg_gti.fits
    unix% pset dmgti userlimit="count_rate<=80.0"
    unix% dmgti
    Input MTL file (background_lc.fits): 
    Output GTI file (bkg_gti.fits): 
    User defined limit string (count_rate<=80.0):
    

    A more complex filter could be applied instead, such as "count_rate>= 5.5&&count_rate<=80.0"; see ahelp dmfiltering for other examples of filtering syntax.

    You can check the resulting parameter file with plist dmgti.

  6. Apply the GTI table

    To apply the GTI table to the dataset, dmcopy is run:

    unix% dmcopy "acis_1843_evt2_0.3-10.fits[@bkg_gti.fits]" acis_1843_evt2_0.3-10_bkgflt.fits 
    
  7. Verify filtering

    Examination of the EXPO keywords shows the amount of filtering that was done in the creation of the new event file:

    unix% dmlist acis_1843_evt2_0.3-10.fits header | grep EXPO
    0117 EXPOSURE                  7854.4664748687 [s]       Real8        Exposure time
    0118 EXPOSUR7                  7854.4664748687 [s]       Real8        Exposure time
    0119 EXPOSUR6                  7854.4664748687 [s]       Real8        Exposure time
    0120 EXPOSUR3                  7851.2665536184 [s]       Real8        Exposure time
    0121 EXPOSUR2                  7854.4664748687 [s]       Real8        Exposure time
    0122 EXPOSUR1                  7854.4664748687 [s]       Real8        Exposure time
    0123 EXPOSUR0                  7854.4664748687 [s]       Real8        Exposure time
    
    unix% dmlist acis_1843_evt2_0.3-10_bkgflt.fits header | grep EXPO
    0117 EXPOSURE                  7812.8664748911 [s]       Real8        Exposure time
    0118 EXPOSUR7                  7812.8664748911 [s]       Real8        Exposure time
    0119 EXPOSUR6                  7812.8664748911 [s]       Real8        Exposure time
    0120 EXPOSUR3                  7809.6665536407 [s]       Real8        Exposure time
    0121 EXPOSUR2                  7812.8664748911 [s]       Real8        Exposure time
    0122 EXPOSUR1                  7812.8664748911 [s]       Real8        Exposure time
    0123 EXPOSUR0                  7812.8664748911 [s]       Real8        Exposure time
    

Summary

The result of this thread is the level=2 event file acis_1843_evt2_0.3-10_bkgflt.fits, which is filtered for high background and restricted to a specific energy range.



Parameters for /home/username/cxcds_param/dmextract.par


#--------------------------------------------------------------------
#
# DMEXTRACT -- extract columns or counts from an event list
#
#--------------------------------------------------------------------
        infile = acis_1843_evt2_0.3-10.fits[sky=field()-ellipse(1628,4116,100,140,0)][bin time=::3.24104] Input event file 
       outfile = background_lc.fits Enter output file name
          (bkg = )                Background region file or fixed background (counts/pixel/s) subtraction
        (error = gaussian)        Method for error determination(poisson|gaussian|<variance file>)
     (bkgerror = gaussian)        Method for background error determination(poisson|gaussian|<variance file>)
      (bkgnorm = 1.0)             Background normalization
          (exp = )                Exposure map image file
       (bkgexp = )                Background exposure map image file
      (sys_err = 0)               Fixed systematic error value for SYS_ERR keyword
          (opt = ltc1)            Output file type: pha1 
     (defaults = ${ASCDS_CALIB}/cxo.mdb -> /soft/ciao/data/cxo.mdb) Instrument defaults file
         (wmap = )                WMAP filter/binning (e.g. det=8 or default)
      (clobber = no)              OK to overwrite existing output file(s)?
      (verbose = 0)               Verbosity level
         (mode = ql)              
    


Parameters for /home/username/cxcds_param/dmgti.par


        infile = background_lc.fits Input MTL file
       outfile = bkg_gti.fits     Output GTI file
     userlimit = count_rate<=80.0 User defined limit string
      (mtlfile = none)            Optional output smoothed/filtered MTL file
     (lkupfile = none)            Lookup table defining which MTL columns to check against (NONE|none|<filename>)
       (smooth = yes)             Smooth the input MTL data?
      (clobber = no)              Clobber output file if it exists?
      (verbose = 0)               Debug level
         (mode = ql)              
    

History

14 Dec 2004 reviewed for CIAO 3.2: no changes
01 Dec 2005 updated for CIAO 3.3: default value of dmextract error and bkgerror parameters is "gaussian", update to screen output in "Verify filtering" step of the Eliminate High Background Times section
01 Dec 2006 updated for CIAO 3.4: kernel parameter removed from dmgti
24 Jan 2008 updated for CIAO 4.0: updated ChIPS plotting syntax; filename and screen output updated for reprocessed data (version N003 event file)
06 Jun 2008 added "opt=all" to dmcopy commands so all blocks are kept, e.g. for grating files
23 Jun 2008 updated image display to place figures inline with text
06 Feb 2009 updated for CIAO 4.1: Python and S-Lang syntax included for ChIPS plotting
05 Feb 2010 updated for CIAO 4.2: changes to the ds9 region file format menu
13 Jan 2011 reviewed for CIAO 4.3: no changes
03 Jan 2012 reviewed for CIAO 4.4: no changes
03 Dec 2012 Review for CIAO 4.5; removed white space in dmgti expressions which might get interpreted as stack seperators.
02 Dec 2013 Review for CIAO 4.6; no changes.
17 Dec 2014 Reviewed for CIAO 4.7; minor edits only.
04 Jan 2017 Review for CIAO 4.9. Removed all opt=all from dmcopy commands due to the dmcopy bug.
16 Apr 2019 Updated to use matplotlib for plotting.
19 Jan 2022 Reviewed for CIAO 4.14. No updates needed.