Filtering Data

Time Filtering in General

Applying time filters to an event file involves four primary steps:

1. Define the filter

   Determine a filtering criterion that can be expressed as a valid dmgti userlimit parameter.

2. Create a Good Time Interval

   A good time interval (GTI) indicates which data are "good", based on the chosen filter definition. The tool dmgti is used to create a GTI table:

   unix% punlearn dmgti
   unix% pset dmgti infile=filename_mtl2.fits
   unix% pset dmgti outfile=myfilter_gti.fits
   unix% pset dmgti userlimit="<filtering criterion>"
   unix% dmgti
   

3. Apply the GTI table

   Apply the GTI table to the event fits file with dmcopy:

   unix% punlearn dmcopy
   unix% dmcopy "filename_evt2.fits[@myfilter_gti.fits]" filename_evt2_flt.fits
   

4. Verify filtering

   Looking at the EXPO keyword is one way to see the results of filtering; this will show the amount of time that was eliminated in creating the filtered event file:

   unix% dmlist event_filename.fits header | grep EXPO
   unix% dmlist event_filename_flt.fits header | grep EXPO
   

   The tool dmkeypar may also be used to examine specific header keywords:

   unix% dmkeypar event_filename.fits EXPOSUR7 echo+
   

Eliminate High Background Times

The Chandra Calibration team notes the following in its study of the ACIS background available from the Calibration page: "A phenomenon not anticipated prior to launch that can seriously affect the scientific value of an observation is background flares, when the count rate can increase by a factor of up to 100. Such flares have been observed anywhere in the orbit, including near the apogee." Therefore, eliminating high-background periods is another type of filtering that might be required for your data:

Here is one example of how to eliminate periods of high background. The ACIS background and Filtering lightcurves threads also discuss ways of filtering out flares.

1. View the image

   An event file may be viewed directly by ds9:

   unix% ds9 acis_1843_evt2_0.3-10.fits &
   

   A log scale (Scale -> log) and some adjustment to the color table show the object and the background. It may also be necessary to adjust the blocking factor to view the whole image at once; try "Bin -> block 8" for a reasonable view.

2. Select the background area

   From this image, regions may be created to define the background area. Select "Region -> Ellipse" from the ds9 toolbar, then hold down the left mouse button at the center of the field, and drag until the ellipse covers the desired area. Clicking on the resulting shape will select the region, allowing you to reposition or resize it; this FAQ shows how to rotate regions in ds9. This region  is used for the example.

   The defined region is saved using the following steps:

   • Region -> File Format-> CIAO
   • Region -> File Coordinate System -> Physical
   • Region -> Save Regions... -> Save As "obj.reg"

   The imaging (ds9) window may be closed by choosing "File -> Exit".

   The resulting obj.reg file will look something like this:

   unix% more obj.reg 
   # Region file format: CIAO version 1.0
   ellipse(1628,4116,100,140,0) # blue
   

3. Extract a background-only lightcurve

   Now it is possible to create a background lightcurve by using the "virtual file" syntax. We do this by specifying the whole field, minus the source object(s). In this case, the ellipse that is defined in obj.reg is omitted; note that we could have also used the exclude syntax to ignore the source (i.e. "exclude sky=region(obj.reg)" in the infile parameter:

   unix% punlearn dmextract
   unix% pset dmextract \
         infile="acis_1843_evt2_0.3-10.fits[sky=field()-ellipse(1628,4116,100,140,0)][bin time=::3.24104]"
   unix% pset dmextract outfile=background_lc.fits
   unix% pset dmextract opt=ltc1
   unix% dmextract
   Input event file  (acis_1843_evt2_0.3-10.fits[sky=field()-ellipse(1628,4116,100,140,0)][bin time=::3.24104]): 
   Enter output file name (background_lc.fits): 
   

   You can check the resulting parameter file with plist dmextract. Here the shorthand "sky" was used to specify x,y coordinates. Since "sky" is an alias of "(x,y)", both work in this context.

4. Examine the lightcurve and define a filter

   The lightcurve that was just created can be plotted and examined in ChIPS:

   unix% chips
   
   Welcome to ChIPS, version CIAO 3.4
   Copyright (C) 1999-2003, Smithsonian Astrophysical Observatory
   
   chips> curve "background_lc.fits[cols time,count_rate]"
   chips> xlabel Time
   chips> ylabel "Background Count Rate"
   

   Which produces Figure 2 . We can see that there is a spike in the lightcurve. The coordinates can be found with the ChIPS pickpoints command:

   chips> pickpoints
   
   Click LMB or tap spacebar to pick point.
   Click RMB or type 'q' to quit picking points.
   
   Point picked: (84274280.000000, 84.512329)
   Point picked: (84274280.000000, 126.068245)
   
   chips> quit
   

   Here we have marked the "top" and "bottom" of the spike.

   To remove this feature from the dataset, we need to define a filter that excludes rates greater than ~85.0 counts/sec.

5. Create a Good Time Interval table

   As mentioned before, the Good Time Interval (GTI) table contains the times during which the data are good, based on the filter definition. For this example, it will be the times when the count rate is below 85.0 counts/sec, as determined in the previous step:

   unix% punlearn dmgti
   unix% pset dmgti infile=background_lc.fits
   unix% pset dmgti outfile=bkg_gti.fits
   unix% pset dmgti userlimit="count_rate <=85.0"
   unix% dmgti
   Input MTL file (background_lc.fits): 
   Output GTI file (bkg_gti.fits): 
   User defined limit string (count_rate <=85.0):
   

   A more complex filter could be applied instead, such as "count_rate >= 5.5 && count_rate <=85.0"; see ahelp dmfiltering for other examples of filtering syntax.

   You can check the resulting parameter file with plist dmgti.

6. Apply the GTI table

   To apply the GTI table to the dataset, dmcopy is run:

   unix% dmcopy "acis_1843_evt2_0.3-10.fits[@bkg_gti.fits]" acis_1843_evt2_0.3-10_bkgflt.fits
   

7. Verify filtering

   Examination of the EXPO keywords shows the amount of filtering that was done in the creation of the new event file:

   unix% dmlist acis_1843_evt2_0.3-10.fits header | grep EXPO
   0082 EXPOSUR0                  7857.6259545448 [s]       Real8        [s] Total exposure time, with all
   0085 EXPOSUR1                  7857.6259545448 [s]       Real8        [s] Total exposure time, with all
   0088 EXPOSUR2                  7857.6259545448 [s]       Real8        [s] Total exposure time, with all
   0091 EXPOSUR3                  7854.4664748687 [s]       Real8        [s] Total exposure time, with all
   0094 EXPOSUR6                  7857.6259545448 [s]       Real8        [s] Total exposure time, with all
   0097 EXPOSUR7                  7857.6259545448 [s]       Real8        [s] Total exposure time, with all
   0100 EXPOSURE                  7857.6259545448 [s]       Real8        [s] Total exposure time, with all
   
   unix% dmlist acis_1843_evt2_0.3-10_bkgflt.fits header | grep EXPO
   0081 EXPOSUR0                  7825.6259545417 [s]       Real8        [s] Total exposure time, with all 
   0084 EXPOSUR1                  7825.6259545417 [s]       Real8        [s] Total exposure time, with all 
   0087 EXPOSUR2                  7825.6259545417 [s]       Real8        [s] Total exposure time, with all 
   0090 EXPOSUR3                  7822.4664748655 [s]       Real8        [s] Total exposure time, with all 
   0093 EXPOSUR6                  7825.6259545417 [s]       Real8        [s] Total exposure time, with all 
   0096 EXPOSUR7                  7825.6259545417 [s]       Real8        [s] Total exposure time, with all 
   0099 EXPOSURE                  7825.6259545417 [s]       Real8        [s] Total exposure time, with all