This help file is split up into the following sections:
- 1. Table filtering on columns with real data types
- 2. Table filtering on columns with integer data types
- 3. Table filtering on columns with character string data types
- 4. Table filtering on columns with bit data type
- 5. Table filtering on columns with logical data type
- 6. Table filtering on vector columns using region filters
- 7. Compound filters
- 8. Exclude filters
- 9. Filters defined in files: using the @ syntax
There are a number of ways to filter a DM block (table or image in a file).
In this section we describe table filtering; see also `ahelp dmimfiltering'
for filtering files which are in image format.
To see which column names you can filter on, use
to display the columns in the main block of file a.fits.
The simplest kind of filter is a range filter
[filter energy=1000:2000]
which specifies that the DM should only see rows in the input file
which satisfy 1000.0 <= energy < 2000.0. You can use this filter
by appending it to a filename or block name in any of the CIAO tools:
unix% dmcopy "a.fits[filter energy=1000:2000]" b.fits
unix% dmcopy "a.fits[events][filter energy=1000:2000]" b.fits
Note that the prefix "filter" is optional:
unix% dmcopy "a.fits[energy=1000:2000]" b.fits
You can filter on multiple quantities:
unix% dmcopy "a.fits[energy=1000:2000,time=5410300:5410320]" b.fits
You can also filter on multiple ranges for each quantity:
unix% dmcopy "a.fits[energy=1000:2000,4000:8000,grade=0,2:4,6]" b.fits
This filter accepts rows which have energies between either 1000 and 2000
or 4000 and 8000, and grades equal to 0, 2 to 4, or 6. Note that
you can leave out the colon if the min and max of a range are the same,
so 0:0 becomes just 0. If you want to express "less than" and "greater than",
you can just retain the colon but omit the min or max, so
means accept energies up to 4000; whilst
means accept all values of detx except 511.0 <= detx < 513.0.
The interpretation is a little different depending on whether the
table column has integer or real data type. For an integer data type
column,
means (4000 <= energy <= 5000), in other words both ends of
the range are included.
The syntax also supports the special pseudo-column #row, the row number
of the unfiltered file:
Filtering is a little more restricted with character string columns.
You can only use the colon syntax, for example:
[filter shape=m:n,point,rectangle,s:z]
Be careful: the range m:n includes everything beginning with m, and
it includes the letter n, but not other strings beginning with n: for
example, "ngc" is not within m:n since in an ASCII ordering ngc > n.
The comparison is case-sensitive.
Columns with bit data type are a special case. The most common example
is the STATUS column in the event files, which is 32 bits wide for
Chandra ACIS event files generated between launch and at least 2003
(it may be increased at a later date).
Suppose for simplicity you instead have a status column with only 6 bits,
and wish to accept rows with the 3rd bit from the end set and the end bit
equal to zero. A simple
numeric filter of the type described above would have to be very complicated
to describe all the numeric values for which these bits have the desired
values; instead, the user supplies a `bitmask string':
The string may only contain the characters 1 (corresponding bit must be
set), 0 (bit must not be set) and x (wild card: bit may have any value).
The bit pattern display convention here is that the rightmost bit
displayed (with the value 0 in the example above) is the least significant
bit, which is bit 32 in the usual FITS convention and bit 0 in the usual
C programmers convention.
Logical data columns contain a value of either true (1) or
false (0). When filtering on a "true" value, any of the
following will work:
unix% dmlist "srclist.fits[double=1]" data
unix% dmlist "srclist.fits[double=T]" data
unix% dmlist "srclist.fits[double=TRUE]" data
unix% dmlist "srclist.fits[double=true]" data
Likewise, for "false" values:
unix% dmlist "srclist.fits[double=0]" data
unix% dmlist "srclist.fits[double=F]" data
unix% dmlist "srclist.fits[double=FALSE]" data
unix% dmlist "srclist.fits[double=false]" data
In CIAO, a `vector column' is a paired column consisting of two components.
For instance, the DET vector column has components DETX and DETY.
You can see which of the columns in the file are vector columns by using
`dmlist filename cols'. There are two ways to filter on a vector column:
define a rectangular region by filtering on each of the components as usual,
unix% dmcopy "evt.fits[detx=4000:5000,dety=3000:4000]" rectangle.fits
or, use a region filter (see `ahelp dmregions' for the full region syntax)
on the vector column, either using its name - in this case DET - or the
two components in parentheses - in this case (DETX,DETY).
unix% dmcopy "evt.fits[det=circle(4500,3500,120)]" circle.fits
unix% dmcopy "evt.fits[(detx,dety)=circle(4500,3500,120)]" circle.fits
The syntax also supports compound (logical OR) filters:
unix% dmcopy \
"evt.fits[(ccd_id=2,chipx=512:513)||(ccd_id=7,chipx=500:520)]" \
two_bits.fits
Note that we do not support arbitrarily complex C-style logical expressions, just
lists of filters separated by ||.
A very useful feature is the ability to invert a filter,
so excluding rows or pixels instead of including them:
unix% dmcopy "evt.fits[exclude sky=region(reg.ds9)]" holes.fits
unix% dmcopy "evt.fits[exclude pha=2:100,grade=7]" clean.fits
This is particularly useful in conjunction with compound
filters (see the previous section):
unix% dmcopy \
"evt.fits[exclude (ccd_id=0:6,8:9,chipx=513)||(ccd_id=7,chipx=512:513)]" \
better.fits
Filter specifications may be applied from a file rather than including
them on the command line. This is especially useful when applying the
same filters to multiple input files. The filter filename is preceded by
an "at" symbol (@).
unix% dmlist "acis_evt2.fits[@filters.lis]" counts
where filters.lis contains:
sky=rotbox(4148.125,4043.625,7.58978,22.338761,44.516094),
pi > 100
Note that each complete filter must be separated with a comma, just as
they would be on the command line. This syntax can also be used to
apply the GTIs (Good Time Intervals) from one file to another, so:
unix% dmcopy "evt2.fits[@gti.fits]" evt2_filtered.fits
filters evt2.fits using the GTIs stored in gti.fits.