Filtering within Sherpa may be performed using one or more of the following means:
The IGNORE and NOTICE commands specify data-portions to be included or excluded, for 1- and 2-D datasets. The syntax for the commands is:
sherpa> {IGNORE | NOTICE} # {arg | FILTER filterExpr}
where # specifies the number of the dataset on which the command will act (default dataset number is 1). To specify an entire dataset or image, arg is ALL. To specify pha channels with quality flags different than 0, arg is BAD (this argument is valid only with the IGNORE command).
See Sections 1.44 and 1.55 for examples.
To interactively specify a filter region from a plot data display, the IGNORE or NOTICE command argument arg is PLOT:
Note: If you are interactively filtering on any dataset other than dataset number one, you must specify the dataset number when issuing the command IGNORE # PLOT or NOTICE # PLOT. Otherwise, the filter you specify from the data display will be applied to dataset number 1.
Please see the LPLOT command and Chapter 2 for further information regarding plotting capabilities within Sherpa .
See Sections 1.44 and 1.55 for additional examples.
Examples:
sherpa> READ DATA example.dat sherpa> LP DATA sherpa> NOTICE ALL sherpa> IGNORE PLOT <left-mouse-click once at the desired minimum x-axis value> <and then left-mouse-click again at the desired maximum x-axis value> sherpa> LP FILTER sherpa> LPLOT DATAIn this example, 1-D data is first displayed. Then the data filter is set to include the entire dataset, with the command NOTICE ALL. The command IGNORE PLOT readies the cursor for selecting the desired filter region, and the user should then left-mouse-click first at the desired minimum x-axis location and then again at the maximum x-axis location. After the second left-mouse-click, the Sherpa command prompt is returned. The command LPLOT FILTER will then plot the data region that you have marked for exclusion. Finally, the dataset is plotted again, with the command LPLOT DATA, showing that the selected regions have been properly excluded.
To interactively specify a filter region from an image data display, the IGNORE or NOTICE command argument arg is IMAGE:
Note: If you are interactively filtering on any dataset other than dataset number one, you must specify the dataset number when issuing the command IGNORE # IMAGE or NOTICE # IMAGE. Otherwise, the filter you specify from the data display will be applied to dataset number 1.
Also note: If you re-display an image, following filtering, the entire image will be displayed, even though the filter has been applied to the data.
Please see the IMAGE command and Chapter 2 for further information regarding imaging capabilities within Sherpa .
See Sections 1.44 and 1.55 for additional examples.
Examples:
sherpa> READ DATA example_img.fits sherpa> IMAGE DATA sherpa> NOTICE ALL <mark regions for exclusion> sherpa> IGNORE IMAGE sherpa> IMAGE FILTERIn this example, 2-D image data is first displayed. Then the data filter is set to include the entire image, with the command NOTICE ALL. Next, the user chooses regions, by placing markers on the displayed image. The command IGNORE IMAGE sets the data filter to exclude the marked regions. The command IMAGE FILTER will display the image data regions that have been marked for exclusion.
To specify a data filter using a filter expression, there are two different syntax options available for the IGNORE or NOTICE command argument filterExpr:
For Syntax Option A: This syntax is a list of data value ranges, and/or specific data values, combined using logical operators. Data value ranges are formed using either a data column name and comparison symbols, or the colon range syntax:
filterExpr colname{symbol# | range}
operator colname{symbol# | range} ...
where colname is the name of an appropriate data column. Note that columns must have assigned names when the data are read into Sherpa , in order to use this syntax option. These column names for a dataset may be viewed using the SHOW DATA command.
The following tables show the available comparison symbols, symbol, and logical operators, operator:
symbol | Description |
---|---|
greater-than | |
greater-than or equal-to | |
less-than | |
less-than or equal-to | |
! | not equal-to |
equal-to | |
operator | Description |
---|---|
& | and |
exclusive or | |
| | inclusive or |
With this syntax, # is a data value being specified. Or, a range may
be specified with one of the following colon range
syntax options:
range | Description |
---|---|
value1:value2 | from numerical data value1 to value2, inclusive |
#: | all data values greater-than or equal-to # |
:# | all data values less-than or equal-to # |
# | all data values equal-to # |
Examples:
sherpa> READ DATA "example_bin.fits[columns x1, x2]" sherpa> NOTICE ALL sherpa> IGNORE FILTER x1=4 | x1=8: | x1=1:3The first command inputs a data file containing named columns, and then all data points are included with NOTICE ALL. The filter expression then specifies column x1 data values equal-to 4, greater-than or equal-to 8, or 1 thru 3, inclusive. This filter expression is equivalent to Example 1 of Subsection 3.2.4, when Axis 0 is named x1. The following commands are also equivalent:
sherpa> IGNORE FILTER x1=4 | x1>=8 | x1=1:3 sherpa> IGNORE FILTER x1=4 | x1>=8 | (x1>=1 & x1<=3)Note that expressions with logical operators can be grouped using parentheses, for control over logical order of operations.
sherpa> IGNORE ALL sherpa> NOTICE FILTER x1=:2 | x1=8This filter expression specifies column x1 data values less-than or equal-to 2, or equal-to 8. This filter expression is equivalent to Example 2 of Subsection 3.2.4, when Axis 0 is x1. The command
sherpa> NOTICE FILTER x1<=2 | x1=8is also equivalent.
sherpa> NOTICE ALL sherpa> IGNORE FILTER x1=:2 | x1=7:This filter expression specifies column x1 data values less-than or equal-to 2, or greater-than or equal-to 7. This filter expression is equivalent to Example 3 of Subsection 3.2.4, when Axis 0 is x1. The command
sherpa> IGNORE FILTER x1<=2 | x1>=7is also equivalent.
sherpa> IGNORE ALL sherpa> NOTICE FILTER x1>5 & x1<8This filter expression specifies column x1 data values greater-than 5 and less-than 8 (i.e. 6 thru 7 inclusive). This filter expression is equivalent to Example 4 of Subsection 3.2.4, when Axis 0 is x1. The following commands are also equivalent:
sherpa> NOTICE FILTER x1=6:7
To specify a data filter using a filter expression, there are two different syntax options available for the IGNORE or NOTICE command argument filterExpr:
For Syntax Option B: This syntax is a comma-delimited list of data value ranges, and/or specific data values, for Axis 0. Data value ranges are formed using a colon, :.
filterExpr range , range, ...
where range is one of the following:
range | Description |
---|---|
value1:value2 | from numerical data value1 to value2, inclusive |
#: | all data values greater-than or equal-to # |
:# | all data values less-than or equal-to # |
# | all data values equal-to # |
Examples:
sherpa> READ DATA "example_bin.fits[columns x1, x2]" sherpa> NOTICE ALL sherpa> IGNORE FILTER 4, 8:, 1:3The first command inputs a data file containing two columns, and then all data points are included with NOTICE ALL. This filter expression specifies Axis 0 data values equal-to 4, greater-than or equal-to 8, or 1 thru 3, inclusive. This filter expression is equivalent to Example 1 of Subsection 3.2.3, when Axis 0 is x1.
sherpa> IGNORE ALL sherpa> NOTICE FILTER :2, 8This filter expression specifies Axis 0 data values less-than or equal-to 2, or equal-to 8. This filter expression is equivalent to Example 2 of Subsection 3.2.3, when Axis 0 is x1.
sherpa> NOTICE ALL sherpa> IGNORE FILTER :2, 7:This filter expression specifies Axis 0 data values less-than or equal-to 2, or greater-than or equal-to 7. This filter expression is equivalent to Example 3 of Subsection 3.2.3, when Axis 0 is x1.
sherpa> IGNORE ALL sherpa> NOTICE FILTER 6:7This filter expression specifies Axis 0 data values greater-than 5 and less-than 8 (i.e. 6 thru 7 inclusive). This filter expression is equivalent to Example 4 of Subsection 3.2.3, when Axis 0 is x1.
Filter expressions for 2-D data utilize filter descriptors that are combined using logical operators. Data value ranges are formed as a comma-delimited list, contained within parantheses as part of the filter descriptor.
filterExpr colname1,colname2 descriptor
operator descriptor ...
where colname1 and colname2 are the names of appropriate data columns. Note that specifying column names is not required; if not specified, Axis 0 (colname1x1) and Axis 1 (colname2x2) are the defaults. The following tables show the available filter descriptors, descriptor, and logical operators, operator:
descriptor | Description |
---|---|
BOX(x,y,width,height) | x and y are the center coordinates of the box, width is twice the distance along the first axis from the center to one edge, and height is twice the distance along the second axis from the center to the edge. |
BOX(x,y,width,height,angle) | x and y are the center coordinates of the box, width is twice the distance along the first axis from the center to one edge, height is twice the distance along the second axis from the center to the edge, and angle is the counter-clockwise rotation angle in degrees, relative to the first axis. |
CIRCLE(x,y,radius) | x and y are the center coordinates of the circle, and radius is the distance from the center to the edge of the circle. |
ELLIPSE(x,y,width,height) | x and y are the center coordinates of the ellipse, width is the distance along the first axis from the center to the edge, and height is the distance along the second axis from the center to the edge. |
ELLIPSE(x,y,width,height,angle) | x and y are the center coordinates of the ellipse, width is the distance along the first axis from the center to the edge, height is the distance along the second axis from the center to the edge, and angle is the counter-clockwise rotation angle in degrees, relative to the first axis. |
POINT(x,y) | x and y are the coordinates of a specific point. |
POLYGON(x1,y1,x2,y2,x3,y3,...,xn,yn) | Each x,y pair specifies one vertex of a polygon, up to n=32 vertices. |
PIE(x,y,start,stop) | x and y are the apex coordinates for a filter in the shape of a pie wedge. The wedge is swept out starting at the angle start and stopping at the angle stop. The angles are given in degrees, increasing in the counter-clockwise direction, relative to the first axis. |
operator | Description |
---|---|
& | and |
and | |
and not | |
exclusive or | |
| | inclusive or |
inclusive or | |
Examples:
sherpa> DATA example_img.fits sherpa> NOTICE ALL sherpa> IGNORE FILTER x1,x2 = BOX(250,250,50,50)This filter expression specifies a box of width 50, height 50, and center at x1,x2 = 250,250. The command
sherpa> IGNORE FILTER BOX(250,250,50,50)is also equivalent.
sherpa> NOTICE ALL sherpa> IGNORE FILTER BOX(250,250,50,50) & CIRCLE(275,225,36)This filter expression specifies an area enclosed both by the specified box and a circle of radius 36 centered at 275,225. The result is a pie wedge.
sherpa> NOTICE ALL sherpa> IGNORE FILTER BOX(250,250,50,50) | CIRCLE(275,225,36)This filter expression specifies an area enclosed either by the specified box or the specified circle.
sherpa> NOTICE ALL sherpa> IGNORE FILTER BOX(250,250,50,50) - CIRCLE(275,225,36)This filter expression specifies an area enclosed by the specified box, and not of a circle of radius 36 centered at 275,225.
A file that contains filter assignments for each data point may be input using the READ command. The filter assignment value should be either 1 for inclusion, or 0 for exclusion of the data point. The syntax of the READ command for an input filter file is:
sherpa> READ FILTER # filespec
See the READ command for the filespec definition and list of accepted file formats.
Examples:
sherpa> READ DATA example1.dat 1 2 sherpa> READ FILTER example1.dat 1 3 sherpa> SHOW FILTERThe first command reads columns 1 and 2 of the ascii data file example1.dat, as dataset number 1. Column 3 of example1.dat contains a filter assignment for each of the data points (1 for the data point to be included; 0 for the data point to be excluded). Current filter assignments for each data point can be reported with the command SHOW FILTER.
The CIAO Data Manipulation (DM) software allows data to be filtered during the input/output process. As part of the CIAO software package, Sherpa may utilize this capability. A short description of the Data Manipulation filtering concept and syntax is given herein. For a more detailed description of the Data Manipulation software please see the Data Manipulation Guide.
In any CIAO software package program using the DM interface (including Sherpa ), the user can pass a ``virtual file'' where an input file is expected. The virtual file is a file that is filtered (and/or binned) from an original data file, during run-time. For example, given a large 32K square HRC image, the user may wish to input only a subset of that file, with say x values within only a certain range. Using the DM filtering syntax, the user may input this filtered file, without first creating an intermediate disk file. That is, Sherpa will only ``see'' the ``virtual'' filtered file that has been specified. With the DM software, the virtual file is implicitly created from a true disk file by filtering or binning, but does not get created on disk or even formatted into a UNIX pipe. Instead, I/O calls in the program return records of the virtual file by internally accessing the actual disk file and figuring out which data passes the filter.
The full syntax for specifying a DM filter is a filename followed by a filtering and/or binning command argument, virtual_file_syntax. This command argument consists of a series of optional qualifiers within brackets, :
filename[block][filter][columns][binning]
Any qualifier my be omitted, but those which are used must appear in the order given above.
DM syntax may be used with the following file types:
filetype: | Extension: | Description: |
---|---|---|
FITS | fits | 2-D fits image |
FITSIMAGE | fits | 2-D fits image |
FITSBIN | fits | 1-D fits binary table |
IMH | imh | 2-D IRAF image |
QP | qp | 2-D PROS qpoe |
See the READ command for information regarding the Sherpa data input syntax for these file types.
Examples:
sherpa> READ DATA 2 "example_bin.fits[2][cols TIME, EXPNO]" FITSBIN sherpa> READ DATA 3 "example_bin.fits[EVENTS][cols time, expno]" FITSBINThe first command reads columns time and expno, from the second extension of the fits binary table file example_bin.fits, as dataset number 2. The second command reads columns time and expno, from the EVENTS extension of the fits binary table file example_bin.fits, as dataset number 3. Note that columns may be specified by case insensitive name. Also, the fits extension can be specified by either the number or the name of the extension (in this example, the name of the second extension is EVENTS).
sherpa> READ DATA "example_img.fits[#1=100:200, #2=100:400]" FITSThis command reads the specified portion (from Axis 0 coordinates 100 to 200 and from Axis 1 coordinates 100 to 400) of the 2-D fits image example_img.fits. Note that, by default, the data is taken from the first fits block for which NAXIS is nonzero. The following command is equivalent:
sherpa> READ DATA "example_img.fits[100:200, 100:400]" FITS
sherpa> READ DATA 1 "example_bin.fits[2][columns #1, #2]" FITSBINThis command reads the first two columns from the second extension of the fits binary table file example_bin.fits, as dataset number 1. Note that column numbers or names must always be specified when reading fits binary table files. The following commands are equivalent:
sherpa> READ DATA 1 "example_bin.fits[2][columns #1, #2]" FITSBIN sherpa> READ DATA 1 "example_bin.fits[2][cols #1, #2]" FITSBIN sherpa> DATA "example_bin.fits[2][cols #1, #2]" FITSBIN
sherpa> DATA "example.qp[bin time=32, pha=32]" FITSIMAGEThis command creates and inputs into Sherpa an image, using the pha and time columns of of the qpoe file example.qp. In this example bin sizes of 32 are used, which will in this case reduce an 8192x8192 image to 256x256. Note that the FITSIMAGE argument is required by the Sherpa READ command syntax (since it is ultimately a fits image that is being input to Sherpa ).
sherpa> DATA "example_bin.fits[bin chipx, chipy]" FITSIMAGEThis command inputs into Sherpa the fits binary table example_bin.fits, but bins the table to create an image using the chipx and chipy columns. Note that the FITSIMAGE argument is required by the Sherpa READ command syntax.
sherpa> DATA "example_bin.fits[bin chipx=200:400:4, chipy=300:400:4]" FITSIMAGEThis command creates and inputs into Sherpa , an image using the chipx and chipy columns of the binary fits table file example_bin.fits. In this example, ranges for the axes, and bin sizes, are given. The resultant image size is restricted. Note that the FITSIMAGE argument is required by the Sherpa READ command syntax. Note however, that the following command is not equivalent:
sherpa> DATA "example_bin.fits[chipx=200:400:4, chipy=300:400:4][bin chipx, chipy]" $\backslash$ \\ FITSIMAGEsince this command does not restrict the overall size of the resultant image.
sherpa> READ DATA "example_bin.fits[EVENTS][time=37277062.1:37277062.2] $\backslash$ \\ [bin X=16350:16420:1,Y=16350:16420:1]" FITSIMAGEThis command creates and inputs into Sherpa , an image using the X and Y columns of the binary fits table file example_bin.fits. In this example, filtering is performed with two columns, for filtering both spatially and in time. The spatial subspace is the subset of the event file for X between 16350 and 16420 and Y in the same range, binned in steps of 1. The time range is from 37277062.1 to 37277062.2 seconds. Note that in this case, the order in the syntax is important. This command first goes to the events block, next filters the data in time, and then makes an image by binning over the requested X and Y region. You can't filter after binning in the same operation. In addition, once the binning is done, the data no longer contains the time information. Note that the FITSIMAGE argument is required by the Sherpa READ command syntax.
cxchelp@head.cfa.harvard.edu