Chandra X-Ray Observatory
	(CXC)
Skip to the navigation links
Last modified: 29 Dec 2016

URL: http://cxc.harvard.edu/ciao/threads/detect_output/

Using the Output of Detect Tools

CIAO 4.9 Science Threads


Overview

Synopsis:

All three CIAO source detect programs celldetect, wavdetect, and vtpdetect have a common output format, the _src file; see the Output of Detect: _src Files section of the Detect Overview thread for more information. This thread shows how the source list can be displayed, edited, and used in CIAO data analysis.

Last Update: 29 Dec 2016 - Reviewed for CIAO 4.9; minor edits only.


Contents


Getting Started

This thread uses products created in the Running celldetect thread.


Displaying the Source List

The sources can easily be displayed overlaid on an image:

unix%  ds9 s3_evt2.fits &

Load the region file from the ds9 menu "Regions" → "Load Regions..." → s3_src.reg (or s3_src.fits).

Either command will produce Figure 1.

[Thumbnail image: ]

[Version: full-size]

[Print media version: ]

Figure 1: Source list overlaid on the event data

The event file is displayed at a blocking factor of 2.


Examining the Source List

The contents of the source list can be viewed with dmlist:

unix% dmlist s3_src.fits blocks
 
--------------------------------------------------------------------------------
Dataset: s3_src.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: SRCLIST                        Table        31 cols x 15       rows

There are 15 sources (one per row) in the file. To see information about the individual sources:

unix% dmlist "s3_src.fits[cols RA,DEC,POS]" data
 
--------------------------------------------------------------------------------
Data for Table Block SRCLIST
--------------------------------------------------------------------------------
 
ROW    RA                   DEC                  POS(X,Y)                                
 
     1       212.8639501188        52.1921273978 (     3961.7571428571,     4035.0714285714)
     2       212.8474548928        52.2255111480 (     4035.7917808219,     4279.3205479452)
     3       212.8360726227        52.2021137798 (     4086.8032786885,     4108.1147540984)
...
    13       212.7873357718        52.2792122394       (     4305.0,     4672.3170731707)
    14       212.7665879663        52.2989206268 (     4397.7476635514,     4816.5981308411)
    15       212.8535943060        52.3232348586 (     4008.4683544304,     4994.3797468354)

Here the RA, Dec, and sky position of each source is given. Any of the columns in the file can be used in the DM filter.


Manipulating the Source List

It is easy to delete unwanted sources from or rearrange the order of the ASCII output (region file) of the detect tools via a text editor. Using the Data Model tools, it is just as straightforward to manipulate the FITS source file as well.

Edit the List of Sources

Once the source list is overlaid on the image, you may see some sources that you want to exclude from the analysis. Currently, ds9 does not number the sources in the field of view. However, double-clicking on a source region brings up the region info box, which lists the region "ID". In Figure 2, the source on the left side of the field of view was chosen - the ID is 12. This number corresponds to the row the region occupies in the FITS file.

NOTE: this technique only works with the first region file loaded into ds9. After that, the imager starts counting up. For example, if you load region_a.fits that has 10 sources, then region_b.fits that has 6, the regions of the second file will be counted from 11 onward. One may, of course, always examine the region file visually to determine which rows to delete.

To create a new region file that doesn't have row 12 in it, dmcopy is used:

unix% dmcopy "s3_src.fits[#row=1:11,13:15]" s3_1gone_src.fits

unix% dmlist s3_1gone_src.fits blocks
 
--------------------------------------------------------------------------------
Dataset: s3_1gone_src.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: SRCLIST                        Table        31 cols x 14       rows

The dmlist command shows that there are only 14 rows in the file now. This new file may be displayed in ds9 (Figure 3) to ensure that the correct source was removed.

[Thumbnail image: The source list with one source removed is displayed on the data.]

[Version: full-size]

[Print media version: The source list with one source removed is displayed on the data.]

Figure 3: Confirming a removed source

The unwanted source (blue cross) is no longer present in the region file (green ellipses).

This filter was fairly simple. It is possible to make more complicated filters with the DM syntax. If you wanted to exclude rows (sources) 8, 12, and 14 from the original file:

unix% dmcopy "s3_src.fits[#row=1:7,9:11,13,15]" s3_3gone_src.fits

unix% dmlist s3_3gone_src.fits blocks
 
--------------------------------------------------------------------------------
Dataset: s3_3gone_src.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: SRCLIST                        Table        31 cols x 12       rows

unix% dmlist s3_3gone_src.fits"[cols ra,dec,net_counts]" data

--------------------------------------------------------------------------------
Data for Table Block SRCLIST
--------------------------------------------------------------------------------
 
ROW    RA                   DEC                  NET_COUNTS
 
     1       212.8639501188        52.1921273978               63.250
     2       212.8474548928        52.2255111480                320.0
     3       212.8360726227        52.2021137798             33.43750
     4       212.8353651758        52.2027847341             87.81250
     5       212.8349227409        52.2031706722              67.6250
     6       212.8067722124        52.2280881209             29.31250
     7       212.8783563224        52.2399625097        31.4285717010
     8       212.8360692107        52.2365737442        20.4285717010
     9       212.8309806527        52.2335386574       125.5714263916
    10       212.7481005259        52.1982204514        51.1428565979
    11       212.7873357718        52.2792122394                 32.0
    12       212.8535943060        52.3232348586        54.7449989319

Sort on a Particular Column

The tool dmsort makes it possible to sort on any column in the FITS file. To order the rows by NET_COUNTS:

unix% dmlist "s3_src.fits[cols net_counts]" data
 
--------------------------------------------------------------------------------
Data for Table Block SRCLIST
--------------------------------------------------------------------------------
 
ROW    NET_COUNTS  
 
     1               63.250
     2                320.0
     3             33.43750
     4             87.81250
     5              67.6250
     6             29.31250
     7        31.4285717010
     8        23.4285717010
     9        20.4285717010
    10       125.5714263916
    11        51.1428565979
    12        25.3977279663
    13                 32.0
    14        76.7916641235
    15        54.7449989319


unix% dmsort s3_src.fits s3_sorted_src.fits keys=net_counts copyall=yes

unix% dmlist "s3_sorted_src.fits[cols net_counts]" data
 
--------------------------------------------------------------------------------
Data for Table Block SRCLIST
--------------------------------------------------------------------------------
 
ROW    NET_COUNTS  
 
     1        20.4285717010
     2        23.4285717010
     3        25.3977279663
     4             29.31250
     5        31.4285717010
     6                 32.0
     7             33.43750
     8        51.1428565979
     9        54.7449989319
    10               63.250
    11              67.6250
    12        76.7916641235
    13             87.81250
    14       125.5714263916
    15                320.0

The default behavior of dmsort is to organize the results in ascending order. To sort NET_COUNTS in descending order, add a dash ("-") before the column name:

unix% dmsort s3_src.fits s3_descend_src.fits keys=-net_counts copyall=yes

unix% dmlist "s3_descend_src.fits[cols net_counts]" data
 
--------------------------------------------------------------------------------
Data for Table Block SRCLIST
--------------------------------------------------------------------------------
 
ROW    NET_COUNTS  
 
     1                320.0
     2       125.5714263916
     3             87.81250
     4        76.7916641235
     5              67.6250
     6               63.250
     7        54.7449989319
     8        51.1428565979
     9             33.43750
    10                 32.0
    11        31.4285717010
    12             29.31250
    13        25.3977279663
    14        23.4285717010
    15        20.4285717010

It is also possible to have several sort priorities (i.e. sort by NET_COUNTS, then NET_COUNTS_ERR); see ahelp dmsort for more information on how to sort files.


Filtering the Source List

If you are interested in only using a certain range of data in your analysis, use the DM filtering syntax to create a new source list with only those items.

To keep the sources that have 40 or more net counts:

unix% dmcopy "s3_src.fits[net_counts=40:]" s3_40cts_src.fits
unix% dmlist "s3_40cts_src.fits[cols net_counts]" data
 
--------------------------------------------------------------------------------
Data for Table Block SRCLIST
--------------------------------------------------------------------------------
 
ROW    NET_COUNTS  

     1               63.250
     2                320.0
     3             87.81250
     4              67.6250
     5       125.5714263916
     6        51.1428565979
     7        76.7916641235
     8        54.7449989319

or those with NET_COUNTS between 25 and 80:

unix% dmcopy "s3_src.fits[net_counts=25:80]" s3_25-80cts_src.fits
unix% dmlist "s3_25-80cts_src.fits[cols net_counts]" data
 
--------------------------------------------------------------------------------
Data for Table Block SRCLIST
--------------------------------------------------------------------------------
 
ROW    NET_COUNTS  
 
     1               63.250
     2             33.43750
     3              67.6250
     4             29.31250
     5        31.4285717010
     6        51.1428565979
     7        25.3977279663
     8                 32.0
     9        76.7916641235
    10        54.7449989319

See ahelp dmfiltering for other variations on the filtering syntax.


Converting Coordinates to Sexagesimal

Often it is desirable to have the list of sources in sexagesimal format, i.e. (RA,Dec) = (hms,dms). This can be done in ds9 or with prop_precess.

  • In ds9: load the event file and display the source list on top of it. To save the regions in another format:

    1. Region → Save Regions... → Save As "s3_src_ds9.reg".
    2. After choosing "OK" in the region filename dialog, a format dialog is opened. Set the format to "CIAO" and the coordinate system to "WCS" and "Equatorial J2000".

    The output (an ASCII region file) still contains the shape definitions, but the coordinates are now given as sexagesimal:

    unix% cat s3_src_ds9.reg
    # Region file format: CIAO version 1.0
    ellipse(14:11:27.347,+52:11:31.66,0.0189976',0.0173604',105.875)
    ellipse(14:11:23.389,+52:13:31.84,0.0189597',0.0151168',31.9394)
    ellipse(14:11:20.657,+52:12:07.61,0.0199099',0.0169644',46.4835)
    ellipse(14:11:20.487,+52:12:10.03,0.0191015',0.0167311',150.605)
    ellipse(14:11:20.381,+52:12:11.42,0.019486',0.0166947',58.3717)
    ellipse(14:11:13.625,+52:13:41.12,0.0163116',0.0149855',114.407)
    ellipse(14:11:30.805,+52:14:23.87,0.0383147',0.0177815',38.4504)
    ellipse(14:11:29.107,+52:13:32.99,0.0271043',0.0226476',149.574)
    ellipse(14:11:20.656,+52:14:11.67,0.0307874',0.0263703',20.7717)
    ellipse(14:11:19.435,+52:14:00.74,0.0259394',0.0182104',26.632)
    ellipse(14:10:59.544,+52:11:53.60,0.0412003',0.0231457',162.832)
    ellipse(14:11:14.446,+52:16:11.20,0.0585538',0.0422066',0.0725874)
    ellipse(14:11:08.960,+52:16:45.17,0.0681381',0.0629584',165.325)
    ellipse(14:11:03.981,+52:17:56.12,0.107363',0.0757421',18.5694)
    ellipse(14:11:24.862,+52:19:23.65,0.12849',0.10075',9.90983)
    

    The file may be redisplayed in ds9, as it retains the CIAO region format.

  • With prop_precess: prop_precess is a proposal tool that may be used to convert the (RA,Dec) in the FITS source list from decimal to sexagesimal format.

    unix% dmlist s3_src.fits"[cols ra,dec]" data,clean > decimal.txt
    
    unix% prop_precess f j2000/deg t j2000/hms : decimal.txt : hms.txt
    Opened input file decimal.txt
    Opened output file hms.txt
    
    unix% cat hms.txt
    RA,Dec J2000.0                          RA,Dec J2000.0
    212.863950    52.192127                 14 11 27.35 +52 11 31.66
    212.847455    52.225511                 14 11 23.39 +52 13 31.84
    212.836073    52.202114                 14 11 20.66 +52 12 07.61
    212.835365    52.202785                 14 11 20.49 +52 12 10.03
    212.834923    52.203171                 14 11 20.38 +52 12 11.41
    212.806772    52.228088                 14 11 13.63 +52 13 41.12
    212.878356    52.239963                 14 11 30.81 +52 14 23.87
    212.871281    52.225830                 14 11 29.11 +52 13 32.99
    212.836069    52.236574                 14 11 20.66 +52 14 11.67
    212.830981    52.233539                 14 11 19.44 +52 14 00.74
    212.748101    52.198220                 14 10 59.54 +52 11 53.59
    212.810193    52.269778                 14 11 14.45 +52 16 11.20
    212.787336    52.279212                 14 11 08.96 +52 16 45.16
    212.766588    52.298921                 14 11 03.98 +52 17 56.11
    212.853594    52.323235                 14 11 24.86 +52 19 23.65
    

Using the Source List in Analysis

Before beginning any analysis with the source list, be aware that the scaling of the size of each region is controlled by the ellsigma parameter in each of the detect tools. From the help files:

ellsigma is a multiplicative factor applied to sigma ... to scale the major and minor 
axes of the ellipses for each source. ellsigma affects both the outfile and the ASCII 
region file (regfile). This feature is included so that the graphics overlay will be 
more visible and under the user's control.

Carefully consider a reasonable value for ellsigma before using these regions for analysis.

Creating Individual Spectra

It is possible to create a spectrum from any of the regions in the source list without having to divide up the file. To extract a PI spectrum of the second source in the file and bin it by a factor of 5:

unix% punlearn dmextract
unix% dmextract "s3_evt2.fits[sky=region(s3_src.fits[#row=2])][bin pi=::1]" \
      spectrum.fits

The contents of the parameter file may be checked using plist dmextract.

The resultant spectrum can be plotted in ChIPS.

unix% chips
-----------------------------------------
Welcome to ChIPS: CXC's Plotting Package
-----------------------------------------

chips> make_figure("spectrum.fits[cols channel,counts]","symbol.style=none")
chips> limits(X_AXIS,0,100)

Figure 4 shows the plot. This data may also be fit in Sherpa, once the proper response files have been created; see the Step-by-Step Guide to Creating ACIS Spectra for Pointlike Sources thread for more information.

[Thumbnail image: The spectrum is plotted as solid white line without symbols or errors.]

[Version: full-size, PNG]

[Print media version: The spectrum is plotted as solid white line without symbols or errors.]

Figure 4: Plot of the spectrum in ChIPS

This spectrum was created from a single source in the event file (id=2).

Quit ChIPS before continuing.

chips> quit

Creating Multiple Spectra

To create an individual spectrum for each of the sources, use the igrid(1:n:steps) stack syntax, where n is the number of sources in the file.

For s3_src.fits, which has 15 sources in it:

unix% dmextract "s3_evt2.fits[sky=region(s3_src.fits[component=igrid(1:15:1)])][bin pi=::1]" \
      multi_spectra_phaII.fits op=pha2

The output file is a type II PHA file that has one spectrum per row:

unix% dmlist multi_spectra_phaII.fits blocks
 
--------------------------------------------------------------------------------
Dataset: multi_spectra_phaII.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: SPECTRUM                       Table         7 cols x 15       rows
Block    3: GTI7                           Table         2 cols x 1        rows

For more information on working with this type of file, see the Displaying the Spectrum section of the Examining Grating Spectra and Regions: PHA2 files thread (the information in that section applies to imaging files as well).


Creating a Composite Spectrum

It is also possible to extract a composite spectrum of all the (faint) sources in a field for a single spectral analysis. This may be useful in the case where the sources might be of similar characteristics, but there are not enough counts in any given source to do individual spectra.

Here we have deleted the sources associated with 3C295 (IDs 3-5):

unix% dmcopy "s3_src.fits[#row=1:2,6:]" s3_nocore_src.fits

Now we can use dmextract to accept all the regions together and create a spectrum:

unix% dmextract "s3_evt2.fits[sky=region(s3_nocore_src.fits)][bin pi=::5]" \
      spectrum_faint.fits

Again, the resultant spectrum can be plotted in ChIPS:

unix% chips
-----------------------------------------
Welcome to ChIPS: CXC's Plotting Package
-----------------------------------------

chips> make_figure("spectrum_faint.fits[cols channel,counts]","symbol.style=none")
chips> limits(X_AXIS,0,100)

Figure 5 shows the plot.

[Thumbnail image: The spectrum is plotted as solid white line without symbols or errors.]

[Version: full-size, PNG]

[Print media version: The spectrum is plotted as solid white line without symbols or errors.]

Figure 5: Composite spectrum

The composite spectrum was created by combining counts from 14 sources.

As mentioned before, this data may also be fit in Sherpa, once the proper response files have been created; see the Step-by-Step Guide to Creating ACIS Spectra for Pointlike Sources thread for more information.

Quit ChIPS before continuing.

chips> quit

Removing Unwanted Point Sources

Often it is desirable to remove point sources from an image for both cosmetic and scientific reasons. For example, if you are analyzing an extended source, an unfortunately located point source may affect the count rates in the object.

Suppose that we want to remove all the point sources from our field of view, except for the three in the core (sources 3-5). First, create a source list that excludes these three regions:

unix% dmcopy "s3_src.fits[#row=1:2,6:15]" s3_holes_src.fits

Now filter the event file with those regions:

unix% dmcopy "s3_evt2.fits[exclude sky=region(s3_holes_src.fits)]" s3_holes_evt2.fits

unix% ds9 s3_holes_evt2.fits &

The resulting image, Figure 6, has holes where the sources used to be. The Obtain and Fit a Radial Profile thread illustrates another case in which you might use this syntax (see the Removing Contaminating Point Sources section).

[Thumbnail image: The sources have been removed from the event file.]

[Version: full-size]

[Print media version: The sources have been removed from the event file.]

Figure 6: Event file with sources removed

The source regions are replaced by areas of zero counts.

A more sophisticated means of removing point sources is to use dmfilth. This tool removes the point source and then fills in the hole where the source was. See ahelp dmfilth for more information and the Identify and Remove Point Sources section of the Image of Diffuse Emission thread for an example.


Using detect output with ds9's catalog tool

Often we think of using the output of the source detect tools as spatial filters or regions. These filters may be used to extract spectra or light curves, or even just smaller images for spatial analysis. However, the source lists, properly vetted, is also a catalog of sources with properties such as fluxes.

ds9 has a very power catalog tool that allows users to overlay multiple catalogs on an image and to perform various kinds of analysis including cross-matching between multiple catalogs. Users unfamiliar with the ds9 catalog may find this tutorial useful:

First we need to convert the region file format into a catalog table. ds9 provides native support for the Virtual Observatory VOtable format but will also import files tab separated column values (tsv). The datamodel ASCII kernel will create such a file using the [opt kernel=text/tsv]" however, array columns and vector columns are not translated into a format that ds9 understands. Therefore we have provided a little script to fix the CIAO .tsv format. Begin by downloading this script.

We can then use this script to convert the FITS source regions into .tsv format.

unix% python src2tsv s3_expmap_src.fits s3_src.tsv
unix% ds9 s3_broad_thresh.img

To load the catalog we use the Catalog tool.

  1. Goto Analysis → Catalog Tool ... and a new window will appear.
  2. In that window goto File → Import → Tab-Separated-Value ...
  3. Select the file from file browser

you will have something resembling Figure 7 and the sources will be identified in the main ds9 window with Figure 8.

[Thumbnail image: ]

[Version: full-size]

[Print media version: ]

Figure 7: ds9 Catalog Tool

The ds9 catalog after importing the source list created by running celldetect with an exposure map.

[Thumbnail image: ]

[Version: full-size]

[Print media version: ]

Figure 8: ds9 Catalog Overlay

Catalog sources are by default identified with green circle points. This can be changed in the Symbol editor that is accessed via the Catalog Tool window.

Once the catalog is loaded you can then plot various of the sources properties, modify the symbols, or even do a cross match with another catalog.

For example, we can load overlay the 2MASS Point Sources catalog by simply going to Analysis → Catalogs → Infrared → 2MASS Point Sources. This will open a new Catalog Tool window, and after changing the symbol to boxes and making them Magenta, we get an image like Figure 9

[Thumbnail image: ]

[Version: full-size]

[Print media version: ]

Figure 9: ds9 catalog with 2MASS sources overlaid

Both the Chandra source list (green circles) and the 2MASS Point Source catalog (magenta squares) are now overlaid on the image.

Now with two catalogs loaded you can perform a simple cross match using the Analysis → Catalogs → Match tool that finds 3 matches using the default search radius. The result is a new Catalog Tool window (Figure 10) and after editing the shape to be a cyan colored box, we can overlay with the other two catalogs as in Figure 11.

[Thumbnail image: ]

[Version: full-size]

[Print media version: ]

Figure 10: Results from Cross Match with 2MASS

The results from doing a cross match of our source list with the 2MASS Point Source catalog. Only 3 matches are identified.

[Thumbnail image: ]

[Version: full-size]

[Print media version: ]

Figure 11: Displaying Cross Match with 2MASS

The results from doing a cross match of our source list with the 2MASS Point Source catalog (large cyan boxes) overlaid with the original image, Chandra sources (small green boxes), and 2MASS sources (magenta circles)

There are many things that can be done with the data once it has been converted to catalog format and loaded into ds9. The symbol editor can be used to create very complex annotations, properties can be plotted, and with a SAMP hub running, the data can be shared among various other Virtual Observatory applications including topcat.


Parameters for /home/username/cxcds_param/dmextract.par


#--------------------------------------------------------------------
#
# DMEXTRACT -- extract columns or counts from an event list
#
#--------------------------------------------------------------------
        infile = s3_evt2.fits[sky=region(s3_src.fits[#row=2])][bin pi=::5] Input event file 
       outfile = spectrum_2.fits  Enter output file name
          (bkg = )                Background region file or fixed background (counts/pixel/s) subtraction
        (error = gaussian)        Method for error determination(poisson|gaussian|<variance file>)
     (bkgerror = gaussian)        Method for background error determination(poisson|gaussian|<variance file>)
      (bkgnorm = 1.0)             Background normalization
          (exp = )                Exposure map image file
       (bkgexp = )                Background exposure map image file
      (sys_err = 0)               Fixed systematic error value for SYS_ERR keyword
          (opt = pha1)            Output file type: pha1 
     (defaults = ${ASCDS_CALIB}/cxo.mdb -> /soft/ciao-4.0/data/cxo.mdb) Instrument defaults file
         (wmap = )                WMAP filter/binning (e.g. det=8 or default)
      (clobber = no)              OK to overwrite existing output file(s)?
      (verbose = 0)               Verbosity level
         (mode = ql)              


History

03 Jan 2005 reviewed for CIAO 3.2: no changes
19 Dec 2005 updated for CIAO 3.3: default value of dmextract error and bkgerror parameters is "gaussian", screen output updated accordingly; updated files in detect data tarfile
01 Dec 2006 updated for CIAO 3.4: ChIPS version
17 Jan 2008 updated for CIAO 4.0: ChIPS plotting syntax updated; ds9 now automatically looks for the "[SRCLIST]" extension in the region file, so it doesn't have to be specified; deg2sexag.sl has not been updated for CIAO 4.0; removed data tarfile; deg2sexag.sl script replaced by prop_precess
13 Jan 2009 updated for CIAO 4.1: provided both Python and S-lang syntax for ChIPS; images are inline
29 Jan 2010 updated for CIAO 4.2: changes to the ds9 region file format menu
13 Jan 2011 reviewed for CIAO 4.3: no changes
09 Jan 2012 reviewed for CIAO 4.4: minor updates to screen output due to changes in the celldetect source list
03 Dec 2012 Review for CIAO 4.5; minor version updates; minor prop_precess changes.
25 Apr 2013 Added new section to use detect output with ds9 catalog tool.
25 Nov 2013 Review for CIAO 4.6. No changes.
16 Dec 2014 Reviewed for CIAO 4.7; minor text edits.
29 Dec 2016 Reviewed for CIAO 4.9; minor edits only.


Last modified: 29 Dec 2016
Smithsonian Institute Smithsonian Institute

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email:   cxchelp@head.cfa.harvard.edu Smithsonian Institution, Copyright © 1998-2017. All rights reserved.