An Image of Diffuse Emission
CIAO 4.16 Science Threads
Overview
Synopsis:
The procedure used here is intended to make a nice image for a poster or paper and to aid in understanding the morphology of the extended emission. Point sources are identified and removed, filling the gaps with a sampling of the background region. The filled image is then smoothed, with the option to exposure-correct the results.
Care must be taken in the scientific interpretation of the final image as it is highly processed.
Purpose:
To create a smoothed image of diffuse emission, replacing point sources by a local estimate of the background emission.
Related Links:
- Analysis Guide: Extended Sources
-
The File Format section of the Using CIAO Region Files: describes how to display region files on an event list, which is used often in this thread.
- Creating Source and Background Files: this method could be used to create the source and background region files as well.
Last Update: 08 Jul 2024 - Removed warning about early dataset.
Contents
- Get Started
- Identify and Remove Point Sources
- Smooth the Image
- Exposure-Correcting the Image (Optional)
- Parameter files:
- History
-
Images
- Figure 1: Spatial variation of the PSF
- Figure 2: Diffuse emission plus point sources
- Figure 3: Original and modified source regions
- Figure 4: Source and background region created by roi
- Figure 5: Looking at the output of the roi tool
- Figure 6: Viewing the background regions
- Figure 7: Image of the diffuse emission
- Figure 8: Smoothed image of the diffuse emission
- Figure 9: Smoothed, exposure-corrected image of the diffuse emission
Get Started
Download the sample data: 315 (ACIS-S, NGC 4038/39)
unix% download_chandra_obsid 315
The data is assumed to have been downloaded from the archive - e.g. with download_chandra_obsid - then reprocessed using chandra_repro to use the latest calibration and software updates.
unix% chandra_repro . repro/ unix% fluximage "repro/acisf00315_repro_evt2.fits[ccd_id=7]" images/ binsize=1 bands=broad psfecf=0.393 Running fluximage Version: 28 November 2018 Using CSC ACIS broad science energy band. Aspect solution repro/pcadf00315_000N001_asol1.fits found. Bad-pixel file repro/acisf00315_repro_bpix1.fits found. Mask file repro/acisf00315_000N004_msk1.fits found. The output images will have 1289 by 1289 pixels, pixel size of 0.492 arcsec, and cover x=3536.5:4825.5:1,y=3171.5:4460.5:1. Running tasks in parallel with 4 processors. Creating aspect histogram for obsid 315 Creating instrument map for obsid 315 Creating exposure map for obsid 315 Thresholding data for obsid 315 Exposure-correcting image for obsid 315 Creating PSF map for obsid 315 The following files were created: The clipped counts image is: images/broad_thresh.img The clipped exposure map is: images/broad_thresh.expmap The PSF map is: images/broad_thresh.psfmap The exposure-corrected image is: images/broad_flux.img
The fluximage script now creates the PSF map needed by wavdetect when the psfecf parameter is specified.
The output of psfmap is an image where the pixel values are the size of a circular region (in arcseconds) needed to enclose a fraction of the PSF at specified energy.
unix% dmstat broad_thresh.psfmap cen- PSFMAP[arcsec] min: 0.24236044288 @: ( 4081 4155 ) max: 3.6420326233 @: ( 4826 3446 ) mean: 1.2048927606 sigma: 0.77728920391 sum: 1328687.0576 good: 1102743 null: 562645
The last line, null:562645, indicates that there are pixels in the PSF map with a value of NaN, indicating pixels in the image that are outside the instrument field-of-view, FOV.
We will use ds9 to visualize the psfmap.
% ds9 \ broad_thresh.img -scale limits 0 2 \ broad_thresh.psfmap -single \ -contour -contour smooth 1 -contour levels "0.5 1 2" -contour copy \ -frame 1 -contour paste wcs red 2 no -contour close \ -pan to 4200 3790 physical -view colorbar no \ -grid yes -grid grid no -grid type analysis -grid border no \ -grid title no -grid axes type exterior -grid numerics vertical yes
Each line is described below
- The application name
- Load the counts image and set the scale limits to 0:2
- Load the psfmap in a new frame. Set ds9 to display a single frame at a time.
- Generate contours on the psfmap at specific levels using the smooth algorithm. Copy the contours
- Switch to counts image and paste the contours from the psfmap image.
- Pan the imager to the specified physical coordinates and remove the color bar
- Turn on the coordinate grids, disable the grid lines and border
- Move the coordinate grid axes to edge of image and rotate labeles.
We have used dmstat and ds9 to look at how the PSF map varies across the image; the contours in Figure 1 show the positions where the ECF is 0.5, 1, and 2 arcseconds (for the chosen energy and ECF fraction).
[Version: full-size]
Figure 1: Spatial variation of the PSF
Identify and Remove Point Sources
1. Create an Image of the Region
Throughout this thread, we will use the image created by fluximage, namely images/broad_thresh.img, and you may proceed to source detection with wavdetect; however, should you want to select the region and energies you are interested in, it may done using dmcopy and an events file - e.g.
unix% dmcopy "repro/acisf00315_repro_evt2.fits[energy=300:7000][bin x=4004.5:4404.5:1,y=3625.5:4025.5:1]" \ original.fits
Note that the get_sky_limits tool can be used to find the sky range covered by an image if you want to match an existing image:
unix% get_sky_limits images/broad_thresh.img Running: get_sky_limits version: 12 September 2012 Checking binning of image: images/broad_thresh.img Image has 1289 x 1289 pixels Pixel size is 1.0 by 1.0 Lower left (0.5,0.5) corner is x,y= 3536.5, 3171.5 Upper right (1289.5,1289.5) corner is x,y= 4825.5, 4460.5 DM filter is: x=3536.5:4825.5:#1289,y=3171.5:4460.5:#1289 mkexpmap xygrid value is: 3536.5:4825.5:#1289,3171.5:4460.5:#1289
We use ds9 to view the image, as shown in Figure 2:
unix% ds9 images/broad_thresh.img -scale log -zoom 0.5 -smooth -invert
[Version: full-size]
Figure 2: Diffuse emission plus point sources
2. Source detection with wavdetect
We can now run wavdetect to identify the point sources. Note that here we have chosen to increase the ellsigma parameter from 3 to 4, since we wish to make sure we excise as much of each source as possible from the image. This value should be modified to suit the requirements of your analysis. Similarly the scales setting should be adjusted to match the data.
unix% punlearn wavdetect unix% pset wavdetect psffile=broad_thresh.psfmap unix% pset wavdetect regfile=sources.reg unix% pset wavdetect ellsigma=4 unix% wavdetect broad_thresh.img sources.fits sources.scell sources.image sources.nbkg wavelet scales (pixels) (2.0 4.0): Image of the size of the PSF (broad_thresh.psfmap):
The contents of the parameter file may be checked using plist wavdetect.
From inspection, one can see that many of the sources detected by wavdetect are not point sources, but clumps of diffuse emission. Scientific judgment must be used to modify or delete regions before saving the final region file. The Using the Output of Detect Tools thread shows how to display and modify source lists.
Here we manually remove those detections which do not look like point sources. In this example, the modifed source list has been saved as sources_mod.reg:
unix% head sources_mod.reg Region file format: DS9 version 4.1 global color=blue dashlist=8 3 width=1 font="helvetica 10 normal roman" select=1 highlite=1 dash=0 fixed=0 edit=1 move=1 delete=1 include=1 source=1 physical Ellipse(4399.06,3337.57,18.9642,13.2067,34.1611) # Ellipse(4304.93,3440.89,14.3973,9.63261,54.2142) # Ellipse(4409.26,3504.19,14.3911,9.85874,48.463) # Ellipse(4292.44,3614.14,7.73868,5.04972,57.2064) # Ellipse(4332.65,3659.78,5.62888,4.18144,64.6945) # Ellipse(4103.68,3663.62,9.55326,4.91821,43.1674) # Ellipse(4270.7,3663.2,3.15498,1.53822,68.7819) #
The source list can contain overlapping regions since this case is handled by the roi tool used in the next section.
[Version: full-size]
Figure 3: Original and modified source regions
We end the section by using dmmakereg to convert the modified source list into FITS format, as required by the roi tool used in the next section:
unix% punlearn dmmakereg unix% dmmakereg "region(sources_mod.reg)" sources_mod.fits wcsfile=broad_thresh.img unix% dmlist sources_mod.fits cols -------------------------------------------------------------------------------- Columns for Table Block REGION -------------------------------------------------------------------------------- ColNo Name Unit Type Range 1 POS(X,Y) pixel Real8 -Inf:+Inf Position 2 SHAPE String[16] Region shape type 3 R[2] pixel Real8(2) -Inf:+Inf Radius 4 ROTANG[2] pixel Real8(2) -Inf:+Inf Angle 5 COMPONENT Int2 - Component number
dmmakereg uses the wcsfile to convert the region from celestial coordinates (RA,Dec) to physical coordinate. It only does this if it needs to. If the input region is already in physical coordinates then the wcsfile is ignored. Simple numeric values without any annotation are treated as being in physial coordinates. A region in celestial coordinates will be recognized either by the use of colon separated sexagesimal coordinates, with the string fk5 preceeding the regions, or with the special "d" following the values to indiciate decimal degrees. More examples are shows in the DM filtering help file.
In this example, the region is already in physical coordinates so no conversion is performed.
3. Create Source and Background Regions
We now use the roi tool to create the source and background regions since it is much-more capable than the previous solution (mkBgReg.pl and mkSubBgReg.pl). This requires three stages:
-
Using roi to create source and background regions for each source;
-
combining the regions into a format that dmfilth can process;
-
and finishing with an optional validation and modification phase.
The roi tool takes care of combining overlapping source regions and excluding neighbouring sources from background regions.
First we run roi, where the output files (there will be one file for each source) are written to the sources/ sub-directory:
unix% mkdir sources unix% punlearn roi unix% pset roi infile=sources_mod.fits unix% pset roi outsrcfile=sources/src%d.fits unix% pset roi bkgfactor=0.5 unix% roi Input src list (sources_mod.fits): Input field of view region (): Input streak region (): Output source list (sources/src%d.fits): Background radius computation method (add|mul|area) (mul): Background radius (0:) (3): unix% pget roi num_srcs 58 unix% dmlist sources_mod.fits counts 60
The contents of the parameter file may be checked using plist roi. Note that we set the bkgfactor parameter to a small, positive value to ensure that the outer radius of the source region is smaller than the inner radius used in background region, since dmfilth can error out when these two boundaries are the same (Figure 4).
Figure 4: Source and background region created by roi
In this example we have barely scratched the functionality of roi. For instance, since the emission we are interested in is all contained within the S3 chip, we do not have to worry about the edges of the chip, and so we have not bothered to set the fovregion parameter. Similarly, as there are no bright sources in the field we do not use a streakregion file, such as created by acis_streak_map.
Please see the ahelp page for roi for more information on this tool.
The parameter file for roi is updated to contain the number of source created; here we have 58. This is smaller than the original number of sources (60) since roi has combined sources that overlap. This behavior can be controlled by changing the tool's group parameter.
The output of roi is a set of FITS region files, each containing a region representing the source (SRCREG) and background (BKGREG) for each source:
unix% ls -1 sources/ | head -5 sources/src1.fits sources/src10.fits sources/src11.fits sources/src12.fits sources/src13.fits unix% dmlist sources/src\*.fits blocks -------------------------------------------------------------------------------- Dataset: sources/src1.fits -------------------------------------------------------------------------------- Block Name Type Dimensions -------------------------------------------------------------------------------- Block 1: PRIMARY Null Block 2: SRCREG Table 5 cols x 1 rows Block 3: BKGREG Table 5 cols x 2 rows ... -------------------------------------------------------------------------------- Dataset: sources/src14.fits -------------------------------------------------------------------------------- Block Name Type Dimensions -------------------------------------------------------------------------------- Block 1: PRIMARY Null Block 2: SRCREG Table 5 cols x 2 rows Block 3: BKGREG Table 5 cols x 8 rows ... -------------------------------------------------------------------------------- Dataset: sources/src17.fits -------------------------------------------------------------------------------- Block Name Type Dimensions -------------------------------------------------------------------------------- Block 1: PRIMARY Null Block 2: SRCREG Table 5 cols x 1 rows Block 3: BKGREG Table 5 cols x 3 rows ...
From the dmlist output, we see that a number of sources can have multiple rows in their SRCREG block - e.g. src14.fits - or more than two rows in the BKGREG block (src14.fits and src17.fits); you expect a minimum of two as we have an ellipse minus the original source ellipse. In Figure 5 we show the source and background regions in src14.fits:
[Version: full-size]
Figure 5: Looking at the output of the roi tool
The dmfilth tool used in the next section requires two files in ASCII region format: one for the sources, and one for the backgrounds. To do this we use the splitroi script, which is part of the CIAO scripts and modules package (although, as described at the end of this section, you can use sed, or similar tools, to create these files for you.
The script takes two arguments; the first is a list of files to use, using the shell's syntax for specifying regular expressions, and the name to use as the start of the output files (".src.reg" and ".bg.reg" are appended to this value):
unix% splitroi "sources/src*.fits" exclude
We now have two ASCII region files - exclude.src.reg and exclude.bg.reg - that can be used by dmfilth. To view them in ds9 you need to convert them to FITS format using dmmakereg; e.g.
unix% dmmakereg "region(exclude.bg.reg)" exclude.bg.fits unix% ds9 broad_thresh.img -scale log -smooth -region exclude.bg.fits
creates Figure 6.
[Version: full-size]
Figure 6: Viewing the background regions
An alternative approach is to use the stack support in dmfilth and create files containing lines that say region(src1.fits) and region(src1.fits[bkgreg]) for each file; in teh following the region() syntax is needed to tell dmfilth that each line represents a region file rather than describing an actual region, as it is in the splitroi approach. For example:
unix% ls -1 sources/src*fits | sed 's/.*/region(&)/' > src.reg unix% ls -1 sources/src*fits | sed 's/.*/region(&[bkgreg])/' > bg.reg unix% head -3 src.reg region(sources/src1.fits) region(sources/src10.fits) region(sources/src11.fits) unix% head -3 bg.reg region(sources/src1.fits[bkgreg]) region(sources/src10.fits[bkgreg]) region(sources/src11.fits[bkgreg])
At this point you can inspect these region files to select which ones should be used and which ones removed. For this example we do not make any changes.
4. Fill in the Holes (dmfilth)
The tool dmfilth offers several options for extrapolating over regions (see the ahelp file for more information.) Here, we use the POISSON method, which assigns pixel values to the source region by sampling the Poisson distribution whose mean is that of the pixel values in the background region. This has been chosen for didactic purposes and may not be the best choice for your purposes (it assumes that the diffuse emission at the source has the same surface brightness as in the background regions which is questionable here for some sources, looking at Figure 6).
We run dmfilth using the region files created in the previous sections:
unix% punlearn dmfilth unix% pset dmfilth infile=broad_thresh.img unix% pset dmfilth outfile=diffuse.img unix% pset dmfilth method=POISSON unix% pset dmfilth srclist=@exclude.src.reg unix% pset dmfilth bkglist=@exclude.bg.reg unix% pset dmfilth randseed=0 unix% dmfilth Input image file (broad_thresh.img): Enter output file name (diffuse.img): Interpolation method (POLY|DIST|GLOBAL|POISSON|BILINT) (POISSON): List of sources to fill in (@exclude.src.reg): List of background regions (@exclude.bg.reg):
The contents of the parameter file may be checked using plist dmfilth.
unix% ds9 diffuse.img -smooth -scale sqrt -invert
The output file is shown in Figure 7. As expected, the point sources are no longer visible.
[Version: full-size]
Figure 7: Image of the diffuse emission
Smooth the Image
The image file is smoothed via the tool aconvolve. A gaussian is used for the kernel specification (kernelspec) and the kernel is normalized by the area (normkernel).
For this data, we define the kernelspec as lib:gaus(2,5,1,7,7). This means the Gaussian:
- has 2 dimensions;
- is embedded in an array 5 sigma in size;
- is normalized to 1;
- has a sigma of 7 pixels along each axis.
Users will have to experiment with different kernelspec definitions to find the optimal one for the dataset.
unix% punlearn aconvolve unix% pset aconvolve infile=diffuse.img unix% pset aconvolve outfile=diffuse.sm.img unix% pset aconvolve kernelspec="lib:gaus(2,5,1,7,7)" unix% pset aconvolve method=fft unix% aconvolve Input file name (diffuse.img): Kernel specification (lib:gaus(2,5,1,7,7)): Output file name (diffuse.sm.img):
The contents of the parameter file may be checked using plist aconvolve and Figure 8 shows the smoothed image, diffuse.sm.img.
unix% ds9 diffuse.sm.img -invert -scale sqrt
[Version: full-size]
Figure 8: Smoothed image of the diffuse emission
Exposure-Correcting the Image (Optional)
From this point, it is also possible to incorporate an exposure map in order to create an exposure-corrected image. Unless there are significant exposure variations across the field, this will not make a difference in the final image; exposure-correcting the data used in this thread did not have a visible effect on the output.
If you used fluximage then you have an exposure map - in this case broad_thresh.expmap, otherwise follow one of the Exposure Map threads.
Use dmimgcalc to divide the unsmoothed image (diffuse.img) by the exposure map:
unix% punlearn dmimgcalc unix% dmimgcalc diffuse.img broad_thresh.expmap diffuse_exp.img div warning: CONTENT has 1 different values. warning: DETNAM has different value...Merged...
Smooth the exposure-corrected image (diffuse_exp.img) with aconvolve, as in the Smooth the Image section:
unix% punlearn aconvolve unix% aconvolve diffuse_exp.img diffuse_exp.sm.img "lib:gauss(2,5,1,7,7)" method=fft unix% ds9 -invert -scale mode 99.5 diffuse.sm.img diffuse_exp.sm.img
In Figure 9 we show that, for this example, there are no major large-scale differences in morphologies in the two smoothed images.
[Version: full-size]
Figure 9: Smoothed, exposure-corrected image of the diffuse emission
Parameters for /home/username/cxcds_param/wavdetect.par # # parameter file for wavdetect # # # input # infile = broad_thresh.img Input file name # # output # outfile = sources.fits Output source list file name scellfile = sources.scell Output source cell image file name imagefile = sources.image Output reconstructed image file name defnbkgfile = sources.nbkg Output normalized background file name # # scales # scales = 2.0 4.0 wavelet scales (pixels) # # end of wtransform parameters # ######################################################################## ######################################################################## # # wrecon parameters # # # PSF size parameters # psffile = broad_thresh.psfmap Image of the size of the PSF (regfile = sources.reg) ASCII regions output file # # output options # (clobber = no) Overwrite existing outputs? (ellsigma = 4) Size of output source ellipses (in sigmas) (interdir = ${ASCDS_WORK_PATH} -> /tmp) Directory for intermediate outputs # ######################################################################### # # wtransform parameters # # # optional input # (bkginput = ) Input background file name (bkgerrinput = no) Use bkginput[2] for background error # # output info # (outputinfix = ) Output filename infix # # output content options # (sigthresh = 1e-06) Threshold significance for output source pixel list (bkgsigthresh = 0.001) Threshold significance when estimating bkgd only (falsesrc = -1.0) Allowed number of false sources per image (sigcalfile = ${ASCDS_CALIB}/wtsimresult.fits -> /soft/ciao/data/wtsimresult.fits) Significance calibration file # # exposure info # (exptime = 0) Exposure time (if zero, estimate from map itself (expfile = ) Exposure map file name (blank=none) (expthresh = 0.1) Minimum relative exposure needed in pixel to analyze it # # background # (bkgtime = 0) Exposure time for input background file # # iteration info # (maxiter = 2) Maximum number of source-cleansing iterations (iterstop = 0.0001) Min frac of pix that must be cleansed to continue # # end of wrecon parameters # ######################################################################## # # run log verbosity and content # (log = no) Make a log file? (verbose = 0) Log verbosity # # mode # (mode = ql)
Parameters for /home/username/cxcds_param/roi.par infile = sources_mod.fits Input src list fovregion = Input field of view region streakregion = Input streak region outsrcfile = sources/src%d.fits Output source list radiusmode = mul Background radius computation method bkgradius = 3 Background radius num_srcs = 58 Number of sources output (group = group) Make 1 srcregion per group or per individual source (targetbkg = all) Make background around all sources or just target? (bkgfactor = 0.5) Amount offset or multipled by src to be excluded by bkg (bkgfunction = add) Add bkgfactor or multiply bkgfactor (maxpix = INDEF) Maximum number of pixels when intersecting (fovres = 1) Pixellation resulolution for fov check (streakres = 0.25) Pixellation resulolution for streak check (ignore_streaksrc = yes) Ignore streak region for source list ? (evtfile = ) Event File (# counts per region) (clobber = no) Remove existing output files? (verbose = 0) tool verbosity (mode = ql)
Parameters for /home/username/cxcds_param/dmfilth.par ## ## DMFILTH -- fill in the hole ## infile = broad_thresh.img Input image file outfile = diffuse.img Enter output file name method = POISSON Interpolation method srclist = @exclude.src.reg List of sources to fill in bkglist = @exclude.bg.reg List of background regions (randseed = 0) Seed for random number generator (clobber = no) OK to overwrite existing output file(s)? (verbose = 0) Verbosity level (mode = ql)
Parameters for /home/username/cxcds_param/aconvolve.par # # aconvolve.par file # # infile = diffuse.img Input file name outfile = diffuse.sm.img Output file name kernelspec = lib:gaus(2,5,1,7,7) Kernel specification # # auxillary outputs # (writekernel = no) Output kernel (kernelfile = ./.) Output kernel file name (writefft = no) Write fft outputs (fftroot = ./.) Root name for FFT files # # processing parameters # (method = fft) Convolution method (edges = wrap) Edge treatment (const = 0) Constant value to use at edges with edges=constant (pad = no) Pad data axes to next power of 2^n (center = no) Center FFT output (normkernel = area) Normalize the kernel # # user specific comments # (clobber = no) Clobber existing output (verbose = 0) Debug level (mode = ql)
History
16 Dec 2004 | updated for CIAO 3.2: minor changes to csmooth parameter file |
21 Dec 2005 | reviewed for CIAO 3.3: no changes |
01 Dec 2006 | updated for CIAO 3.4: uses aconvolve instead of csmooth for smoothing, updated image to match |
16 Jan 2008 | updated for CIAO 4.0: kernel parameter removed from aconvolve and wavdetect |
29 Jan 2009 | updated for CIAO 4.1: images are inline |
06 May 2009 | check the version of the CIAO scripts package instead of the individual script |
05 Feb 2010 | updated for CIAO 4.2: ObsID 315 file version and corresponding changes to source detections |
13 Jan 2011 | reviewed for CIAO 4.3: no changes |
09 Jan 2012 | reviewed for CIAO 4.4: the thread now assumes that the data has been processed by chandra_repro, uses fluximage to create the necessary images, updated to use the new interface to wavdetect, and use the new tool roi and script splitroi to generate the source and background regions rather than the deprecated mkBgReg.pl and mkSubBgReg.pl routines; added Figure 9 showing the exposure-corrected version of the image. |
24 Jul 2012 | added clarifying remark to Identify and Remove Point Sources section. |
03 Dec 2012 | Review for CIAO 4.5; cleanup a bit of "new in ciao 4.4" related stuff. |
25 Nov 2013 | Review for CIAO 4.6. Highlighted fact that early mission data needs chandra_repro to update headers for parameter block keywords. |
27 Jun 2014 | updated file names and tool syntax |
17 Dec 2014 | Reviewed for CIAO 4.7. No changes. |
03 Jan 2016 | Review for CIAO 4.9. No changes. |
19 Mar 2018 | Added a note about region coordinate systems and added wcsfile to the dmmakereg command. |
10 Dec 2018 | Updated for CIAO 4.11; fluximage can now be used to create the psfmap. |
28 Mar 2019 | Recreated psfmap figure with ds9. |
13 Jan 2022 | Reviewed for CIAO 4.14. Minor tweaks to wording only. |
08 Jul 2024 | Removed warning about early dataset. |