Last modified: December 2022

URL: https://cxc.cfa.harvard.edu/ciao/ahelp/dmtype2split.html
AHELP for CIAO 4.16

dmtype2split

Context: Tools::Table

Synopsis

Split up an array column into multiple files (e.g. PHA type 2 to PHA type 1).

Syntax

dmtype2split  infile outfile [clobber] [verbose]

Description

CIAO extends the concept of type 1 and type 2 PHA files to general tables. A type 1 file of size N is a simple table consisting of scalar columns each with N rows. A type 2 file of size N x M is a table with M rows in which each column is either scalar or an array of size N. Each row of the type 2 file corresponds to a whole type 1 file, and `dmtype2split' performs this mapping.

The mapping consists of turning each array column entry from `sideways' to `vertical', so that an array of 4096 values becomes a column with 4096 rows. Scalar columns map to keywords in the output file.

Each of the specified rows in the type 2 file is copied to a type 1 file. Each output file has the basic header keywords and subspace copied from the input. If the input file is a pha file, keywords required by the OGIP format are also written; otherwise, generic keywords are written. For each scalar column in the input, the coordinate keyword information is written, and the datatype, value, and unit are written as a keyword. For each input column that is an array, a column is created from the datatype, value, and unit; the row of input data is read and written to the output column.

Specifying Rows

The rows can be specified in any order, but the Data Model sorts them numerically before processing. If supplying several input rows and a stack of output filenames, it is recommended that the rows be given in order so that the mapping to the filename is clear. The rows may be comma- or colon-delimited, e.g., "1:4,6,9:12". A colon-delimited interval is inclusive (e.g., "2:5" means rows 2, 3, 4, 5). If the number of rows specified does not equal the number of output files specified, the program exits with an error message. If no rows are specified, rows 1:n are processed, where n is the number of files in the outfile stack.

Specifying Output Block Name

By default, the output block name will be "SPECTRUM" since the most common use of this tool is the TYPE:II PHA files. Users can choose a different output block name by specifying it as part of the outfile file name.


Examples

Example 1

In this example we are going to take the following ASCII file, which contains two colums - the scalar x and vector y with 4 elements - and split it into two FITS files:

unix% cat vals.dat
# x y[4]
1.2 2.3 4.5 6.2 -3.4
4.5 0.1 2.3e2 -2301 43.9
unix% dmlist vals.dat blocks
 
--------------------------------------------------------------------------------
Dataset: vals.dat
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: vals.dat                       Table         2 cols x 2        rows
unix% dmlist vals.dat cols
 
--------------------------------------------------------------------------------
Columns for Table Block vals.dat
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range
   1   x                                 Real8          -Inf:+Inf            
   2   y[4]                              Real8(4)       -Inf:+Inf            
unix% dmtype2split vals.dat v1.fits,v2.fits
unix% dmlist v1.fits blocks
 
--------------------------------------------------------------------------------
Dataset: v1.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: SPECTRUM                       Table         1 cols x 4        rows
unix% dmlist v1.fits cols
 
--------------------------------------------------------------------------------
Columns for Table Block SPECTRUM
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range
   1   y                                 Real8          -Inf:+Inf            
unix% dmlist v1.fits data,clean
#  y
                 2.30
                 4.50
                 6.20
                -3.40

The first file, v1.fits, contains a single column, y, with 4 rows, containing the values of the first y[4] array in vals.dat. As the x column in vals.dat is a scalar, the value for the first row is written as a keyword rather than a column; e.g.

unix% dmkeypar v1.fits x echo+
1.2

Example 2

unix% cat out.stk
v1.dat
v2.dat
unix% dmtype2split vals.dat @out.stk"[opt kernel=text/dtf]"

This is a repeat of the previous example, but this time the output format has been changed to one of the ASCII formats supported by CIAO, rather than FITS. The dmascii ahelp page contains more information on the ASCII support in the CIAO Data Model. An external stack file was used so that the opt specifier only needed to be given once, rather than once per file; ahelp stack contains more information on how stacks are used in the CIAO tools.

In this case the "DTF" flavor was chosen because it is the only format that supports keywords. If "text/simple" had been used for the kernel instead, the output files would not contain any information on the value of the x column.

Example 3

unix% dmtype2split pha2.fits @outlist

Create output files (names in the stack outlist, (see ahelp stack for more information), one per row of the input file pha2.fits.

Example 4

unix% dmtype2split "pha2.fits[#row=1:3,6]" @outlist

Extract the first four rows and the sixth row from pha2.fits. The output stack file outlist must contain five file names.

Example 5

unix% dmmakereg "polygon(1,2,3,4,5,6)" poly.fits
unix% dmtype2split "poly.fits[cols x,y]" poly_table.fits"[REGION]"

In this example we create a FITS region file with 1 row where the X and Y columns are 4 element arrays.

unix% dmlist poly.fits cols 
--------------------------------------------------------------------------------
Columns for Table Block REGION
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range
   1   POS(X,Y)[4]          pixel        Real8(4)       -Inf:+Inf            Position
   2   SHAPE                             String[16]                          Region shape type
   3   R[2]                 pixel        Real8(2)       -Inf:+Inf            Radius
   4   ROTANG[2]            deg          Real8(2)       -Inf:+Inf            Angle
   5   COMPONENT                         Int2           -                    Component number

We then use dmtype2split to transpose that single row into a single table with scalar X and Y columns. The output block name, "REGION", is specified as part of the output file name. The output file then looks like

unix% dmlist poly_table.fits blocks,cols
 
--------------------------------------------------------------------------------
Dataset: poly_table.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: REGION                         Table         2 cols x 4        rows
 
--------------------------------------------------------------------------------
Columns for Table Block REGION
--------------------------------------------------------------------------------
 
ColNo  Name                 Unit        Type             Range
   1   X                    pixel        Real8          -Inf:+Inf            
   2   Y                    pixel        Real8          -Inf:+Inf            

Parameters

name type ftype def min max reqd stacks
infile file input       yes  
outfile file output       yes yes
clobber boolean   no        
verbose integer   0 0 5    

Detailed Parameter Descriptions

Parameter=infile (file required filetype=input)

Input type II file

The input file should contain at least one array column; that is, a dmlist opt=cols call on the file name returns at least one column with "[n]" after its name, where n is an integer.

Parameter=outfile (file required filetype=output stacks=yes)

Enter output file name(s)

The number of file names must equal the number of rows in the input file.

Parameter=clobber (boolean default=no)

OK to overwrite existing output file(s)?

Parameter=verbose (integer default=0 min=0 max=5)

Verbosity level


Bugs

Corrupted error message

The error message that dmtype2split uses to report a problem opening the input file can contain random characters.

Caveats

Does not support files with string columns

dmtype2split does not support files with string columns. They must be removed from the file for dmtype2split to reformat the remaining columns.

The extension name for the output file must be included as part of the output file name(s).

Since dmtype2split is used for many different types of files it does not set the extension name, rather leaving it up to the user to specify. If the block name is omitted, the basename (no path) of the file name is used.

unix% dmtype2split "pha2[#row=1]" pha1
unix% dmlist pha1 blocks
 
--------------------------------------------------------------------------------
Dataset: pha1
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: pha1                           Table         7 cols x 8192     rows

Workaround:

As some analysis systems require certain files to contain specific block names to comply with different standards, the block name may need to be set as shown below

unix% dmtype2split "pha2[#row=1]" "pha1[SPECTRUM]"
unix% dmlist pha1 blocks
 
--------------------------------------------------------------------------------
Dataset: pha1
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: PRIMARY                        Null        
Block    2: SPECTRUM                       Table         7 cols x 8192     rows

See Also

chandra
eventdef
dm
dmfiltering, dmopt
tools::composite
combine_grating_spectra
tools::core
dmappend
tools::gratings
detilt, dewiggle, symmetrize, tg_bkg, tg_choose_method, tg_create_mask, tg_findzo, tg_resolve_events, tgdetect, tgdetect2, tgextract, tgextract2, tgidselectsrc, tgmask2reg, tgmatchsrc, tgsplit
tools::response
mktgresp
tools::table
dmgroup, dmjoin, dmmerge, dmpaste, dmsort, dmtabfilt, dmtcalc
tools::timing
dmgti