dmascii

Description

CIAO users are familiar with the flexible filtering and binning capability that the Data Model tools provide with FITS files. Since CIAO 4.0, the same tools also work on ASCII (text) files, using a new part of the software library called the 'ASCII kernel'.

This kernel enables easy text file manipulation by DM-specific tools such as dmlist, dmcopy, dmstat, dmtcalc, and prism. However, other CIAO tools (e.g. aconvolve) are not guaranteed to work smoothly with ASCII files.

For example, a raw text file containing a table with three columns and four rows:

  unix% cat sample.dat

     21.0  41.3  21.8
     22.0  41.1  20.2
     23.0  43.8  17.3
     24.0  12.3  11.1

dmlist can be used to select a range of data from two of the columns:

  unix% dmlist "sample.dat[col3=11:20][cols col2,col3]" data,clean
  
  #  col2                 col3
                  43.80                17.30
                  12.30                11.10

This command may be repeated with dmcopy to create an output file of the filtered data in FITS or text format:

  unix%  dmcopy "sample.dat[col3=11:20][cols col2,col3]"  filtered.fits
  unix%  dmcopy "sample.dat[col3=11:20][cols col2,col3]" "filtered.txt[opt kernel=text/simple]"

Additional examples are included later is this document. Note that the DM creates FITS format output by default; the kernel option must be specified every time to make the output be a text file.

Supported Text Formats

There are currently three formats recognized by the ASCII kernel:

      text/raw
        Raw text table format consists of free-format columns with
        no header information.  There are only two supported datatypes
        for values in this format: dmDOUBLE and dmTEXT.  All columns 
        are scalar and are given the default names "col1", "col2", etc.

      text/simple
        The simple format is compatible with the SM plotting program.
        It is similar to raw, but allows for the inclusion of header 
        information provided as a series of comment lines.  Each line 
        of the header must begin with the comment character, (see 
        "comment" option).  Either the first or last line of this header 
        block, may be used to specify the table column names, (see 
        "colnames" option).  In its briefest form, the header consists 
        of a single line of column names.

      text/dtf or dtf-fixed
        Data Text Format (DTF) is a pseudo-FITS format with support for
        the full list of datatypes, header keywords and data subspaces.
        Free format is the default, but fixed-format fields are also 
        supported.  This format MUST be used in order to define an image.

      

On input, the particular ascii format will be auto-determined. The user may override this by including the "kernel" option on the input file name. Since the default output kernel is FITS, the user MUST specify the output ascii flavor by including the "kernel" option in the output file name.

Ascii Kernel Options

There are several options that may be used to tailor the interface for a particular text file. You can use these options to read text files with a slightly different structure from the default, for example by skipping header lines or changing the field separator. Multiple options should be provided as a comma-separated list.

[opt colnames={value}]

Specifies which header line defines the column names. This line begins with a comment character, followed by a list of names delimited by the same character as the data. Supported values are "first", "last", and "none". If the "none" value is given, columns are defined from the data and auto-named "col1", "col2", etc. . Column names may contain alphanumeric characters as well as the hash (#), underscore (_) or dash (-) character, although the latter should be avoided if possible. One may define array and vector columns by using the following name syntax.

            name(cpt1,cpt2) = Vector of 2 components.  e.g. POS(X,Y)
            name[size]      = Array of length n.       e.g. PHAS[3]
        

[opt comment={value}]

Lines that begin with the given character (e.g. "#") will be treated as comments. For text/raw format files, all comment lines are ignored. For text/simple format, comment lines which occur prior to the first data line will be retained, while any occuring within the data segment will be ignored. There is one special comment line which the "colnames" option controls, as described above.

[opt nullstr={value}]

On input, this value specifies an arbitrary string which should be interpreted as representing a NULL value. This is in addition to the 'default' NULL values for each datatype:

            INTEGER:  {empty}, {tnull}, -, INDEF, INF
            REAL   :  {empty}, -, NaN, INDEF
            STRING :  {empty}
        

On output, this string will be used to represent all NULL values.

[opt skip={value}]

Skip the given number of lines at the beginning of the file. This helps handle some formats with fixed headers. For example, '[opt skip=3]' will skip the first three (3) lines of the file.

[opt sep={value}]

Define the given character(s) to be the separator for data fields. Any printing ASCII character from space (' ') to tilde ('~') may be a separator character, except the single quote ('), double quote ("), and backslash (\) characters. In addition, the non-printing tab character (HT) may be used (specified as '\t'). If more than one character is to be used, or if the space or comma character is used, the list of separators must be enclosed by quotes. Each instance of the character is interpreted as a new field.

Examples:
   [opt sep=:]     = colon delimited values
   [opt sep=" "]   = space delimited values
   [opt sep=":;"]  = values delimited by EITHER colon or semi-colon

[white] qualifier

If the "white" qualifier is included, the separator is treated as whitespace. This means that if you have multiple separator characters next to each other, they only count as one separator.

The defaults for each output format are:

Limitations

The ASCII kernel was developed to allow CIAO users to use the familiar DM syntax in manipulating and filtering text files; it is not intended as a replacement for the FITS kernel in pipelines. The following are some limitations of the kernel:

• While all the basic Data Model tools should work with text file input, it has not been fully tested with other CIAO tools (e.g. csmooth), and some may not be compatible with the ASCII kernel.
• Region filtering on Ascii images is not yet supported.
• To create a dataset with more than one block or write header keywords, you MUST use the DTF flavor. Therefore, the simple and raw formats cannot be used with any CIAO tools which require multiple blocks and header keywords.

Samples

Below are basic samples of each of the ASCII formats.

  Table: text/raw
    alpha 167413425.5456684232 319 434 2119.869 4193.105
    beta  167413425.5456684232 219 532 2020.054 4096.020
    gamma 167413425.5867084265 607 483 3448.656 4140.115
    delta 167413427.1456484795 420 331 4306.355 4289.419


  Table: text/simple
    #name time tx ty sky(x,y)
    alpha 167413425.5456684232 319 434 2119.869 4193.105
    beta  167413425.5456684232 219 532 2020.054 4096.020
    gamma 167413425.5867084265 607 483 3448.656 4140.115
    delta 167413427.1456484795 420 331 4306.355 4289.419


  Table: text/dtf
    XTENSION='TABLE'
    HDUNAME = "MyINFO"
    TFIELDS = 7
    TTYPE1  = "name    "          
    TFORM1  = "10A     "           / data format of field.
    TTYPE2  = "time    "           / Time of event
    TFORM2  = "1D      "           / data format of field.
    TTYPE3  = "tx      "           / Tile position - X
    TFORM3  = "1I      "           / data format of field.
    TTYPE4  = "ty      "           / Tile position - Y
    TFORM4  = "1I      "           / data format of field.
    TTYPE5  = "x       "           / Sky position - X
    TFORM5  = "1E      "           / data format of field.
    TTYPE6  = "y       "           / Sky position - Y
    TFORM6  = "1E      "           / data format of field.
    TTYPE7  = "useFlag "           / Record usable? 
    TFORM7  = "1L      "           / data format of field.
    TTYPE8  = "status  "           / Event status bits
    TFORM8  = "3X      "           / data format of field.
    MTYPE1 = sky
    MFORM1 = x,y
    END
    
    alpha 167413425.5456684232 319 434 2119.869 4193.105 T 001
    beta  167413425.5456684232 219 532 2020.054 4096.020 T 010
    gamma 167413425.5867084265 607 483 3448.656 4140.115 F 011
    delta 167413427.1456484795 420 331 4306.355 4289.419 T 100



  Image: text/dtf
    XTENSION='IMAGE'
    HDUNAME = "MyIMG"
    BITPIX  =                   16
    NAXIS   =                    2
    NAXIS1  =                    4
    NAXIS2  =                    3
    MTYPE1  = "sky     "
    MFORM1  = "x,y     "
    END

    1  2  3  4 
    5  6  7  8 
    9 10 11 12