Bugs: Data Model

A list of bugs fixed in CIAO 4.0 is available.

Caveats

1. The Data Model does not support writing to gzip'ed files.

Bugs

General

1. The BLANK header keyword that an older version of some FTOOLs (e.g. chimgtyp) write to the FITS file header causes problems for dmcopy.

2. If you use "[opt null=nan]" on an integer image, the BLANK keyword is set to 0, since nan isn't a valid integer.

3. You may get a seg fault if you try to create a very large image. What constitutes "very large" depends on the data type, but for long and float images, 8192x8192 pixels seems to be the threshold.

4. # DMCOPY (CIAO 4.0): Bad data type in filter string formatting

5. No more than nine subspace components are allowed. (01 Dec 2006)

6. Block names containing blank spaces are treated as bad syntax

Filtering Data

1. Multi-valued string filters don't work

2. Creating a vector on-the-fly when region filtering

3. Applying a bit-filter expression to an integer column does not work, nor does it cause an error.

4. Using incorrect syntax with the rectangle shape does not fail when filtering.

5. Filtering an image on logical coordinates causes problems when the short cut of omitting a number (i.e. to indicate the default value) is used.

6. Trying to exclude a region filter with update=no will cause the image to be filtered by the region's bounding box.

7. There is a bug in region and range filters.

8. Using "!=" with string filtering causes dmlist to fail. (05 Mar 2007)

ASCII Kernel

1. When a file has a string-valued column, warning messages about 'too many array elements' and 'truncation of data' appear. (11 Dec 2007)

2. Using non-default separators without the 'white' option (11 Dec 2007)

3. Copying a DTF-FIXED format file with two columns which abut each other (11 Dec 2007)

4. Virtual columns are not supported. (11 Dec 2007)

5. Autodetermination of column type fails with whitespace. (15 Jan 2008)

6. DTF-FIXED keyword comments may be truncated. (15 Jan 2008)

7. Units not recorded in the "units" keyword field. (15 Jan 2008)

Rebinning Images

1. Rebinning an image with different values for the two axes causes the coordinate information to be lost

Caveats

1. The Data Model does not support writing to gzip'ed files.

   The Data Model does not support writing to gzip'ed files, even if the CIAO installation was configured with gzip support. General write support for gzip'ed files will not be added anytime in the near future, so users should gunzip their files first if it is necessary to modify them.

Bugs

General

1. The BLANK header keyword that an older version of some FTOOLs (e.g. chimgtyp) write to the FITS file header causes problems for dmcopy.

   Workarounds:

   1. Use a CIAO tool in place of the FTOOL that created the BLANK keyword, e.g. instead of chimgtyp, try

      unix% dmcopy input.fits"[opt type=i4,null=-9999]" output.fits 
      

      changing the opt type as appropriate.

   2. Delete the BLANK keyword before passing the file to dmcopy or any other CIAO tool.

2. If you use "[opt null=nan]" on an integer image, the BLANK keyword is set to 0, since nan isn't a valid integer.

   Users should specify "[opt null=-999]", or some other number that is not likely to be a valid pixel value.

3. You may get a seg fault if you try to create a very large image. What constitutes "very large" depends on the data type, but for long and float images, 8192x8192 pixels seems to be the threshold.

   (The image doesn't have to be square; it just needs to have 8192^2 pixels.)

   This condition may be met when the "update=no" option is used. Normally, when you filter a dataset, the data subspace (which describes the boundaries of each column's data and therefore is the intersection of the initial minima and maxima with any subsequent filters) gets updated to reflect the filtering. However, when you give the "update=no" option, you instruct the DM not to update the subspace to reflect the current filter. Therefore, the full ranges for x and y are used in the binning, and you get a 8192x8192 image (and a seg fault, for the reason described above).

4. # DMCOPY (CIAO 4.0): Bad data type in filter string formatting

   If an unsigned byte column exists in the file, the DM will generate this warning. The output is unaffected by this bug.

5. No more than nine subspace components are allowed. (01 Dec 2006)

   The Data Model doesn't allow a file to have more than nine subspace components. If the input file has more than that number, only the first nine are copied to the output file.

6. Block names containing blank spaces are treated as bad syntax

   If the extension in the input file name has a space in it, the CIAO tools will fail, e.g.

   unix% rmfimg "conx-ea-101106-b10.rsp[SPECRESP MATRIX]" rsp.img
   #   (CIAO 4.0): Failed to open conx-ea-101106-b10.rsp[SPECRESP MATRIX]: 
   Must have RMF file 
   

   Workaround:

   For most tools, refering to the block by number instead of name will allow the tool to run, e.g. "conx-ea-101106-b10.rsp[3]".

Filtering Data

1. Multi-valued string filters don't work

   "col=foo" is okay, but "col=foo,bar" isn't.

   Workaround:

   Use "col=foo,col=bar" instead.

2. Creating a vector on-the-fly when region filtering

   When region-filtering images, you can create a vector on the fly from any two axes by using a filter like "(#1,#3)=circle(...)". Although the image is filtered correctly with a temporary vector, the region filter isn't recorded in the subspace. Hence, tools that use the filtered file don't know that pixels outside the filter region are invalid. As a result, dmstat reports no nulls in the filtered image (unless you explicitly tell the DM to set pixels outside the filter to null by using "opt null=...").

3. Applying a bit-filter expression to an integer column does not work, nor does it cause an error.

4. Using incorrect syntax with the rectangle shape does not fail when filtering.

   For example, setting xmax > xmin and/or ymax > ymin. Instead it appears that the Data Model simply swaps the min and max values.

5. Filtering an image on logical coordinates causes problems when the short cut of omitting a number (i.e. to indicate the default value) is used.

   The exit status of dmcopy is also incorrectly set to 0 (success):

   unix% dmcopy "image.fits[#1=1:20,#2=:]" delme.fits
   # DMCOPY (CIAO 4.0): [ftColRead]: FITS error 308 bad first element number in 
   dataset image.fits Block 1 PRIMARY
   
   unix% echo $status
   0
   

   Workarounds:

   1. Omit the "#2=:" from the filter

   2. Specify a range for both elements: [#1=1:20,#2=1:20]

6. Trying to exclude a region filter with update=no will cause the image to be filtered by the region's bounding box.

   For example:

   unix% dmcopy "acis_img.fits[exclude sky=region(src.fits)][opt full,update=no]" filtered.fits 
   

   The regions are correctly excluded; however, the image is also clipped at the bounding box around all the excluded shapes, so the corners of a few chips are removed.

   Workarounds:

   1. Remove update=no. In this case, the Data Model internally inverts all exclude filters to be an inclusive filter, and correctly filters the image.

      Be aware that this process is much slower if the region is large. In that case, it will also add a large region keyword to the file's header, noticeably slowing down any operation on that file.

   2. For ASCII region files, it is also possible to manually invert the filter in the file. The "field()" region syntax is used to include the entire field, then remove the undesired sources. For instance,

      # Region file format: CIAO version 1.0
      circle(1635.5,4113.5,135.11408)
      circle(3975,4233,20)
      circle(2565.5,4129.5,40)
      circle(2129.5,4007.5,40)
      

      would become

      # Region file format: CIAO version 1.0
      field()
      -circle(1635.5,4113.5,135.11408)
      -circle(3975,4233,20)
      -circle(2565.5,4129.5,40)
      -circle(2129.5,4007.5,40)
      

      and the dmcopy filtering command would be

      unix% dmcopy "acis_img.fits[sky=region(src.ascii)][opt full,update=no]" filtered.fits 
      

7. There is a bug in region and range filters.

   This command:

   unix% dmcopy "input.fits[sky=circle(4096,4096,100),y=4020:4100,4250,4350]" \
         output.fits
   

   fails to do the y filter altogether. This bug also applies to exclude filters.

8. Using "!=" with string filtering causes dmlist to fail. (05 Mar 2007)

   For example, the following commands both fail:

   unix% dmlist "region.fits[shape!=Annulus]" data
   
   unix% dmlist catalog.fits"[COMMENT!='weak'][cols COMMENT]" data
   

ASCII Kernel

1. When a file has a string-valued column, warning messages about 'too many array elements' and 'truncation of data' appear. (11 Dec 2007)

   These are (almost always) spurious and may be ignored.

2. Using non-default separators without the 'white' option (11 Dec 2007)

   If you use non-default separators without the 'white' option, values are parsed strictly between the separators treating whitespace as significant, so that with '[opt sep=&]'

   &4.3& 
   

   is read as a number, but

   & 4.3 & 
   

   is read as a string with value " 4.3 " (with leading and trailing space). This means you won't be able to apply numeric filtering to that column.

3. Copying a DTF-FIXED format file with two columns which abut each other (11 Dec 2007)

   Copying a DTF-FIXED format file with two columns which abut each other directly - i.e. no space between the columns - does not work. One byte of the data is lost.

4. Virtual columns are not supported. (11 Dec 2007)

   In the DM, you can normally do

   unix% dmlist evt.fits"[cols ra,dec]"  data
   

   even though RA and Dec are just coordinate systems defined on the X and Y columns in the file; the DM applies the transform on the fly. This doesn't work yet for ASCII files.

5. Autodetermination of column type fails with whitespace. (15 Jan 2008)

   If there is whitespace surrounding a value and the "white" option is not specified, the ASCII kernel set the column datatype to "string":

   unix% cat input.dat
   Kosmos-251 & 12 & a
   Kosmos-251 & 12 & a
   Kosmos-251 & 12 & a
   
   unix% dmlist input.dat"[opt sep=&]" cols
   
   --------------------------------------------------------------
   Columns for Table Block input.dat
   --------------------------------------------------------------
   
   ColNo  Name                 Unit        Type             Range
      1   col1                              String[1024]
      2   col2                              String[1024]
      3   col3                              String[1024]
   
   
   unix% dmlist input.dat"[opt sep=& ,white]" cols
   
   --------------------------------------------------------------
   Columns for Table Block input.dat
   --------------------------------------------------------------
   
   ColNo  Name                 Unit        Type             Range
      1   col1                              String[1024]
      2   col2                              Real8          -Inf:+Inf
      3   col3                              String[1024]
   

6. DTF-FIXED keyword comments may be truncated. (15 Jan 2008)

   DTF-FIXED header lines may be up to 1024 characters long. However, if the keyword is longer than the FITS standard, the comment is truncated.

   unix% input.txt output.dtf'[opt kernel=text/dtf-fixed]'
   

   In input.txt:

   TTYPE14 = 'Class' / LV Class Exo: M = missile [B = tactical ballistic  
   missile (except Redstone) apo=80:200]
   R = research rocket O = orbital LV V = RTV Y = Exo weather rocket X =  
   Big test rocket D= Deep space launch
   

   In output.dtf:

   TTYPE14 = "Class " / LV Class Exo: M = missile [B = tactical  
   ballistic missile (except Redstone) apo
   

7. Units not recorded in the "units" keyword field. (15 Jan 2008)

   The ASCII kernel does not recognize the "units" segment of a keyword line. The syntax is:

   keyword-name whitespace = whitespace value whitespace / [unit]  description
   

   For example, if the header of samp.dtf contains

   #TEXT/DTF
   KEYSAMP = "test" / [pixel] Test keyword with units
   ONTIME0 = 7.9552000074089E+03 / [s] Sum of GTIs
   END
   
   unix% dmcopy samp.dtf samp.fits
   
   unix% dmlist samp.dtf opt=header
   0001 KEYSAMP              test                           String       [pixel] Test keyword with units
   0002 ONTIME0                   7955.2000074089           Real8        [s] Sum of GTIs
   
   unix% dmlist samp.fits opt=header
   0002 KEYSAMP              test [pixel]                   String       Test keyword with units
   0003 ONTIME0                   7955.2000074089 [s]       Real8        Sum of GTIs
   

   In DTF format, the [units] segment stays with the comment, while the FITS version places it in the units field for the keyword.

Rebinning Images

1. Rebinning an image with different values for the two axes causes the coordinate information to be lost

   For example:

   unix% dmcopy acis.img"[bin x=::5,y=::6]" acis5x6.img
   

   Using the same value for both axes works correctly:

   unix% dmcopy acis.img"[bin (x,y)=::5]" acis5.img
   

Bugs fixed in CIAO 4.0

The following is a list of bugs that were fixed in the CIAO 4.0 software release.

1. The DM does not complain when given a component in a FITS region file that doesn't exist (e.g. "file.fits[component=99]" when there's no component numbered 99).

   Since no rows in the file match that filter, you get an output file with no rows.