|
|
Bugs: Data Model
A list of
bugs fixed in CIAO 4.1
is available.
Caveats
The Data Model does not support writing to gzip'ed files.
Bugs
If you use "[opt null=nan]" on an integer image,
the BLANK keyword is set to 0, since nan isn't a valid
integer.
You may get a seg fault if you try to create a very large
image. What constitutes "very large" depends on the data
type, but for long and float images, 8192x8192 pixels seems
to be the threshold.
# DMCOPY (CIAO 4.1): Bad data type in filter string formatting
Header comment values may not exceed 81 characters.
(17 Jul 2009)
Multi-valued string filters don't work
Creating a vector on-the-fly when region filtering
Applying a bit-filter expression to an integer column does
not work, nor does it cause an error.
Using incorrect syntax with the rectangle shape does not
fail when filtering.
Filtering an image on logical coordinates causes problems
when the short cut of omitting a number (i.e. to indicate
the default value) is used.
Trying to exclude a region filter with update=no
will cause the image to be filtered by the region's bounding
box.
There is a bug in region and range filters.
Using "!=" with string filtering causes dmlist to fail.
(05 Mar 2007)
The "or" syntax ("||") doesn't work on string columns
(17 Jul 2009)
Virtual columns are not supported.
(11 Dec 2007)
DTF-FIXED keyword comments may be truncated.
(15 Jan 2008)
Rebinning an image with different values for the two axes
causes the coordinate information to be lost
The DM doesn't catch incorrect binning syntax
(17 Jul 2009)
Running a DM tool on an image where one of the axes has been
filtered results in an error.
(17 Jul 2009)
Caveats
-
The Data Model does not support writing to gzip'ed files.
The Data Model does not support writing to gzip'ed files,
even if the CIAO installation was configured with
gzip support. General write support for gzip'ed files will
not be added anytime in the near future, so users should
gunzip their files first if it is necessary to modify
them.
Bugs
-
If you use "[opt null=nan]" on an integer image,
the BLANK keyword is set to 0, since nan isn't a valid
integer.
Users should specify "[opt null=-999]", or some
other number that is not likely to be a valid pixel
value.
-
You may get a seg fault if you try to create a very large
image. What constitutes "very large" depends on the data
type, but for long and float images, 8192x8192 pixels seems
to be the threshold.
(The image doesn't have to be square; it just needs to
have 8192^2 pixels.)
This condition may be met when the "update=no"
option is used. Normally, when you filter a dataset, the
data subspace (which describes the boundaries of each
column's data and therefore is the intersection of the
initial minima and maxima with any subsequent filters)
gets updated to reflect the filtering. However, when you
give the "update=no" option, you instruct the DM not to
update the subspace to reflect the current filter.
Therefore, the full ranges for x and y are used in the
binning, and you get a 8192x8192 image (and a seg fault,
for the reason described above).
-
# DMCOPY (CIAO 4.1): Bad data type in filter string formatting
If a logical column exists in the file, the DM will
generate this warning. The output is unaffected by this
bug.
Header comment values may not exceed 81 characters.
(17 Jul 2009)
-
Multi-valued string filters don't work
"col=foo" is okay, but "col=foo,bar"
isn't.
Workaround:
Use "col=foo,col=bar" instead.
-
Creating a vector on-the-fly when region filtering
When region-filtering images, you can create a vector on
the fly from any two axes by using a filter like
"(#1,#3)=circle(...)". Although the image is
filtered correctly with a temporary vector, the region
filter isn't recorded in the subspace. Hence, tools that
use the filtered file don't know that pixels outside the
filter region are invalid. As a result, dmstat
reports no nulls in the filtered image (unless you
explicitly tell the DM to set pixels outside the filter to
null by using "opt null=...").
Applying a bit-filter expression to an integer column does
not work, nor does it cause an error.
-
Using incorrect syntax with the rectangle shape does not
fail when filtering.
For example, setting xmax >
xmin and/or ymax > ymin.
Instead it appears that the Data Model simply swaps the min
and max values.
-
Filtering an image on logical coordinates causes problems
when the short cut of omitting a number (i.e. to indicate
the default value) is used.
The exit status of dmcopy is also incorrectly set to 0
(success):
unix% dmcopy "image.fits[#1=1:20,#2=:]" delme.fits
# DMCOPY (CIAO 4.1): [ftColRead]: FITS error 308 bad first element number in
dataset image.fits Block 1 PRIMARY
unix% echo $status
0
Workarounds:
Omit the "#2=:" from the filter
Specify a range for both elements:
[#1=1:20,#2=1:20]
-
Trying to exclude a region filter with update=no
will cause the image to be filtered by the region's bounding
box.
For example:
unix% dmcopy "acis_img.fits[exclude sky=region(src.fits)][opt full,update=no]" filtered.fits
The regions are correctly excluded; however, the image is
also clipped at the bounding box around all the excluded
shapes, so the corners of a few chips are removed.
Workarounds:
-
Remove update=no. In this case, the Data Model
internally inverts all exclude filters to be an inclusive
filter, and correctly filters the image.
Be aware that this process is much slower if the region is
large. In that case, it will also add a large region
keyword to the file's header, noticeably slowing down any
operation on that file.
-
For ASCII region files, it is also possible to manually
invert the filter in the file. The "field()"
region syntax is used to include the entire field, then
remove the undesired sources. For instance,
# Region file format: CIAO version 1.0
circle(1635.5,4113.5,135.11408)
circle(3975,4233,20)
circle(2565.5,4129.5,40)
circle(2129.5,4007.5,40)
would become
# Region file format: CIAO version 1.0
field()
-circle(1635.5,4113.5,135.11408)
-circle(3975,4233,20)
-circle(2565.5,4129.5,40)
-circle(2129.5,4007.5,40)
and the dmcopy filtering command would be
unix% dmcopy "acis_img.fits[sky=region(src.ascii)][opt full,update=no]" filtered.fits
-
There is a bug in region and range filters.
This command:
unix% dmcopy "input.fits[sky=circle(4096,4096,100),y=4020:4100,4250,4350]" \
output.fits
fails to do the y filter altogether. This bug also applies
to exclude filters.
-
Using "!=" with string filtering causes dmlist to fail.
(05 Mar 2007)
For example, the following commands both fail:
unix% dmlist "region.fits[shape!=Annulus]" data
unix% dmlist catalog.fits"[COMMENT!='weak'][cols COMMENT]" data
-
The "or" syntax ("||") doesn't work on string columns
(17 Jul 2009)
For example, this command does not find all instances of
"PMterm" in the selected columns:
unix% dmlist stat.fits"[src=PMterm||det=PMterm||mst=PMterm]" counts
13
Compare to
unix% dmlist stat.fits"[cols det,src,mst]" data,clean | grep PMterm | wc -l
27
-
Virtual columns are not supported.
(11 Dec 2007)
In the DM, you can normally do
unix% dmlist evt.fits"[cols ra,dec]" data
even though RA and Dec are just coordinate systems defined
on the X and Y columns in the file; the DM applies the
transform on the fly. This doesn't work yet for ASCII
files.
-
DTF-FIXED keyword comments may be truncated.
(15 Jan 2008)
DTF-FIXED header lines may be up to 1024 characters long.
However, if the keyword is longer than the FITS standard,
the comment is truncated.
unix% input.txt output.dtf'[opt kernel=text/dtf-fixed]'
In input.txt:
TTYPE14 = 'Class' / LV Class Exo: M = missile [B = tactical ballistic
missile (except Redstone) apo=80:200]
R = research rocket O = orbital LV V = RTV Y = Exo weather rocket X =
Big test rocket D= Deep space launch
In output.dtf:
TTYPE14 = "Class " / LV Class Exo: M = missile [B = tactical
ballistic missile (except Redstone) apo
-
Rebinning an image with different values for the two axes
causes the coordinate information to be lost
For example:
unix% dmcopy acis.img"[bin x=::5,y=::6]" acis5x6.img
Using the same value for both axes works correctly:
unix% dmcopy acis.img"[bin (x,y)=::5]" acis5.img
-
The DM doesn't catch incorrect binning syntax
(17 Jul 2009)
unix% dmcopy foo.lis"[bin col1=0.5:1024.5:1,col2=0.5:1024:5:1]"
foo.img
The bad syntax in the col2 binning - 0.5:1024:5:1
- should produce an error. Instead it ignores the ":1"
and bins col2 to a length of 205.
-
Running a DM tool on an image where one of the axes has been
filtered results in an error.
(17 Jul 2009)
unix% dmextract
Input event file (ccd3.sky4.fits[y=3767:][bin sky=annulus(3786,3767,0:380:4)]):
Enter output file name (rprof.fits):
# dmextract (CIAO 4.0 Beta 2): WARNING: Input file, "ccd3.sky4.fits[y=3767:]",
has no rows in it.
Bus error
The following is a list of bugs that were fixed
in the CIAO 4.1
software release.
-
Copying a DTF-FIXED format file with two columns which abut
each other
(11 Dec 2007)
Copying a DTF-FIXED format file with two columns which
abut each other directly - i.e. no space between the
columns - does not work. One byte of the data is lost.
-
Block names containing blank spaces are treated as bad syntax
If the extension in the input file name has a space in it,
the CIAO tools will fail, e.g.
unix% rmfimg "conx-ea-101106-b10.rsp[SPECRESP MATRIX]" rsp.img
# (CIAO 4.1): Failed to open conx-ea-101106-b10.rsp[SPECRESP MATRIX]:
Must have RMF file
Workaround:
For most tools, refering to the block by number instead of
name will allow the tool to run,
e.g. "conx-ea-101106-b10.rsp[3]".
-
When a file has a string-valued column, warning messages
about 'too many array elements' and 'truncation of data'
appear.
(ASCII kernel)
(11 Dec 2007)
These are (almost always) spurious and may be ignored.
-
Using non-default separators without the 'white' option
(ASCII kernel)
(11 Dec 2007)
If you use non-default separators without the 'white'
option, values are parsed strictly between the separators
treating whitespace as significant, so that with '[opt
sep=&]'
is read as a number, but
is read as a string with value " 4.3 " (with
leading and trailing space). This means you won't be able
to apply numeric filtering to that column.
-
Autodetermination of column type fails with whitespace.
(15 Jan 2008)
If there is whitespace surrounding a value and the
"white" option is not specified, the ASCII kernel
set the column datatype to "string":
unix% cat input.dat
Kosmos-251 & 12 & a
Kosmos-251 & 12 & a
Kosmos-251 & 12 & a
unix% dmlist input.dat"[opt sep=&]" cols
--------------------------------------------------------------
Columns for Table Block input.dat
--------------------------------------------------------------
ColNo Name Unit Type Range
1 col1 String[1024]
2 col2 String[1024]
3 col3 String[1024]
unix% dmlist input.dat"[opt sep=& ,white]" cols
--------------------------------------------------------------
Columns for Table Block input.dat
--------------------------------------------------------------
ColNo Name Unit Type Range
1 col1 String[1024]
2 col2 Real8 -Inf:+Inf
3 col3 String[1024]
-
Units not recorded in the "units" keyword field.
(ASCII kernel)
(15 Jan 2008)
The ASCII kernel does not recognize the "units" segment of
a keyword line. The syntax is:
keyword-name whitespace = whitespace value whitespace / [unit] description
For example, if the header of samp.dtf contains
#TEXT/DTF
KEYSAMP = "test" / [pixel] Test keyword with units
ONTIME0 = 7.9552000074089E+03 / [s] Sum of GTIs
END
unix% dmcopy samp.dtf samp.fits
unix% dmlist samp.dtf opt=header
0001 KEYSAMP test String [pixel] Test keyword with units
0002 ONTIME0 7955.2000074089 Real8 [s] Sum of GTIs
unix% dmlist samp.fits opt=header
0002 KEYSAMP test [pixel] String Test keyword with units
0003 ONTIME0 7955.2000074089 [s] Real8 Sum of GTIs
In DTF format, the [units] segment stays with the comment,
while the FITS version places it in the units field for
the keyword.
|
![[CIAO Logo]](../imgs/ciao_logo_navbar.gif){kind=logo}