Bugs: Data Model

• as values get larger (somewhere between 3456789012345677 and 5678900123456779) the binning boundaries stop adjusting by 0.5 pixels. The math is called, it is just that the precision of the value (as double) at that point no longer retains that level of precision.
• values greater than 2^53 (approx 9x10^15) will not cast with integer level precision, resulting in incorrect bin bounds, and number of bins
• values greater than LONGLONG_MAX (9223372036854775807) have an additional issue preventing it from applying the 0.5 pixel offset for the binning bounds. Since, as stated above, the action has no effect, this is a secondary concern since these values don't retain even integer level precision.

Application of the binning is done in 'double' space, possibly applying a weight factor. As such, data values > 2^53 will not cast to double with integer precision and will be distributed as if they were double-precision data

The code has been migrated to use a centralized parsing method which performs more type-based safeguards against over/under flow

• overflow cases now cap at the type_MIN|MAX and return -1 (BAD) rather than wrapping the value and returning 2 (GOOD)
• long/ulong: now caps at 32-bit limits; where before it returned the converted values up to 64-bit limits before wrapping, ( the size of long on 64-bit platforms ). FITS/DM considers 'long' to be 32-bit integers.
• float overflow: now caps at FLT_MIN|MAX; previously returned Inf
• now handles (u)longlong cases.

CRVALn (coordinate reference value) is stored in the FITS header in Scientific Notation with 13 decimal place precision (eg: 5.6789001234568E+15). For Bin ranges larger than 10^13, precision will be lost, and the physical axis coordinates read out will not match what they were originally assigned.

When creating a file with a Linear WCS, the CXCDM adjusts the transform parameters to force a half bin offset. While mathematically consistent with the values input, the result may be confusing especially when the transform is well known, eg °C to °F.

-------------------------------------------------------------------------------
World Coord Transforms for Columns in Table Block simple2.dat
--------------------------------------------------------------------------------

ColNo Name
2: tempF = +32.90 [degree F] +1.80 * (tempC -0.50)

which is mathematically correct, but more commonly written as just

   tempF = +32 + 1.8 * tempC

There are three issues with the generic use of TC* keywords in FITS files read and written by DM tools.

The first issue: TC*n[A-Z]. The DM function that composes this keyword does not strip off the 'P' from the before adding number and letter to the end.

Second issue: TCTY* keys are not recognized and therefore are not processed.

Third issue: DM is not always retaining the T*NAM information when it stores information on the DM descriptor, resulting in output keywords T*TYP instead.

When a FITS file is copied, the EXTNAME is forced on output to be the same as the HDUNAME. Typical Chandra/CIAO files require these to be the same; however, data from other missions and projects may not have the same requirement.

This bug is triggered when filtering an image with the [exclude ] synatx with a region together with the [opt full] directive to retain the original image size.

unix% dmcopy "img.fits[exclude sky=region(ciao.reg)][opt full]" filt_img.fits

The pixels outside a box that bounds the region are also filtered out (ie set to 0).

Workaround:

In many cases users can easily invert the logic in their region files to avoid needing to use the [exclude ] directive. This can be done any time the region only contains included shapes. For example:

unix% cat ciao.reg
circle(...A...)
circle(...B...)
circle(...C...)
...

unix% cat ciao.reg | awk ' BEGIN {print "field()"} {print "-"$0}' > exclude_ciao.reg
unix% cat exclude_ciao.reg
field()
-circle(...A...)
-circle(...B...)
-circle(...C...)
...

This file can then be used without needing to use the [exclude ] synatx

unix% dmcopy "img.fits[sky=region(exclude_ciao.reg)][opt full]" filt_img.fits

This has the advantage of also generally being much faster.

dm region filtering with unnamed axes may fail for complex regions, eg

dmlist my_img.fits[circle(10,10,10)-box(0,0,30,30)]

The work around is to explicilty use the axis name, eg

dmlist my_img.fits[sky=circle(10,10,10)-box(0,0,30,30)]

When filtering on WCS columns, the range is taken by converting range of the parent columns and using those as the limits of the WCS columns. When the transform is highly non-linear, eg the TAN-P transform used to go from DETX,DETY to THETA,PHI, this can leads to incorrect limits and incorrect filters. Users who want to filter on WCS columns should give explict ranges and not rely on the computed min/maxes.

bad%  dmcopy "evt.fits[theta=:1]"
good% dmcopy "evt.fits[theta=0:1]"

When region-filtering images, you can create a vector on the fly from any two axes by using a filter like "(#1,#3)=circle(...)". Although the image is filtered correctly with a temporary vector, the region filter isn't recorded in the subspace. Hence, tools that use the filtered file don't know that pixels outside the filter region are invalid. As a result, dmstat reports no nulls in the filtered image (unless you explicitly tell the DM to set pixels outside the filter to null by using "opt null=...").

Trying to use one column that is part of a vector column in a region expression can lead to a crash

% dmlist hrcf04482N003_evt2.fits"[(tg_lam,tg_d)=field()]" counts
4381489
# 24359: Received error signal SIGSEGV-segmentation violation.
# 24359: An invalid memory reference was made.
# 24359: segmentation fault: DMLIST (1) is:
exit_upon_error->NULL   

Where tg_d is part of the rd = (tg_r,tg_d) vector column.

The only work-around is to create a temporary file that dismantels the vector column by removing the other column

% dmcopy hrcf04482N003_evt2.fits"[cols -tg_r]" tmp_evt
% dmcopy tmp_evt"[(tg_lam,tg_d)=field()]" counts
4381489

Filter on an array column is not supported. The tool will run; however, the results are unpredictable.

SIMPLE ASCII files cannot have 0 rows. If there are 0 rows, then the column header definition row is incorrectly read. For example:

unix% echo "#foo" > bar
unix% dmlist bar cols

--------------------------------------------------------------------------------
Columns for Table Block bar
--------------------------------------------------------------------------------

ColNo  Name                 Unit        Type             Range
   1   foo                               String[4]

unix% dmlist bar data

--------------------------------------------------------------------------------
Data for Table Block bar
--------------------------------------------------------------------------------

ROW    foo

     1 #foo


unix% dmlist bar counts
1

The SIMPLE ASCII file should have 0 rows and we see from the data option that the column definition row has been read in as data.

In the DM, you can normally do

unix% dmlist evt.fits"[cols ra,dec]"  data

even though RA and Dec are just coordinate systems defined on the X and Y columns in the file; the DM applies the transform on the fly. This doesn't work yet for ASCII files.

DTF-FIXED header lines may be up to 1024 characters long. However, if the keyword is longer than the FITS standard, the comment is truncated.

unix% input.txt output.dtf'[opt kernel=text/dtf-fixed]'

In input.txt:

TTYPE14 = 'Class' / LV Class Exo: M = missile [B = tactical ballistic
missile (except Redstone) apo=80:200]
R = research rocket O = orbital LV V = RTV Y = Exo weather rocket X =
Big test rocket D= Deep space launch

In output.dtf:

TTYPE14 = "Class " / LV Class Exo: M = missile [B = tactical
ballistic missile (except Redstone) apo

ASCII files may contain keyword and column descriptions longer than 80 characters. This can cause random failures in the DM (unable to open columns, wrong data type, etc).

For example:

unix% dmcopy acis.img"[bin x=::5,y=::6]" acis5x6.img

Using the same value for both axes works correctly:

unix% dmcopy acis.img"[bin (x,y)=::5]" acis5.img

For images without a physical coordinate system the DM will internally create one. This is not written to the output file which may lead to errors if the image size changes due to spatial filtering as the output file then has a different physical WCS compared to the input file.