Skip to the navigation links
Last modified: 24 October 2023

URL: https://cxc.cfa.harvard.edu/ciao/bugs/dmtcalc.html

Bugs: dmtcalc


Table of Contents

Caveats

Bugs


Caveats

Output columns with same name as input require special consideration.

For example, The expression "if(a)then(b)" does not work as expected in dmtcalc if two different columns are used in the comparison.

unix% dmtcalc in.fits out.fits "expr=if(pi>500)then(pi=-1)" 

works as expected ("pi" used for both pieces of the conditional), but

unix% dmtcalc in.fits out.fits "expr=if(energy>5000)then(pi=-1)"

does not work. This is because the pi value in the expression is not always initialized to the input. Instead you must explicitly initialize it

unix% dmtcalc in.fits out.fits "expr=pi=pi;if(energy>5000)then(pi=-1)"

The #nan can be use for float-point NaN checks, but integer NULL values cannot be checked the same way.


Bugs

log() of integer values

Using the log or log10 operator on integer columns returns an integer value. The value is truncated.

Workaround:

If unsure of the datatype, the value can be explicitly cast to a real-valued number

newcol=log((float)oldcol)
newcol=log(oldcol*1.0)
  

When using status bits in an expression, only the last condition is checked.

When using status bits in an expression such "status==X1F,X3T,X18F", only the last condition (here, "X18F") is checked.

Workaround:

Set the logic explicitly:

  • logical "AND": status=((bits==X1T)&&(bits==X2T))
  • logical "OR": status=((bits==X1T)||(bits==X2T))

The tool does not follow the usual order of operations of mathematics when evaluating an expression.

For example, the expression

x+814*24*3600-260086780.04

is treated as

x+(814*24*(3600-260086780.04))

instead of the expected

x+(814*24*3600)-260086780.04

Workaround:

Include the parentheses, as shown in the last code snippet, to force the correct order of operations.

When creating a new double vector column, the results are incorrect if one (but not both) of the values in the expression are integers, even if it is cast as a double.

For example, this file will work:

unix% cat ok.expr
(r=(double){3.5,4.}) 

unix% dmlist output_ok.fits'[cols r]' data
---------------------------------------------
Data for Table Block HISTOGRAM
---------------------------------------------

ROW R[2]

     1 [ 3.50 4.0]
     2 [ 3.50 4.0]
     3 [ 3.50 4.0]
(etc.)

while this does not:

unix% cat not_ok.expr
(r=(double){3.5,4}) 

unix% dmlist output_not_ok.fits'[cols r]' data
-------------------------------------------------
Data for Table Block HISTOGRAM
-------------------------------------------------

ROW R[2]

     1 [ 3.50 0]
     2 [ 3.50 0]
     3 [ 3.50 0] 
(etc.)

Virtual columns cannot be used in an expression

For example, when using an event list, one cannot say

unix%  dmtcalc evt1.fits foo.fits exp='dra=(ra-ra_nom)'

because the 'ra' column is not a real column in the file; see the virtual columns dictionary entry for an explanation.

Workaround:

Use Data Model column renaming:

unix%  dmtcalc evt1.fits"[cols foo=ra,*]" foo.fits expr='dra=(foo-ra_nom)'

Units are not preserved, even for simple calculations.

If you calculate npha as npha=pha; which should have units of PHA [adu]; but it will be unitless. Even more confusing is pha=pha will be unitless since pha in the output file is treated a new column.

The speed basically scales as O(N^2): small files are not so bad, but big files slow down exponentially.

Problems accessing individual elements in a vector and/or array column

There are various problems doing things like:

pha[9]=1;
foo=pha[9];

where you try to access the individual elements in a vector and/or array column.

Changing the value of an existing column

There are issues when changing values of existing columns: what info is kept, what is thrown away (e.g. data types, null, units, descriptions, order in file). In particular there are various problems with arrays vs. vector vs. vector-array columns.

There is no way to access elements of an array column such as PHAS, which is a 3x3 array (or 5x5 in VFAINT mode).

Booleans do not work on array columns

For example, you cannot do above=(phas>20) since the PHAS array contains 9 (or 25) values. Currently only the first value in the array is checked.

Using vector components in a complicated expression may cause dmtcalc to hang.  (01 Jun 2006)

For example:

expression="dist=((4125-x)^2)-((4025-y)^2)"

Workaround:

Refer to the component by vector notation instead:

expression="dist=((4125-sky[0])^2)-((4025-sky[1])^2)"

dmtcalc tries to do this subsitution internally in the first example, but fails.

dmtcalc cannot be used to modify values of vector columns.

Vector columns such as SKY(X,Y), POS(X,Y), CHIP(CHIPX,CHIPY), RD(TG_R,TG_D), cannot be modified using dmtcalc.

Internally, dmtcalc treats vector columns as arrays. So SKY(X,Y) becomes SKY[0] and SKY[1] which is what is written on output if the values are set. For example

unix% dmtcalc wav_srclist.fits modified_srclist expression="pos=pos"
unix% dmlist wav_srclist.fits | grep POS
   5   POS(X,Y)             pixel        Real8          2288.50:     4900.50 Physical coordinates
5:    EQSRC(RA_SRC ) = (+230.1725)[deg] +TAN[(-0.000136667)* (POS(X)-(+4096.50))]
ciao:/data/da/Docs/ciaoweb/ciao47/bugs% 

unix% dmlist modified_srclist | grep -i POS
 26   POS[2]                            Real8(2)       -Inf:+Inf            User defined column

shows that just copying the values in the POS vector column creates a POS array.

Workaround:

Users can modify vector columns using the Python Crates package

from pycrates import read_file

tab = read_file("wav_srclist.fits")
pos = tab.get_column("pos").values
pos[0][0] = new_X_value
pos[0][1] = new_Y_value
tab.write("modified_srclist")

The valid range of vector columns is not preserved in the output file, even when not used or modified.

When the input file contains a vector column, the legal range of values stored in the raw FITS TLMIN/TLMAX keywords associated with each raw column are not preserved. The MIN/MAX for each raw column in the vector are set equal to the 1st column.

So in this example, a wavdetect source list with the vector column POS(X,Y) (which are columns number 5 and 6), have different ranges:

unix% dmlist wav_src.fits  header,clean,raw | grep 'TLM..[56]'
TLMIN5       =    2028.50000000     /                     
TLMAX5       =    5970.50000000     /                     
TLMIN6       =    1800.50000000     /                     
TLMAX6       =    5630.50000000     /                     

After running dmtcalc, the values are different. The values for the Y column have been changed to be equal to the range of the X column.

unix% dmtcalc wav_src.fits new_col expression="newcol=#trand"

unix% dmlist new_col header,clean,raw | grep 'TLM..[56]'
TLMIN5       =    2028.50000000     /                     
TLMAX5       =    5970.50000000     /                     
TLMIN6       =    2028.50000000     /                     
TLMAX6       =    5970.50000000     /                   

These column ranges are used when the file is filtered, which can produce incorrect results.