Bugs: dmtcalc

For example, The expression "if(a)then(b)" does not work as expected in dmtcalc if two different columns are used in the comparison.

unix% dmtcalc in.fits out.fits "expr=if(pi>500)then(pi=-1)" 

works as expected ("pi" used for both pieces of the conditional), but

unix% dmtcalc in.fits out.fits "expr=if(energy>5000)then(pi=-1)"

does not work. This is because the pi value in the expression is not always initialized to the input. Instead you must explicitly initialize it

unix% dmtcalc in.fits out.fits "expr=pi=pi;if(energy>5000)then(pi=-1)"

Using the log or log10 operator on integer columns returns an integer value. The value is truncated.

Workaround:

If unsure of the datatype, the value can be explicitly cast to a real-valued number

newcol=log((float)oldcol)
newcol=log(oldcol*1.0)
  

When using status bits in an expression such "status==X1F,X3T,X18F", only the last condition (here, "X18F") is checked.

Workaround:

Set the logic explicitly:

• logical "AND": status=((bits==X1T)&&(bits==X2T))
• logical "OR": status=((bits==X1T)||(bits==X2T))

For example, the expression

x+814*24*3600-260086780.04

is treated as

x+(814*24*(3600-260086780.04))

instead of the expected

x+(814*24*3600)-260086780.04

Workaround:

Include the parentheses, as shown in the last code snippet, to force the correct order of operations.

For example, this file will work:

unix% cat ok.expr
(r=(double){3.5,4.}) 

unix% dmlist output_ok.fits'[cols r]' data
---------------------------------------------
Data for Table Block HISTOGRAM
---------------------------------------------

ROW R[2]

     1 [ 3.50 4.0]
     2 [ 3.50 4.0]
     3 [ 3.50 4.0]
(etc.)

while this does not:

unix% cat not_ok.expr
(r=(double){3.5,4}) 

unix% dmlist output_not_ok.fits'[cols r]' data
-------------------------------------------------
Data for Table Block HISTOGRAM
-------------------------------------------------

ROW R[2]

     1 [ 3.50 0]
     2 [ 3.50 0]
     3 [ 3.50 0] 
(etc.)

For example, when using an event list, one cannot say

unix%  dmtcalc evt1.fits foo.fits exp='dra=(ra-ra_nom)'

because the 'ra' column is not a real column in the file; see the virtual columns dictionary entry for an explanation.

Workaround:

Use Data Model column renaming:

unix%  dmtcalc evt1.fits"[cols foo=ra,*]" foo.fits expr='dra=(foo-ra_nom)'

If you calculate npha as npha=pha; which should have units of PHA [adu]; but it will be unitless. Even more confusing is pha=pha will be unitless since pha in the output file is treated a new column.

There are various problems doing things like:

pha[9]=1;
foo=pha[9];

where you try to access the individual elements in a vector and/or array column.

There are issues when changing values of existing columns: what info is kept, what is thrown away (e.g. data types, null, units, descriptions, order in file). In particular there are various problems with arrays vs. vector vs. vector-array columns.

For example, you cannot do above=(phas>20) since the PHAS array contains 9 (or 25) values. Currently only the first value in the array is checked.

For example:

expression="dist=((4125-x)^2)-((4025-y)^2)"

Workaround:

Refer to the component by vector notation instead:

expression="dist=((4125-sky[0])^2)-((4025-sky[1])^2)"

dmtcalc tries to do this subsitution internally in the first example, but fails.

Vector columns such as SKY(X,Y), POS(X,Y), CHIP(CHIPX,CHIPY), RD(TG_R,TG_D), cannot be modified using dmtcalc.

Internally, dmtcalc treats vector columns as arrays. So SKY(X,Y) becomes SKY[0] and SKY[1] which is what is written on output if the values are set. For example

unix% dmtcalc wav_srclist.fits modified_srclist expression="pos=pos"
unix% dmlist wav_srclist.fits | grep POS
   5   POS(X,Y)             pixel        Real8          2288.50:     4900.50 Physical coordinates
5:    EQSRC(RA_SRC ) = (+230.1725)[deg] +TAN[(-0.000136667)* (POS(X)-(+4096.50))]
ciao:/data/da/Docs/ciaoweb/ciao47/bugs% 

unix% dmlist modified_srclist | grep -i POS
 26   POS[2]                            Real8(2)       -Inf:+Inf            User defined column

shows that just copying the values in the POS vector column creates a POS array.

Workaround:

Users can modify vector columns using the Python Crates package

from pycrates import read_file

tab = read_file("wav_srclist.fits")
pos = tab.get_column("pos").values
pos[0][0] = new_X_value
pos[0][1] = new_Y_value
tab.write("modified_srclist")

When the input file contains a vector column, the legal range of values stored in the raw FITS TLMIN/TLMAX keywords associated with each raw column are not preserved. The MIN/MAX for each raw column in the vector are set equal to the 1st column.

So in this example, a wavdetect source list with the vector column POS(X,Y) (which are columns number 5 and 6), have different ranges:

unix% dmlist wav_src.fits  header,clean,raw | grep 'TLM..[56]'
TLMIN5       =    2028.50000000     /                     
TLMAX5       =    5970.50000000     /                     
TLMIN6       =    1800.50000000     /                     
TLMAX6       =    5630.50000000     /                     

After running dmtcalc, the values are different. The values for the Y column have been changed to be equal to the range of the X column.

unix% dmtcalc wav_src.fits new_col expression="newcol=#trand"

unix% dmlist new_col header,clean,raw | grep 'TLM..[56]'
TLMIN5       =    2028.50000000     /                     
TLMAX5       =    5970.50000000     /                     
TLMIN6       =    2028.50000000     /                     
TLMAX6       =    5970.50000000     /                   

These column ranges are used when the file is filtered, which can produce incorrect results.