Bugs: CRATES
Table of Contents
Caveats
Bugs
Crates does not support np.uint32 (unsigned 32bit integers) arrays as images.
Trying to create a table with 0 rows with an vector column will crash when the crate is written.
Crates cannot read a table that has 0 rows and contains vector array column(s).
Crates will not copy the keywords in an empty/null FITS Primary extension
Region filters in the crates subspace are not writen when the file is saved.
Trying to write a file with a Byte data-type in the subspace fails.
Adding a column without a name creates a column called () in the output file
Numerical subspace ranges are not propagated correctly
(Mac OS X)
Caveats
Crates does not support subspaces on byte columns.
OGIP compliance
Cannot add rows to a column that is part of a virtual column.
Users can not add rows a column which is a component of a vector using Crates (the new values are set to 0). User can extend scalar and array columns. To work around this, users must add rows the vector column itself rather than the components.
Bugs
Crates is unable to write files that have REGION filters (subspace) stored as additional blocks in FITS files.
Specificially if the DSREF keywords reference REGION blocks.
Crates does not support np.uint32 (unsigned 32bit integers) arrays as images.
An error is raised when trying to write the crates to a file.
Trying to create a table with 0 rows with an vector column will crash when the crate is written.
>>> cr = TABLECrate () >>> col = CrateData () >>> col.name = 'POS' >>> col.vdim = 2 >>> cpt_list = ['X','Y'] >>> cpt_list = CrateDict() >>> for nn in ['X','Y']: ... cpt = CrateData() ... cpt.name = nn ... cpt.parent = col ... cpt.vdim = 1 ... cpt_list[nn] = cpt ... >>> col._set_cptslist(cpt_list) >>> cr.add_column (col) >>> col.values = np.empty (0, dtype=np.float) >>> cr.write ('tmp.fits', clobber=True) Segmentation fault
Crates has problems trying to populate a string column.
>>> cd = CrateData () >>> cd.values = np.empty (1, dtype='|S1') >>> cd.values.fill ('b') >>> cd.values array(['\xf8'], dtype='|S1')
The array values should be filled a single letter 'b'
Crates cannot read a table that has 0 rows and contains vector array column(s).
>>> tab = read_file("acis_fov.fits[ccd_id=10]") >>> print(tab.get_colnames()) ValueError: all the input arrays must have same number of dimensions
Crates generates a misleading "File exists but is not readble" error when asked to open a non-existent block within the file.
>>> tab = read_file("myfile[does_not_exist"]) IOError: File exists but is not readable.
Whereas the following works:
>>> tab = read_file("myfile")
Problem reading images containing unnamed WCS transforms
Crates has problems reading images that contain WCS transforms, but have chosen to omit the transform names. For example in a standard Chandra observtaion the SKY X,Y image axes have a WCS transform to RA and DEC.
Some older files may have images where the actual names of the transformed axes was omitted. This has been seen in some ROSAT weight maps (WMAPs) attached to older spectrum files.
>>> ww = read_file("orig.pi[wmap]") >>> ii = ww.get_image() RuntimeError:
Problem with single byte bit columns
Crates has problems when trying to write out a bit column, when the column has less than 8 bits (so fill only 1 byte).
Subspace information corrupted when images are deleted.
The subspace information for an image is corrupted when the image values are replaced (image deleted and then re-added) as one might do to change the data-type of the image.
>>> fimgcrate = read_file (infile) >>> fimgcrate_image = fimgcrate.get_image() >>> values = fimgcrate_image.values >>> fimgcrate.delete_image() >>> values = values.astype("float32") >>> fimgcrate_image.values = values >>> fimgcrate.add_image(fimgcrate_image) >>> fimgcrate.write(outfile) /soft/ciao/lib/python2.7/site-packages/pycrates/io/dm_backend.py:1361: UserWarning: Unable to delete columns or images in update mode. warnings.warn("Unable to delete columns or images in update mode.")
Crates will convert all transforms into their binned form.
Transforms are assumed to be applied to image pixels, so when they are created they are adjusted to match what would be an image pixel boundary, even if the transform is created on a table column.
So, for example trying to create a transform that converts degrees F to C:
tempF = +32 + 1.8 * tempC
A linear transform with scale=1.8 and offset=32, is written out as
tempF = +32.90 [degree F] +1.80 * (tempC -0.50)
While mathematically correct, the extra half "pixel" offset is unnecessary when dealing with transforms applied to table columns.
Crates will not copy the keywords in an empty/null FITS Primary extension
The keywords in the FITS primary extension are not written by the CrateDataset's write() method
>>> from pycrates import * >>> tab = read_file(infile) >>> tab.get_dataset().write(outfile)
If infile has a NULL Primary extesnion (only keywords, no data), the keywords are not written to the output file.
Writing NaN to keywords causes crash
Attempting to write a NaN value to a keyword results in a crash of the python interpreter
>>> tab = read_file("a.fits") >>> key = CrateKey() >>> key.name="foobar" >>> key.value = np.nan >>> tab.add_key( key ) >>> write_file( tab, "b.fits") RuntimeError: dmKeyWrite() could not write key. 'foobar' >>> quit() Exception IOError: IOError('FITS error 402 writing key foobar',) in <boundmethod CrateDataset.__del__ of Crate Dataset: ... *** glibc detected *** /export/ciao-4.5/ots/bin/python: double free or corruption (!prev): 0x0000000003026480 *** ... (core dumped) ipython --profile chips -i -c "\"${cmd}\""
1D Image transforms
pycrates can only access the 1st transform associated with an image. If a 2D image has only 1 transform (eg sky=(x,y)) then there is no problem; however, if the image has two seprate 1D transforms (eg time and pi), pycrates only provides access to the 1st (x) axis.
>>> from pycrates import * >>> cr = read_file('test.dat[bin kt=1.15:1.55:0.1,abund=0.8:1.2:0.2;factor]') >>> cr.get_axisnames() ['kt'] >>> cr.get_axis('abund') KeyError: 'abund not found in list of image axes.' >>> cr.get_transform('kt') <pytransform.LINEARTransform; proxy of <Swig Object of type 'LINEARTransform *' at 0x103692150> > >>> cr.get_transform('abund') KeyError: 'abund not found in list of image axes.'
Region filters in the crates subspace are not writen when the file is saved.
Byte datatype subspace columns
pycrates may crash if the file contains subspace columns with byte data-type. Examples include some HRC datasets with byte type SUB_MJF.
Workaround:
Workaround: for the HRC case, you can remove the subspace column with
% dmcopy hrc_evt[subspace -sub_mjf] hrc_evt_mod
Write access is not robustly checked.
Users need to be careful when trying to modify a file in place as the checks on whether or not a file is writeable are not robust. Simply specifying mode="rw" will not produce an error nor an exception if the file is not actually writeable.
unix% chmod 400 myfile.fits unix% python >>> from pycrates import * >>> rr = read_file("hrcf01801N006_evt2.fits", mode="rw") >>> # do something to modify file >>> rr.write()
The above sequence will not geneate an exception even though the file is not writeable.
Trying to write a file with a Byte data-type in the subspace fails.
HRC event files have a byte type (8 bit unsigned integer) SUB_MJF subspace column present. Trying to read in the file into a crate and writing it back out will fail.
unix% dmlist hrcf01801N006_evt2.fits subspace -------------------------------------------------------------------------------- Data subspace for block EVENTS: Components: 1 Descriptors: 33 -------------------------------------------------------------------------------- ... 31 ENDMNF Int4 0:127 32 SUB_MJF Byte 0:64 33 CLKTICKS Int4 0:1091567608 unix% python >>> from pycrates import * >>> rr = read_file("hrcf01801N006_evt2.fits") >>> write_file(rr, "/tmp/copy") Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/hlui.py", line 985, in write_file crate.write(outfile=filename, clobber=clobber) File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/tablecrate.py", line 294, in write backend.write( self, outfile=outfile ) File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 534, in write self.__write_block( crate, close_flag ) File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 578, in __write_block self.__write_subspace(block, crate) File "/export/ciao-4.5/lib/python2.7/site-packages/pycrates/io/dm_backend.py", line 626, in __write_subspace ssitem.unit, ssitem.range_min, ssitem.range_max ) TypeError: dmSubspaceColCreate() mins or maxes argument is not a supported type.
The only workaround is to remove the SUB_MJF subspace before reading it into a crate.
unix% dmcopy "hrcf01801N006_evt2.fits[subspace -SUB_MJF]" copy_evt.fits unix% python >>> from pycrates import * >>> rr = read_file("copy_evt.fits") >>> write_file(rr, "copy2")
Using numpy views of arrays can lead to data corruption.
numpy uses a special indexing scheme to access arrays that have been sliced (ie truncated at either end) or when for example transposing 2D arrays. The origianal data are preserved and the user only sees a view of the array.
The crates python to datamodel C interface does not handle views correctly. It may result in data being truncated or simply producing nonsensical results.
Users should make sure that any arrays sent to pycrates are in C_CONTIGUOUS order and ALGINED
>>> foo = np.arange(10).reshape(5,2) >>> foo2 = bob.T >>> foo.flags C_CONTIGUOUS : True F_CONTIGUOUS : False OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False >>> foo2.flags C_CONTIGUOUS : False F_CONTIGUOUS : True OWNDATA : False WRITEABLE : True ALIGNED : True UPDATEIFCOPY : False
Files with 1D linear transforms
pycrates does not support 1D, linear transforms. It tries to create a 2D transform which causes memory corruption and may lead to either a crash (Segmentation Violation) or could lead to incorrect results.
Users can check if their file contains a 1D transform by using dmlist to list the columns and look for any coordinates transforms that look like
unix% dmlist myfile.fits cols ... World Coord Transforms for Columns in Table Block EVENTS -------------------------------------------------------------------------------- ColNo Name 3: CPC(CPCX) = (+0)[mm] +(+0.0240)* (chip(chipx)-(+0.50)) (CPCY) (+0) (+0.0240) ( (chipy) (+0.50)) 4: MSC(PHI ) = (+0)[deg] +TAN-P[(+0.000136667)* (det(detx)-(+4096.50))] (THETA) (+0) (+0.000136667) ( (dety) (+4096.50)) 5: EQPOS(RA ) = (+270.1153)[deg] +TAN[(-0.000136667)* (sky(x)-(+4096.50))] (DEC) (-24.0418 ) (+0.000136667) ( (y) (+4096.50)) 6: pha_COORD = pha
Here, the pha column has a 1D linear transform attached to it and would cause problems with pycrates.
Another example would be
unix% dmlist dmextract29b.out cols ... -------------------------------------------------------------------------------- World Coord Transforms for Columns in Table Block HISTOGRAM -------------------------------------------------------------------------------- ColNo Name 4: CEL_R = +0 [arcsec] +0.4920 * (R -0) 9: CEL_AREA = +0 [arcsec**2] +0.2421 * (AREA -0) 17: CEL_BRI = +0 [count/arcsec**2] +4.1311 * (SUR_BRI -0) 18: CEL_BRI_ERR = +0 [count/arcsec**2] +4.1311 * (SUR_BRI_ERR -0)
All these columns are 1D linear transforms that would cause pycrates a problem.
Some example of files with linear transforms include
- Radial profiles output from dmextract
- ACIS Blank Sky background files with '2005' date
- Event files
- Aspect histogram files created by asphist
Currently there is no generic work around. If you do not need to use the column with the linear transform, you can just exclude it using a [cols -column_name] filter. If you need to use the column that has a transform attached, contact CXC Helpdesk and we will try to help with a custom solution.
Adding a column without a name creates a column called () in the output file
>>> cr = TABLECrate() >>> cd = CrateData() >>> cd.values = np.arange(20,31) >>> add_col(cr, cd) >>> cr.get_colnames() [''] >>> cr.write("/tmp/temp.fits") >>> !dmlist /tmp/temp.fits cols -------------------------------------------------------------------------------- Columns for Table Block HDU2 -------------------------------------------------------------------------------- ColNo Name Unit Type Range 1 () Int4 -
Numerical subspace ranges are not propagated correctly
(Mac OS X)
This file was read with CRATES and then written out again. The numerical subspace ranges have been replaced by "DEFAULT":
dmlist pha_copy.fits"[REGION]" subspace -------------------------------------------------------------------------------- Data subspace for block REGION: Components: 1 Descriptors: 11 -------------------------------------------------------------------------------- --- Component 1 --- 1 SPEC_NUM Int2 1:16384 2 ROWID String 3 SHAPE String 4 TG_R Real8 DEFAULT 5 TG_D Real8 DEFAULT 6 R Real8 DEFAULT 7 ROTANG Real8 DEFAULT 8 COMPONENT Int2 DEFAULT 9 INCLUDE Int2 DEFAULT 10 TG_SRCID Int2 DEFAULT 11 TG_M Int2 DEFAULT