Last modified: December 2019

URL: https://cxc.cfa.harvard.edu/ciao/ahelp/crates.html
AHELP for CIAO 4.12

crates

Context: crates

Synopsis

Reading, writing, and changing ASCII and FITS files in Python

Description

The Crates module allows users to read and write ASCII and FITS files in Python, with support for the same features as provided by the CIAO Data Model.

>>> cr = read_file('tbl.dat[energy=500:7000][cols x,y]')
>>> x = get_colvals(cr, 'x')
>>> y = get_colvals(cr, 'y')

reads in the x and y columns from the file 'tbl.dat', after applying a filter on the energy column of 500 to 7000, and then returns the column values in the x and y variables as NumPy arrays.

Functions and methods

The Crates module provides a number of routines for common tasks, like get_colvals. For more complex, uncommon routines, there are also a large number of methods provided by the objects themselves. Try the Python dir and help commands for more information: e.g.

>>> cr = read_file('evt.fits')
>>> x = cr.get_column('x')
>>> dir(x)
...
>>> help(x)

Examples

Example 1

>>> cr = read_file("evt2.fits")
>>> print(get_keyval(cr, "OBJECT"))
ELAIS-S1 A
>>> print(get_keyval(cr, "EXPOSURE") / 1000)
31.7432921583

Here we get the values of the OBJECT and EXPOSURE keywords from the header of evt2.fits. The return value from get_keyval() uses the appropriate Python type, so OBJECT is a string and the exposure time is a number.

Example 2

>>> from pycrates import *
>>> cr = read_file("evt2.fits[ccd_id=7,energy=300:7000]")
>>> e = copy_colvals(cr, "energy")
>>> emin = e.min() / 1000
>>> emax = e.max() / 1000
>>> emean = e.mean() / 1000
>>> print("Mean E = {0} min= {1} max= {2}".format(emean, emin, emax))
Mean E = 3.10554851929 min= 0.300000335693 max= 6.99996289063

Here we read in the energy column from an event file and calculate the minimum, maximum and mean energy in keV.

Example 3

>>> import numpy as np
>>> from pycrates import *
>>> cr = read_file("0.5-7.0_bin2.img")
>>> img = copy_piximgvals(cr)
>>> logimg = np.log10(img)
RuntineWarning: divide by zero encountered in log10
>>> from matplotlib import pyplot as plt
>>> plt.imshow(logimg, origin='lower')
>>> expmap = read_file("2.3_bin2.expmap")
>>> plt.imshow(expmap.get_image().values, alpha=0.2,
origin='lower',cmap="gray",vmin=1,vmax=10)

Here we display the logarithm of the pixel values using the matplotlib imshow command and then overlay the exposure map for the observation, using transparency to ensure that the image can still be seen.

The divide-by-zero warning

The warning about divide by zero comes from NumPy and can be turned off using the seterr routine; for example

>>> olderr = np.seterr(divide='ignore')

Example 4

>>> x = np.arange(1, 10)
>>> y = np.sin(x) * np.cos(x)
>>> z = x**2 + y**2
>>> xcol = CrateData()
>>> xcol.name = "x"
>>> xcol.values = x
>>> ycol = CrateData()
>>> ycol.name = "y"
>>> ycol.values = y
>>> zcol = CrateData()
>>> zcol.name = "zz"
>>> zcol.values = z
>>> zcol.unit = 'cm**2 s'
>>> cr = TABLECrate()
>>> add_col(cr, xcol)
>>> add_col(cr, ycol)
>>> add_col(cr, zcol)
>>> write_file(cr, "out.fits")
>>> write_file(cr, "out.dat[opt kernel=text/simple]")
>>> quit()
unix% cat out.dat
#TEXT/SIMPLE
# x y zz
1 4.5464871341284e-01 1.206705452608
2 -3.7840124765396e-01 4.143187504226
3 -1.3970774909946e-01 9.019518255158
4 4.9467912331169e-01 16.24470743504
5 -2.7201055544468e-01 25.07398974227
6 -2.6828645900022e-01 36.07197762408
7 4.9530367784744e-01 49.24532573329
8 -1.4395165833253e-01 64.02072207994
9 -3.7549362338584e-01 81.14099546120

Here we create two files containing the contents of the x, y, and z arrays in columns called "x", "y", and "zz" respectively. The file out.fits is a FITS binary table whereas out.dat is an ASCII file (see ahelp dmascii for more information on ASCII support in the CIAO Data model).

The make_table_crate routine from the crates_contrib.utils module can be used to simplify the above.

Example 5

>>> ivals = np.arange(12).reshape(3, 4)
>>> cd = CrateData()
>>> cd.values = ivals
>>> cr = IMAGECrate()
>>> add_piximg(cr, cd)
>>> cr.name = "SIMPLE"
>>> write_file(cr, "img.fits")

Here we create an image, with dimensions of 4 (x) by 3 (y), add it to an IMAGECrate using add_piximg (note that add_image is a ChIPS call that displays an image), and write it out to the file img.fits. After these commands we have:

>>> print(cr)
   Crate Type:        <IMAGECrate>
   Crate Name:        SIMPLE
>>> quit()

unix% dmlist img.fits blocks
 
--------------------------------------------------------------------------------
Dataset: img.fits
--------------------------------------------------------------------------------
 
     Block Name                          Type         Dimensions
--------------------------------------------------------------------------------
Block    1: SIMPLE                         Image      Int4(4x3)
unix% dmstat img.fits centroid-
SIMPLE
    min:	0 	      @:	( 1 1 )
    max:	11 	      @:	( 4 3 )
   mean:	5.5 
  sigma:	3.4520525295 
    sum:	66 
   good:	12 
   null:	0 

The make_image_crate routine from the crates_contrib.utils module can be used to simplify the above.

Example 6

unix% cat qconv.py
#!/usr/bin/env python
import sys
import pycrates as pyc
cr = pyc.read_file(sys.argv[1])
sky = cr.get_transform("SKY")
x = float(sys.argv[2])
y = float(sys.argv[3])
lcoord = [[x, y]]
scoord = sky.apply(lcoord)
print("Logical {0} -> SKY {1}".format(lcoord[0], scoord[0]))

This simple Python script (which has no sanity checks such as ensuring the number of command-line arguments is correct) will convert the logical coordinates given on the command line to SKY coordinates using the transform taken from the first argument. An example run (which assumes the executable bit is set) is given below:

unix% ./qconv.py img.fits 1 1
Logical [1.0, 1.0] -> SKY [ 2009.5 3149.5]

Loading Crates

The Crates module is automatically imported into Sherpa sessions, otherwise use one of the following:

from pycrates import *

or

import pycrates

Documentation

The Crates and transform library ahelp documentation can be listed by saying ahelp -c crates and ahelp -c transform

In particular: ahelp read_file, ahelp crate, ahelp cratedataset, ahelp cratedata, and ahelp cratekey.

There is also Python documentation available, on the module itself - e.g.

>>> import pycrates
>>> help(pycrates)

or for specific functions and methods

>>> help(read_file)
>>> help(cr.get_colnames)

Objects and metadata

The Crates library uses a set of Python objects to represent data. These objects provide access to the data and metadata - such as units, description, and other fields - stored in the files.

Object Description
TABLECrate, IMAGECrate Represents a block of a file containing a table - a set of columns - or a single n-Dimensional image.
CrateDataset, PHACrateDataset, RMFCrateDataset A set of blocks stored in a single file.
CrateData This is used to represent column and image data.
CrateKey A keyword, containing the name, value, and optional unit and description fields.
CrateSubspaceData Represent the filtering applied to a column or image.

Creating objects

Objects can be created using the normal Python idiom, by calling the constructor, or - in some cases - can be created by routines such as read_file and get_key.

The Crates library also uses objects from the transforms and region modules.

Subspace

The subspace information stored in a file can be accessed with the get_subspace_data method of a crate. There can be multiple blocks of subspace data - normally one per ACIS chip but combining filters with OR rather than AND will also lead to this - which means that you need to provide an integer and column name.

crate.get_subspace_data(cptnum, item)

Here crate refers to the return value of read_file, the cptnum argument is a positive integer, indicating the component number, and item is a string indicating the column or axis name. Note that in CIAO 4.12 there is no way to determine how many blocks there are in a file or what columns are stored in the subspace block.

>>> cr = read_file('a2142_smoothed.fits')
>>> s1 = cr.get_subspace_data(1, 'sky')
>>> print(s1)
   Name:            sky: x, y
   Unit:            
   Range Minimum:   []
   Range Maximum:   []
   Region String:   Box(3840,3670,1460,1460)

>>> sx = cr.get_subspace_data(1, 'x')
>>> sy = cr.get_subspace_data(1, 'y')
>>> print(sx)
   Name:            x
   Unit:            
   Range Minimum:   [ 3110.]
   Range Maximum:   [ 4570.]
   Region String:   DEFAULT

>>> print(sy)
   Name:            y
   Unit:            
   Range Minimum:   [ 2940.]
   Range Maximum:   [ 4400.]
   Region String:   DEFAULT

>>> import region
>>> region.regPrintRegion(s1.region)
1	Box(3840,3670,1460,1460) (Flag_Coord: Unknown) (Flag_Radius: Unknown)
>>> s2 = cr.get_subspace_data(1, 'time')
>>> print(s2)
   Name:            time
   Table Name:      GTI3
   Unit:            
   Range Minimum:   [  2.29755146e+08   2.29758594e+08   2.29795394e+08]
   Range Maximum:   [  2.29758591e+08   2.29795391e+08   2.29800307e+08]
   Region String:   DEFAULT

>>> s3 = cr.get_subspace_data(1, 'ccd_id')
>>> print(s3)
   Name:            ccd_id
   Unit:            
   Range Minimum:   [3]
   Range Maximum:   [3]
   Region String:   DEFAULT

The return value - a CrateSubspaceData object - has the following fields:

Field Description
name The column name.
unit The units of the column, or ''.
range_min The minimum value of the data; this can contain multiple values when sets of ranges were used to filter the column.
range_max The maximum value of the data; this can contain multiple values when sets of ranges were used to filter the column.
region If a spatial filter was applied then this is a region object representing the filter, otherwise it is None.

Two subspace objects can be compared using the Python equality check; e.g.

>>> cr1.get_subspace(1, "time") == cr1.get_subspace(2, "time")

There is no way in CIAO 4.12 to add to or delete from the subspace information in a Crate.

History and Comments

The COMMENT and HISTORY records can be retrieved from a Crate with the get_comment_records and get_history_records methods respectively. As an example:

>>> cr = pycrates.read_file('evt.fits')
>>> hist = cr.get_history_records()
>>> comm = cr.get_comment_records()

The get_all_records method returns both COMMENT and HISTORY records. New records can be added using the add_record method on the crate, and the pycrates add_record, add_comment, and add_history functions, as shown in the following fanciful example:

>>> cr.add_record('COMMENT', 'Yeah yeah how groovy')
>>> pycrates.add_comment(cr, 'Yes, very groovy')

History entries can be created using the history module:

>>> import history
>>> pnames = ["infile", "outfile", "grooviness"]
>>> pvals = ["okay.fits", "better.fits", 3]
>>> hist = history.HistoryRecord(tool='groovy', param_names=pnames,
param_vals=pvals)
>>> pycrates.add_history(cr, hist)

Changes in CIAO 4.11

Memory leaks

A number of memory leaks have been fixed in the Crates modules, and error conditions are now better handled in a number of low-level routines.

History and Comments

Crates now supports HISTORY and COMMENT keywords: that is, they are accessible from the crate using methods such as get_comment_records and get_history_records, and will be written out when the file is saved. Note that the order of these records (with respect to the keywords) is not guaranteed to be maintained.

The history module has been added to help creating history records.

Please use the Python help function for these methods, as there are no ahelp files for them.

Changes in CIAO 4.8

Adding columns

The add_column method of a table crate now works correctly when the column number is set to 0 (previously the new column was added to the end of the table, now it is the first column). The add_column method and add_col call now create an empty subspace entry for vector columns, accessible with the get_subspace_data method of the crate.

Support for variable-length arrays

Support for variable-length arrays has been improved and the CrateData object now supports the is_varlen and get_fixed_length_array methods.

Tri-state logical columns

The FITS convention for boolean columns supports a "null" value, which means that it can represent three states: true, false, and unknown. Any such null, or unknown, value will be converted to False when the data is read in to a Crate (e.g. when read by read_file), and so the information that this cell value was a null will be lost.


Bugs

See the bug pages on the CIAO website for an up-to-date listing of known bugs.

Refer to the CIAO bug pages for an up-to-date listing of known issues.

See Also

contrib
add_colvals, make_image_crate, make_table_crate, scale_image_crate, smooth_image_crate, write_arrays, write_columns
crates
add_col, add_key, add_piximg, col_exists, copy_colvals, copy_piximgvals, cratedata, cratekey, create_vector_column, create_virtual_column, delete_col, delete_key, delete_piximg, get_axis_transform, get_col, get_col_names, get_colvals, get_crate_item_type, get_crate_type, get_key, get_key_names, get_keyval, get_number_cols, get_number_rows, get_piximg, get_piximg_shape, get_piximgvals, get_transform, get_transform_matrix, is_pha, is_rmf, is_virtual, key_exists, print_crate_names, read_file, read_pha, read_rmf, set_colvals, set_key, set_keyval, set_piximgvals, write_file, write_pha, write_rmf