<Return to CalDB Release Notes>
CalDB PUBLIC Release Notes
CALDB 4.0.0 is not a public release CalDB version

I. Introducing Chandra CALDB4

Chandra CalDB4 is a new calibration database structure and interface, that is being introduced with the release
of CIAO 4.1 on December 15, 2008.

CalDB4 conforms to the general FITS paradigm established in the pre-existing HEASARC CalDB system,
and can be maintained in the same directory space. However, CALDB4 is not compatible with Chandra CalDB 3.5.0
or any earlier versions of the Chandra CalDB, and may not be stored in the same directory tree as those earlier versions.

The specific version of CalDB4 that will be released to the public is "4.1.0,"
to coincide with CIAO "4.1," and hopefully prevent confusion that would result if users were to believe that
CALDB "4.0.0" is compatible with CIAO "4.0" (released December 2007), which it definitely is not.
The new CalDB will only work with CIAO 4.1 or later versions.

User Documentation:
"Introduction to CalDB4"

II. Summary of Changes

The CALDB4 software and directory/index structure has been defined to address these additional general requirements:
1) To provide an index and directory structure that is applicable to an arbitrary mission configuration, and controllable by CalDB managers.
2) To provide the necessary automation of calibration parameter identification and evaluation.
3) To eliminate hard-coding of parameter names and/or values wherever possible in CIAO 4.1, and later in the DS.
4) To accommodate mission-independent tooling in the long term.

CalDB 4.1.0 (so designated to correspond to CIAO 4.1 to prevent confusion) implements a new, flexible index system,
with keyword parameter/index column control set by the CalDB manager. In addition to the flexible indexing, the
software interface employs a hand-shaking implementation with the CalDB to identify necessary parameters for the
requested calibration data, and to supply the necessary values as available from the input data file headers, or as supplied
by the user when necessary. All that is required of the tool (or the user) to begin selection of calibration data is to know
the mission (CHANDRA) and the name of the particular calibration data needed (such as DET_GAIN or QE). The context
of the search determines the other required parameter values to get the right data.

The developer or the user do not need to know which parameters are required to select the data; the interface will request
the appropriate information automatically, usually from an events, pha, pha2, or pi data file.

A snapshot copy of the CALDB4 requirements specifications is available here.

This document currently specifies the new flexible index system,
CalDB configuration file "caldb.config," the CalDB query interface and algorithm, and the index builder software.

CIAO 4.1 incorporates the new caldb4 software interface and associated library (See ECR-2008-026, concurrent),
and supports the index builder tool "calindex".
CIAO 4.1 and CalDB 4.1.0 are mutually required for full functionality of both.

CALDB4 and the specific implementation of CALDB 4.1.0 have been built to address Chandra CalDB requirements
and to address the above general specifications, with the ongoing tacit approval of HEASARC CalDB personel.
HEASARC has been involved in the early specification and investigation of new requirements. Our general progress
has been included in annual reviews of the HEASARC CXC support contract. The CALDB4
requirements and implementations are to serve as a possible new HEASARC CalDB standard, with incumbent updates
to the OGIP standard taken as needed. We have built CALDB4 for Chandra, but upon sufficient internal review and
approval, intend to deliver the software and tools for CALDB4 to HEASARC pending verification of all specified
requirements and possibly some interfacing additions to be specified later.

III. Technical Details

1. CalDB Configuration "caldb.config"

The CalDB configuration file locates the individual CalDB trees and index files for specific compliant
missions. The file is located at $CALDB/software/tools, and must contain the specs for all missions included
in the $CALDB directory tree for all the data to be accessible through the software interface. "caldb.config" is an ASCII
In the old specification CHANDRA's caldb configuration appears as:
CHANDRA ACIS CALDB data/chandra/acis caldb.indx CALDB data/chandra/acis
CHANDRA EPHIN CALDB data/chandra/ephin caldb.indx CALDB data/chandra/ephin
CHANDRA HRC CALDB data/chandra/hrc caldb.indx CALDB data/chandra/hrc
CHANDRA PCAD CALDB data/chandra/pcad caldb.indx CALDB data/chandra/pcad
CHANDRA SIM CALDB data/chandra/sim caldb.indx CALDB data/chandra/sim
CHANDRA TEL CALDB data/chandra/tel caldb.indx CALDB data/chandra/tel
The above information specifies the MISSION: CHANDRA and the INSTRUMENTs available to Chandra
under the old paradigm, where all calibration data are separated by the INSTRUME (keyword) to which they apply.
ACIS, EPHIN, HRC, PCAD, and SIM all correspond to legitimate instruments. TEL had to be added to
subsume the calibration data for PIXLIB, HRMA, and the GRATINGs.
In the new caldb.config for Chandra CalDB 4.1.0:
CHANDRA ACIS CALDB data/chandra/acis caldb.indx CALDB data/chandra/acis CALDB data/chandra/acis key.config
CHANDRA EPHIN CALDB data/chandra/ephin caldb.indx CALDB data/chandra/ephin CALDB data/chandra/ephin key.config
CHANDRA HRC CALDB data/chandra/hrc caldb.indx CALDB data/chandra/hrc CALDB data/chandra/hrc key.config
CHANDRA PCAD CALDB data/chandra/pcad caldb.indx CALDB data/chandra/pcad CALDB data/chandra/pcad key.config
CHANDRA SIM CALDB data/chandra/sim caldb.indx CALDB data/chandra/sim CALDB data/chandra/sim key.config
CHANDRA DEFAULT CALDB data/chandra/default caldb.indx CALDB data/chandra/default CALDB data/chandra/default key.config
CHANDRA DEFAULT CALDB data/chandra/pimms caldb.indx CALDB data/chandra/pimms CALDB data/chandra/pimms key.config

Note that each row contains three more elements, corresponding to the location and name of the "key.config" file, which gives the keyword
configuration of the new flexible INDEX files that are built by "calindex".

Also, the TEL branch, which was not entirely appropriate for storing HRMA, PIXLIB, and GRATING files, has been eliminated. Instead,
there is the designation of DEFAULT to replace (and break) the now-outdated INSTRUME separation paradigm for CalDB files. When
calibration data for mission elements do not correspond to any particular INSTRUME value, these elements may be indexed and stored under
one or more directories designated as DEFAULT in caldb.config. So, it is no longer necessary to address HRMA or GRATING files
in the code, by hardcoding "TEL" as the instrument associated with HRMA or HETG, for example.

2. Directory structure: (Full table listing is included here only for completeness.)

All branches have relatively predictable structures, with one exception: the "pimms/" branch, which
is a special case of files used only to set up the PIMMS effective area files during the software build. Under
pimms/, the files are divided between those for ACIS configurations, and those for HRC.



"caldb.indx" file for ACIS

"key.config" file for ACIS caldb.indx

index/ (Stores versioned index files linked to "caldb.indx"




















"caldb.indx" file for DEFAULT

"key.config" file for DEFAULT caldb.indx














"caldb.indx" file for EPHIN

"key.config" file for EPHIN caldb.indx




"caldb.indx" file for HRC

"key.config" file for HRC caldb.indx


















"caldb.indx" file for PCAD

"key.config" file for PCAD caldb.indx











"caldb.indx" file for PIMMS

"key.config" file for PIMMS caldb.indx





"caldb.indx" file for SIM

"key.config" file for SIM caldb.indx




3. Mandatory Index Columns:

The following columns are required in all CalDB index files. In some cases, such as the CAL_DIR and
the CAL_FILE, the requirement is obvious, since the location, filename, extension number, and calibration
code name are primary to a CalDB structure. Some designations are required for backward compatibility
with previously existing HEASARC standards, which the requirements document above specifies and
acknowledges must be maintained. However, despite being mandatory, they do not impinge on the
flexibility in building a CalDB for an arbitrary mission.

The mandatory columns and associated keywords are given in the Requirements Document, Table 1, pp. 4 and 5.
I list the columns here for completeness, with definitions:

Index file Column Name
Equivalent Header
Keyword, if any

”ONLINE” for calibration files present in the
CALDB directory tree. ”OFFLINE” or
”<device name>” for calibration files not
present in the CALDB tree. See below for
information on constructing the fully qualified
path to the calibration file from CAL_DEV,

The path to the directory containing the calibration
file specified by CAL_FILE.

The name of the calibration file.
CCNM* (CCNM0001)
The name of the type of calibration data. Used as
the basis for identifying the type of data being
CBD* (varies CBD10001
to CBD90001)
The calibration boundary conditions. These are
additional query constraints that are either
sufficiently infrequent that including a query
column in the index file is inappropriate, or that
are not readily represented as query columns in the
index file (examples would be numeric ranges).

The FITS extension number that contains the
calibration data. May be zero if the calibration
data are stored in the primary HDU (for example,
an image). <This is 1 less than the data model
block number.>
The “quality” of the calibration, defined as
follows: 0 = “Good. No known issues.”; 1 =
“Some calibration issues known. May be default
for processing or analysis.”; 2 = “Calibration
issues identified and replacement needed or
anticipated. May be used, but will be superseded
in the near future.”; 3 = “Former operational
mode, good enough for processing or analysis of
archival data taken in that mode. Significant
issues known; however recalibration of this mode
is unlikely.”; 4 = “Strongly deprecated or
superseded. Issue a warning to this effect if
used. Reprocessing needed if these data were used
in earlier processing; same for earlier analysis.
Kept to enable selection of these data as related
files.”; 5 = “ Bad. Never to be used again. To be
retired. A better option is available in the CalDB,
including related files data.”; 9
= “Dataset no
longer exists.”.

The UTC date when this entry was added to the
index file, in yyyy-mm-dd format.

CALDB4 has adopted the above from the previous HEASARC CalDB specification, and has added none to this list.
The calindex software has added the designation for the CAL_QUAL Equivalent header keyword as "CAL_QUAL",
which it now reads and puts the value into the index file.

4. Predefined Index File Optional Columns
Certain columns have been established by the accumulated CalDB history to contain certain types of data and are
interpreted by software (or not) in particular ways. CALDB4 has defined these as "optional", but having a particular
designation and type. In so doing, we have added a few of our own new columns in CalDB4, some of which are in
analogous to the pre-existing ones. These are in Table 2 of the requirements document, pp. 6 and 7.

The calibration file class, one of “PCF”, ”BCF”,
”CPF”, or “USR”. Not interpreted by software.
The type of the calibration file, one of ”DATA”,
”TASK”, or “FEF”. Not interpreted by software.
Short string description of the calibration data.
Not interpreted by software.
The beginning valid date of the calibration, in
yyyy-mm-dd format. This column is interpreted
by software.
The beginning UTC valid time of the calibration,
in hh:mm:ss format. This column is interpreted
by software.

The MJD corresponding to CAL_VSD/CAL_VST.
This column is interpreted by software.
The ending UTC valid date of the calibration, in
yyyy-mm-dd format. This column is interpreted
by software.
The ending UTC valid time of the calibration, in
hh:mm:ss format. This column is interpreted by

The MJD corresponding to CAL_VED/CAL_VET.
This column is interpreted by software.
Numeric “fidelity” of calibration data. May be
used to select amongst different calibrations of the
same quality that satisfy the query requirements.
The ordering of the FIDELITY is defined such
that numerically greater values correspond to
higher fidelity calibration data. This column is
interpreted by software.

NOTES:   CAL_CLAS, CAL_DTYP, AND CAL_DESC are all mandatory columns and keywords in the existing
HEASARC standard. They are not mandatory in CALDB4.
CAL_VED, CAL_VET and END_TIME are analogous to the CAL_VSD, CAL_VST, and REF_TIME counterparts.
These are not supported in the existing HEASARC standard, but are added in CALDB4.

5. Keyword Configurations and "key.config":

Each branch of CalDB 4.1.0 has a particular specification for the index file in "key.config" as listed in
the table above. "key.config," like "caldb.config," is always an ASCII file. In CalDB 4.1.0, the key.config
files reside just along side of the "caldb.indx" links. A key.config file looks like the following:
# Time-stamp: <2008-11-10 10:38:02 caldbmgr>
# CHANDRA key.config table
# for INSTRUME = <Manager specifies this>
#colName   dataType   format   hdrKey   queryCol   nullVal
# ---------------------------------------------------------
TELESCOP   string      a10    TELESCOP    yes        ""
INSTRUME  string      a10    INSTRUME    yes      "NONE"
CAL_DESC    string      a70    CDES*       no       "NONE"
CAL_VSD      string      a10    CVSD*       no         ""
CAL_VST      string      a8       CVST*       no         ""
REF_TIME    double      d      ""                 no         0.0
CAL_VED     string      a10    CVED*       no         ""
CAL_VET     string      a8      CVET*        no         ""
END_TIME   double      d      ""                 no         0.0
FIDELITY     double      d       FDLT*       no         0.0

The pound sign designates a comment, ignored by calindex.

The columns are the index file column name, data type, format, the corresponding header keyword in the CalDB files,
the specification of whether the column is a query column,  and null value. The column names are added to the index by
calindex, in addition to the mandatory columns described above. When a column is a "query" column, the value corresponding
to the "header keyword" designated thereto, is searched for in the input file headers whenever a CalDB call is made to
the index in question. This then is part of the built-in hand-shaking between the interface and the CalDB index structure,
to determine what is important in the selection of particular CalDB data.

As indicated in the caldb.config specification above for CHANDRA, there are seven distinct branches now in CalDB 4.1.0,
with corresponding caldb index and key.config files. The specific key.config and resulting caldb.indx files will appear at
the following locations, once CALDB4 has been installed locally, and $CALDB has been set:


6. CHANGES to CalDB files in CALDB 4.1.0:
The data in CALDB 4.1.0 are largely the same as the data in CalDB version 3.5.1; in fact only one file has been added
and it contains no new data, but simply accommodates a special RMF building case (See ECR_2008_028). Pre-existing
CalDB filenames are all preserved, so that the transition in analysis may be seamless.

The only changes made to CalDB files from 3.5.1 to 4.1.0 are to the various HDU headers. Such changes include:

1) Eliminate unnecessary boundary conditions from the CalDB index files (such as ENERGY ranges, CCD_ID
specifications with they are not needed, or redundant specifications
for GRATING or GRATTYPE, which are now handled as index columns in CalDB 4.1.0.) These unnecessary
boundary conditions are moved from the CBD* keyword series to a new series CBO* for Optional boundary
conditions. They remain in the file headers, but are not interpreted by calindex.

2) Add calibration validity end date and time keywords CVED0001 and CVET0001
3) Add a CAL_QUAL specification if it is to be non-zero.
4) Add a FIDELITY specification if needed (FDLT0001).

"calindex" uses all of these keyword values in building the new index files.

<Return to CalDB Release Notes>