Downloading CALDB 3.5.5

CALDB 3.5.5 is now only for use with CIAO 3.4!!!

CalDB 3.5.5 released 18 SEP 2009

Contents:

• Determining your current CalDB version
• Upgrade CALDB 3.5.4 or earlier version to 3.5.5
• Install the full CALDB 3.5.5
• Check the Index Configuration
• Verify that the CALDB is Working

Determining your current CalDB version

You will need to have started CIAO 3.4 and have $CALDB assigned in order to proceed. If you have not installed CIAO at all, we strongly recommend you proceed with installation of our latest version, available on the CIAO 4.1 download web page. If you are already using CIAO 4.1 or later version, DO NOT USE THIS WEB PAGE for CalDB upgrades. Proceed instead to the CALDB 4 download web page to determine if your CalDB is up to date.

The easiest way to determine which version of the Chandra CalDB you currently have installed, is to search the last line of the caldb_version.fits file under $CALDB/docs/chandra/caldb_version/. An appropriate CIAO command for this purpose is as follows:

unix% dmlist \
$CALDB/docs/chandra/caldb_version/caldb_version.fits"[cols CALDB_VER]" \
data,clean | tail -1

If the CalDB version you determine is version 4.1.0 or later, please DO NOT continue with this installation!. You may have loaded the wrong CIAO version and are definitely in the wrong CalDB area. CalDB 3.5.5 is only to be used for CIAO 3.4. If you mean to maintain your CIAO 3.4/CALDB 3.x.x installation, you must load CIAO 3.4, and move to the appropriate directory for CalDB 3.x.x.

Upgrade CalDB 3.5.4 or earlier version to 3.5.5

If your site has already installed CalDB 3.5.4 or earlier version, you may elect simply to install the 3.5.5 upgrade by using the "main" file in Table 1 above. Follow instructions 1-4 in this section to complete your upgrade. You will want to install the ACIS background tar file if your current CalDB version is 3.4.0 or earlier. If you have already installed the PSF library, or don't need it, you will not need to download or unpack the "psflib" tar files in Table 1.

1. Change directories to the existing CALDB location.

2. Download the CalDB 3.5.5 main gzipped tar file.

3. Download the ACIS background file tar ball acis_bkgrnd_3.5.4.tar.gz

4. Unzip the two tar balls using gunzip. We prefer you then un-tar both unzipped tar balls using gtar xvUf. (You must include the "U" option so that the command will appropriately reset the links to the index files and your upgrade will be properly installed.) If you do not have gtar installed locally, you may use tar xvf instead.

   LINUX Note: The "U" option may be omitted, to avoid an error message indicating that gtar could not unlink certain directories because they are non-empty. The error is harmless, in any case.

   unix% gunzip caldb_3.5.5_main.tar.gz
   unix% gunzip acis_bkgrnd_3.5.4.tar.gz
   unix% gtar xvUf caldb_3.5.5_main.tar ./docs ./data
   uniz% gtar xvUf acis_bkgrnd_3.5.4.tar
   

   Note also that we have added the "./docs ./data" specification to the gtar command when unpacking the "main" tar ball above. This causes only the ./docs and ./data directories under $CALDB to be updated from the tar file. When upgrading a pre-existing CalDB, there is no need to update your ./software directory, which is included in the "main" tar ball.

5. UPGRADE FINISHED! Proceed to Check the index configuration below.

Install the CALDB

The instructions for installing the CALDB are basically the same whether you are creating a local or sitewide installation.

1. Select a location for your CALDB at your site. This directory needs to be accessible by all users who are running CIAO.

   If you are attempting to upgrade your existing CalDB version 2.x, you may install the files below in the same directory you used before, or you may elect to rebuild your CalDB in a new location.

2. Download CALDB 3.5.5 Full Installation from Table 1. There are four tar ball files in all, including the "main" tar ball.

   The ACIS Background files have been packaged separately to reduce the space requirements for CalDB when users don't need the background datasets. These background files have now been upgraded with CalDB 3.5.4, as of 01 Apr 2009. They add about 668 MB to the total size of the CalDB. Still, if they are necessary, they can be downloaded rather quickly and unpacked AFTER the MAIN 3.5.5 tarball has been unpacked. We clearly recommend downloading and installing these files on multi-user sites.

   Note that the PSF library files are individually gzipped, then tarred together. It is not necessary to include all these files in the installation as they require a lot of disk space (approximately 6.5 GB); see the PSF Library Manual to determine which files you will require. For sitewide installations (i.e. installations maintained by a systems administrator), it may be preferable to unpack them all if space allows.

3. From within the CALDB directory, gunzip and untar the main file, e.g.:

   unix% gunzip caldb_3.5.5_main.tar.gz
   unix% gtar xvUf caldb_3.5.5_main.tar
   

   If you use gtar, then you should use the xvUf (include the "U" option) in case there are filename conflicts which may not be appropriately clobbered. If you do not have access to gtar then you may use tar xvf instead without worry. The result will be three directory trees at that level, namely docs, data, and software:

   unix% ls
   data/           docs/           software/
   

   The docs directory contains the manifests and versioning files. while software/ contains the tools. The data directory is most important, containing the full CALDB data directory tree.

4. Unzip and untar the ACIS background package, unless you are certain that you will not need any of the blank-field files for making background datasets. Typically observers of extended sources need these files. We recommend multi-user sites to download and install these files in case anyone in your group needs them. NOTE: By unzipping the tarball in the SAME LOCATION AS THE MAIN TARBALL, you get the bkgrnd files exactly where you will want them.

   NOTE: The ACIS blank-sky background files have been upgraded as of CalDB 3.5.4 (July 2009). Therefore you must download and install the files below to have the most up-to-date BKGRND files, and for the script "acis_bkgrnd_lookup" to work properly.

   unix% gunzip acis_bkgrnd_3.5.4.tar.gz
   unix% gtar xvUf acis_bkgrnd_3.5.4.tar
   

5. Optional: untar the PSF library files. The gzipped files will be placed in the ACIS or HRC branch. To unzip the files (which will require 6.5 GB of extra space), move to the appropriate directory - either ./data/chandra/acis/cpf/2dpsf/ (ACIS) or ./data/chandra/hrc/cpf/2dpsf/ (HRC) - and gunzip the files individually.

6. Delete the tarfiles in order to save space. You may wish to complete the installation and verify that the CALDB is working before doing so.

Check the Index Configuration

To ensure that the index files are linked correctly, run this ls command from your CALDB directory:

unix% ls -l data/chandra/*/caldb.indx
ls -l data/chandra/*/caldb.indx
lrwxrwxrwx ... data/chandra/acis/caldb.indx -> ./index/caldbN0049.indx 
lrwxrwxrwx ... data/chandra/ephin/caldb.indx -> ./index/caldbN0003.indx
lrwxrwxrwx ... data/chandra/hrc/caldb.indx -> ./index/caldbN0036.indx 
lrwxrwxrwx ... data/chandra/pcad/caldb.indx -> ./index/caldbN0014.indx
lrwxrwxrwx ... data/chandra/sim/caldb.indx -> ./index/caldbN0004.indx
lrwxrwxrwx ... data/chandra/tel/caldb.indx -> ./index/caldbN0016.indx

where "..." represents the file owner and timestamp. Check that the results from your system match those shown here.

Verify that the CALDB is Working

The first part of this step requires that you have FTOOLS available; if you do not, skip to the quizcaldb example. Note that you do not need FTOOLS in order to use the CALDB with CIAO, only for this verification step.

chkcif

The FTOOL chkcif is the method for verifying the installation of a CALDB after unpacking it. chkcif ("check CALDB index file") reads the index listings and

• verifies that the index for a given instrument is present and in the proper form.
• verifies that all ONLINE files in the index are present in the CALDB. If a file that is indexed as ONLINE is missing, the tool reports it to STDOUT and then exits with an error status.
• lists the OFFLINE files as offline and does not check for them.

A file that is marked OFFLINE is not included in the CalDB directory structure. By the HEASARC standard, such a file might be available in an off-line system such as another disk drive or a web location, and isn't accessible by chkcif. For the Chandra CalDB, "OFFLINE" indicates that a file has been removed from circulation, and is NO LONGER AVAILABLE, and should not be used. Hence, when chkcif returns that a file is OFFLINE, this is normal behavior for the CalDB, and does not indicate any problem with your configuration.

Note that the tool does not verify that each calibration file is whole or uncorrupted, only that it exists in the proper location in the directory structure.

Note that you need to start CIAO 3.4 in the working window (to define some environment vairables) before running chkcif. In this example, CIAO is installed in /soft/ciao:

unix% source /soft/ciao/bin/ciao.csh
unix% chkcif chandra acis STDOUT
 ** chkcif 1.1.4
 ... Contents of CIF /soft/ciao/CALDB/data/chandra/acis/caldb.indx verified
 ... See STDOUT
     for more information
 ** chkcif 1.1.4 completed successfully

This command needs to be repeated for all the Chandra CALDB instrument branches, namely "hrc", "ephin", "pcad", "sim", and "tel".

IMPORTANT: IF the ACIS background files have not been downloaded and unpacked, AND/OR if the ACIS PSF library has not been downloaded and unpacked, then the ACIS branch will report errors, as shown in this example. Otherwise, the tool should return the "chkcif ... completed successfully" line for each instrument, verifying that the CALDB is correctly installed. The same holds for the HRC branch if the HRC PSF library is not downloaded.

quizcaldb

It is also possible to test the installation by querying for a particular file with quizcaldb; see the Querying the CALDB page for more information on queries.

First, start CIAO in the working window. In this example, we request the ACIS phase 2 response input file (mkacisrmf) for an observation taken on 28 June 2001 at 11:00:00 UTC, reprocessed with a CTI-correction applied only to the FI chips. Let us say we are analyzing data from ACIS-S3 (CCD_ID=7):

unix% quizcaldb infile="none" mission="chandra" instrument="ACIS" \
      detector="-" date="2001-06-28" time="11:00:00" filter="-" \
      codename="SC_MATRIX" expr="ccd_id.eq.7.and.cti_app.eq.pppppnpnpp"

/soft/ciao/CALDB/data/chandra/acis/cpf/p2_resp/acisD2000-01-29p2_respN0005.fits[AXAF_CTI_SCATTER_G]

The filename and extension ("FUNCTION") are returned by quizcaldb, indicating that the tool was able to access the CALDB properly. In this case, the CALDB directory in located in /soft/ciao/CALDB/.