Installing CIAO 4.11 Using the ciao-install Script (Recommended)
CIAO 4.11 Science Threads
The ciao-install script is designed to simplify and automate the installation and upgrade of CIAO and the Chandra Calibration DataBase (CALDB). ciao-install FTPs the selected files to your computer, verifies them via md5sum, and unpacks them. The script configures CIAO, sets up the command-line ahelp system, and (optionally) runs the CIAO smoke tests.
The ciao-install.README file file has detailed information on the script. For a summary of options, run the script with the "--help" option.
Last Update: 8 Nov 2019 - updated for CALDB 4.8.5.
- Quick Start: Running ciao-install
- System Requirements
- Installation Issues
- Download the ciao-install script
- Where should CIAO and CALDB be installed?
- Installing the CIAO Software
- How to Install the CALDB in an Independent Directory
- Copy existing CALDB files from previous CIAO installation
- Create a CIAO alias
- Check the CALDB installation
- Additional options
- Clean Up
Quick Start: Running ciao-install
Just looking for a reminder on how to run the ciao-install script? Here is it:
unix% bash /<path>/ciao-install
or, as described further below,
unix% bash /<path>/ciao-install --help
For a detailed description of running the script, read the rest of this thread.
CIAO is supported on a number of operating systems, which are listed on the Platform Support page. Certain system requirements are necessary to successfully install and run CIAO; review these system requirements before continuing with the installation.
Users should be aware of these installation items before installing CIAO 4.11. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.
Remove old parameter files
With every new CIAO release, some parameter files have changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:
unix% rm ~/cxcds_param4/*
For CIAO versions 4.5 to 4.7, inclusive, the IPyton configuration files used by Sherpa and ChIPS were stored in $HOME/.ipython-ciao4.5/. In CIAO 4.8 this location has changed back to $HOME/.ipython-ciao/, which was used in CIAO 4.0 to 4.4. Any modifications made to files in $HOME/.ipython-ciao45/ will need to be made to those in $HOME/.ipython-ciao/.
A warning message may be displayed the first time that ChIPS and Sherpa are run, indicating that old files (from CIAO 4.4 or earlier) were found in $HOME/.ipython-ciao/, and that they have been moved to avoid a possible conflict.
Remove the .fontconfig cache
When switching from a 32-bit to 64-bit build (or vice versa), incompatible .fontconfig files are cached on the user's system. To ensure that the graphical applications such as ChIPS run correctly, remove the .fontconfig cache:
unix% rm ~/.fontconfig/*
With the release of CIAO 4.8, there are no longer any 32-bit builds of CIAO, so this issue will only arise if you maintain multiple versions of CIAO.
Download the ciao-install script
Note: If you download multiple copies of ciao-install it is likely that your browser will append a number onto the file name, eg "ciao-install(1)". Be sure to check that you are using the last/correct version of this file.
Where should CIAO and CALDB be installed?
The CIAO software package and the Chandra Calibration Database (CALDB) can be installed in any location and do not need to be installed with root or super-user privileges.
CIAO unpacks into a directory called ciao-<version>/, i.e. ciao-4.11 for this release. If the current version of CIAO were to be installed into /usr/local/, the full path to CIAO would be /usr/local/ciao-4.11/. This directory (/usr/local/ciao-4.11/) is referred to as the root directory of CIAO in the rest of this thread. It is also the value that the $ASCDS_INSTALL environment variable is set to once CIAO has been started.
In the rest of this thread we will assume CIAO is being installed into the /soft/ directory.
CALDB may be installed in two ways:
- within the CIAO installation, in the directory $ASCDS_INSTALL/CALDB
- in a separate directory, in which case $ASCDS_INSTALL/CALDB will be a soft link to the actual CALDB location.
This thread shows both installation methods.
Users can also now automatically copy or link to an existing CALDB installation. For example if users are upgrading from the previous version of CIAO which has a full copy of the CALDB, then this may then allow ciao-install to download just the much smaller CALDB patch file needed to update that existing CALDB. An example of this is shown below.
Installing CIAO as the root user
It is is strongly suggested that CIAO be installed as a non-privileged, regular user and not as the root user. If CIAO must be installed in a directory owned by root and you are following the manual process, then
- create the ciao-4.11/ directory in the desired location, as the root user;
- use the chown utility to change ownership of this directory to the installer's username;
- unpack CIAO into the newly-created directory as the installer, not as the root user.
Installing the CIAO Software
In this example, the CALDB will be installed as a subdirectory of CIAO. If you want your CALDB to be in a different location, refer to "How to install the CALDB in an independent directory".
ciao-install may be run from any location. It prompts for required information. Here are the prompts and sample answers for a user named "ciaouser"; each prompt is explained in the following section:
unix% bash /<path>/ciao-install ciao-install v1.13 installing ciao-4.11 Requested packages: sherpa chips tools prism obsvis contrib CALDB_main Script log file is /home/ciaouser/ciao-install-19-08-07.09.38.04.log Download directory for tar files (/home/ciaouser): /soft/download CIAO installation directory (/home/ciaouser): /soft Run smoke tests? (y|n) (y): Delete tar files after install? (y|n) (n): Save these settings? (y|n) (y): Preparing to install CIAO for Linux 64 bit
Note that tab completion may be used at these prompts.
1. Requested packages
After printing the script version number, ciao-install lists the packages that were requested for this installation:
sherpa chips tools prism obsvis contrib CALDB_main
This example is installing the standard installation: CIAO tools, ChIPS and Sherpa, the CALDB (except for the blank-sky files), and the science scripts.
2. The installation log
The log file - ciao-install-<date>.log - records the script status information; this is also printed to the screen. The log file can be used to help diagnose problems via the CXC Helpdesk.
3. Download directory for tar files
The download directory is a location to which the software and CALDB tarfiles are downloaded before the software is installed. For this installation the directory name is /soft/download/:
Download directory for tar files (/home/ciaouser): /soft/download
4. CIAO installation directory
The next prompt is for the location of CIAO; we want CIAO to be installed into /soft/ciao-4.11 so we answer /soft rather than accept the default value:
CIAO installation directory (/home/ciaouser): /soft
5. Run smoke tests?
The next prompt asks you whether to run the CIAO smoke tests; these are some simple tests we provide to help validate the installation. Is is strongly suggested that you accept the default value of y here:
Run smoke tests? (y|n) (y):
6. Delete tar files after install?
Should the tarfiles be deleted when ciao-install is down running? The default value is n:
Delete tar files after install? (y|n) (n):
The tarfiles can also be removed manually later.
7. Save these settings?
The final prompt is to ask you whether the default answers for the prompts should be updated to use your current selection. If you answer "yes" (y), then the values will be written to the text file ~/.ciaoinstall.rc and will be used the next time the tool is run. In this case we accept the default value:
Save these settings? (y|n) (y):
8. Which build will be installed
After the prompts, the script prints out a line indicating which CIAO build will be installed:
Preparing to install CIAO for Linux 64 bit
At this point, the script begins to download the software tarfiles and install them (e.g. gunzip and untar them). The name, filesize, and download location of each file is printed to the screen:
Downloading file ciao-control (1 Kb) to /soft/download Downloading file ciao-4.11-bin-core-Linux64.tar.gz (81 Mb) to /soft/download Verifying file /soft/download/ciao-4.11-bin-core-Linux64.tar.gz Installing file /soft/download/ciao-4.11-bin-core-Linux64.tar.gz in /soft/ Downloading file ciao-4.11-bin-chips-Linux64.tar.gz (50 Mb) to /soft/download Verifying file /soft/download/ciao-4.11-bin-chips-Linux64.tar.gz Installing file /soft/download/ciao-4.11-bin-chips-Linux64.tar.gz in /soft/ Downloading file ciao-4.11-bin-prism-Linux64.tar.gz (743 Kb) to /soft/download Verifying file /soft/download/ciao-4.11-bin-prism-Linux64.tar.gz Installing file /soft/download/ciao-4.11-bin-prism-Linux64.tar.gz in /soft/ Downloading file ciao-4.11-bin-obsvis-Linux64.tar.gz (1 Mb) to /soft/download Verifying file /soft/download/ciao-4.11-bin-obsvis-Linux64.tar.gz Installing file /soft/download/ciao-4.11-bin-obsvis-Linux64.tar.gz in /soft/ Downloading file ciao-4.11-bin-graphics-Linux64.tar.gz (29 Mb) to /soft/download Verifying file /soft/download/ciao-4.11-bin-graphics-Linux64.tar.gz Installing file /soft/download/ciao-4.11-bin-graphics-Linux64.tar.gz in /soft/ Downloading file ciao-4.11-bin-sherpa-Linux64.tar.gz (238 Mb) to /soft/download Verifying file /soft/download/ciao-4.11-bin-sherpa-Linux64.tar.gz Installing file /soft/download/ciao-4.11-bin-sherpa-Linux64.tar.gz in /soft/ Downloading file ciao-4.11-bin-tools-Linux64.tar.gz (87 Mb) to /soft/download Verifying file /soft/download/ciao-4.11-bin-tools-Linux64.tar.gz Installing file /soft/download/ciao-4.11-bin-tools-Linux64.tar.gz in /soft/ Downloading file ciao-4.11.1-patch-bin-Linux64.tar.gz (19 Mb) to /soft/download Verifying file /soft/download/ciao-4.11.1-patch-bin-Linux64.tar.gz Installing file /soft/download/ciao-4.11.1-patch-bin-Linux64.tar.gz in /soft/ Downloading file ciao-4.11-contrib-4.tar.gz (2 Mb) to /soft/download Verifying file /soft/download/ciao-4.11-contrib-4.tar.gz Installing file /soft/download/ciao-4.11-contrib-4.tar.gz in /soft/
If you get an error about being unable to download the CALDB files, please refer to the Downloading the CALDB: FTP passive mode note at the end of this thread. The files sizes will be different for different platforms.
After the files are downloaded and unpacked, the script configures CIAO and reindexes the CIAO help files ("ahelp"):
running chcon to allow CIAO to work with SELinux Running configure ./configure Re-indexing ahelp system Creating binary compiled python modules
The 'running chcon' line only appears on Linux installations. It indicates that the installation has been updated to run under Security-Enhanced Linux, if is installed on your system.
The script will then run the smoke tests. Since the smoke tests include tests of a number of the GUI applications in CIAO, windows will appear and disappear during these tests. This can make using your computer difficult, since these windows may take the mouse focus or overlap your working windows.
The screen output provides a count of how many tests have been run:
Running smoke tests Requested test suite includes 42 tests [1/42] Running test ahelp-smoke001 ... PASS [2/42] Running test prism-smoke001 ... PASS [3/42] Running test prism-smoke002 ... PASS ...
This will continue until all the tests have finished, at which point you should see
[40/42] Running test tools-dmstat-smoke001 ... PASS [41/42] Running test tools-dmstat-smoke002 ... PASS [42/42] Running test tools-dmstat-smoke003 ... PASS All smoke tests PASSED. Smoke tests complete. All tests passed! Processing complete! Script Log file is /home/ciaouser/ciao-install-19-08-07.09.38.04.log
Verifying test successes
A successful test is logged with a "... PASS", while an unsuccessful test reports:
[7/42] Running test <testname> ... FAIL Review /tmp/smoke.username/<testname>/diff.log for details
The smoke tests print a summary of successful and failed tests. If all tests were successful:
All smoke tests completed: PASS ************************************************************* Smoke tests complete. All tests passed!
or if any failed:
1 out of 42 smoke tests FAILED. !! CIAO MAY HAVE FAILED TO INSTALL PROPERLY !! Please check http://cxc.harvard.edu/ciao4.7/bugs/smoke.html for a list of known issues before contacting the Help Desk.
Known Smoke Test Issues
There is a bug page for known installation and smoke test issues. If you experience a problem that isn't listed there, submit it to CXC Helpdesk.
You can now proceed to the creating a CIAO alias step.
How to Install the CALDB in an Independent Directory
The destination directory for the CALDB must exist before running ciao-install. In this example, we want to install the CALDB into /soft/CALDB. The directory is created, then --caldb switch is used to set this location in ciao-install.
unix% mkdir /soft/CALDB unix% bash /<path>/ciao-install --caldb /soft/CALDB ciao-install v1.13 installing ciao-4.11 Requested packages: sherpa chips tools prism obsvis contrib CALDB_main Script log file is /home/ciaouser/ciao-install-19-08-07.09.38.04.log Download directory for tar files (/home/ciaouser): /soft/download CIAO installation directory (/usr): /soft Run smoke tests? (y|n) (y): Delete tar files after install? (y|n) (n): Save these settings? (y|n) (y):
ciao-install is run one time: it installs the CIAO software in /soft, installs the CALDB in /soft/CALDB, and links CIAO to the CALDB directory. It then runs the smoke tests, as described in detail in the "Installing the Software: CALDB within CIAO" section.
ciao-install can also be used to apply patches to CIAO, the CALDB, and the contributed software scripts. When applying CALDB patches, users should not use the --caldb option. ciao-install will apply the CALDB patches through the existing symbolic links to the independent CALDB directory.
Copy existing CALDB files from previous CIAO installation
Users can now choose to automatically copy the CALDB files from an existing CIAO directory into the current version. This may allow ciao-install to then download just the much smaller CALDB patch file.
unix% bash /path/ciao-install --copy-caldb /soft/ciao-4.7/CALDB ciao-install v1.13 installing ciao-4.11 Requested packages: sherpa chips tools prism obsvis contrib CALDB_main Script log file is /home/ciaouser/ciao-install-19-08-07.09.38.04.log Download directory for tar files (/soft): CIAO installation directory (/soft): Run smoke tests? (y|n) (y): Delete tar files after install? (y|n) (n): Preparing to install CIAO for Linux 64 bit Downloading file ciao-control (1 Kb) to /soft Copying CALDB from /soft/ciao-4.7/CALDB to /soft/ciao-4.11 ... Installing file /soft/ciao-4.9-bin-tools-Linux64.tar.gz in /soft CALDB patch download required. Downloading file caldb_4.8.5_upgrade.tar.gz (607 Mb) to /soft Verifying file /soft/caldb_4.8.5_upgrade.tar.gz Installing file /soft/caldb_4.8.5_upgrade.tar.gz in ciao-4.8/CALDB ...
To save disk space, some users may wish to simply create a link to the existing CALDB directory. This can be done using the --link-caldb option. Note: users should be careful when using links since any patches/upgrades will be visible to all the versions of CIAO that link to that location and removing the old CIAO installation will break the current version.
Create a CIAO alias
It is strongly suggested that you create an alias to make it easy to set up and use CIAO.
The alias syntax depends on the shell (tcsh, csh, bash); see this FAQ for help in determining which shell you are using.
csh/tcsh users should add the following
to their $HOME/.cshrc file.
alias ciao "source /soft/ciao-4.11/bin/ciao.csh"
bash users should add the following
to their $HOME/.bashrc file.
alias ciao="source /soft/ciao-4.11/bin/ciao.bash"
ksh users should add the following
to their $HOME/.login file.
alias ciao="source /soft/ciao-4.11/bin/ciao.ksh"
Users can create the file if one does not already exist. These are simply text files that can be edited with the users favorite text editor (emacs, vi, gedit, etc.).
Once the alias is defined, you will be able to simply type
and CIAO will be setup and ready to use in that window.
Check the CALDB installation
The scripts package contains the check_ciao_caldb tool which performs a simple test of the Chandra CALDB installation. Users must have selected the default full CIAO installation or have selected the Scripts Package for this utility to be installed.
After starting CIAO, running the tool should produce screen output like the following:
unix% ls -1F /soft/ciao-4.11/CALDB/ data/ docs/ software/ unix% check_ciao_caldb CALDB environment variable = /soft/ciao-4.11/CALDB CALDB version = 4.8.5 release date = 2018-09-11T16:00:00 UTC CALDB query completed successfully.
If CALDB is installed in a different location, the check_ciao_caldb output will include the line:
CALDB environment variable = /soft/ciao-4.11/CALDB which is a link to = /soft/CALDB/
For a more-thorough validation please follow the chkcif steps described on the download CALDB page.
The ciao-install script has several other additional options. Use --help to review them.
unix% bash /path/ciao-install --help ciao-install v1.13 Usage: /path/ciao-install [options...] -h --help Print this message --download-only Download only Do not install --install-only Install only Do not download --download <dir> Download directory --prefix <dir> Install directory --logdir <dir> Log file directory --config <--with-top=dir> Extra configure switches --caldb <dir> Location of the CALDB --copy-caldb <dir> Location of to copy into ciao --link-caldb <dir> Location of to link into ciao --system <system> System to install (Linux64, osxml64, osxElCap) --batch Batch mode (no prompts) --silent Silent mode (implies batch) No tty output --delete-tar Delete tar files after install. --add <segment> Add additional segment to CIAO. --remove <segment> Force this segment to not install in CIAO. -f --force Force re-install -v --version Report version and exit
Finally, you may delete the tarfiles from the original download directory. For this example we would say:
unix% cd /soft/download unix% rm ciao-4.11*gz unix% rm caldb_4.8.5_main.tar.gz
You may also delete the log file(s) - e.g. ciao-install-<date>.log - once you have reviewed them for errors.
Downloading the CALDB: FTP passive mode
Some users are unable to download the CALDB via ciao-install. A message of this form is printed:
ERROR: Unable to retrieve ftp://cda.harvard.edu/pub/arcftp/caldb_4/caldb_4.8.5_main.tar.gz ERROR: Unable to download caldb_4.8.5_main.tar.gz ERROR: Bad Download. Please try again. If the problem continues Please contact the CfA helpdesk.
This is because the user's FTP has defaulted to extended passive mode for the file transfer. Extended passive mode is not supported for the Archive web server, which hosts the CALDB tarfiles.
The workaround is for users to create a $HOME/.netrc file containing the following:
default macdef init epsv4 off
Note that there is a blank line after the "epsv4 off" entry.
Some Mac users have reported than an additional "passive off" line may be necessary, so that the .netrc file appears as:
default macdef init epsv4 off passive off
The re-run ciao-install to download the missing files.
The CIAO installation is now complete. For help getting started with the software, read the Introductory science threads.
|20 Apr 2009||The CIAO 4.1.2 release coincides with the beta release of the ciao-install script.|
|16 Jun 2009||clarified why ciao-install must be run twice to install CALDB in a separate location|
|15 Dec 2009||updated for the CIAO 4.2 release: ciao-install is the recommended method for installing CIAO. Many enhancements have been made to the script: it now downloads and unpacks the tarfiles, installs the software, and runs the smoke tests.|
|29 Dec 2009||Added Installing on Snow Leopard note|
|16 Mar 2010||Added Downloading the CALDB: FTP passive mode note|
|05 Apr 2010||updated for CALDB 4.2.1: version number of CALDB main tarfile|
|19 Apr 2010||updated for CALDB 4.2.2: version number of CALDB main tarfile|
|27 Sep 2010||updated for CALDB 4.3.1: version number of CALDB main|
|15 Dec 2010||updated for the CIAO 4.3 and CALDB 4.4.1 releases: minor changes to screen output|
|15 Dec 2011||reviewed for the CIAO 4.4 and CALDB 4.4.7 releases: minor changes to screen output|
|13 Jun 2012||reviewed for the CIAO 4.4.1 and CALDB 4.4.10 releases: minor changes to screen output|
|03 Dec 2012||Review for CIAO 4.5|
|19 Nov 2013||Updated for CIAO 4.6. Cosmetic updates|
|09 May 2014||Re-order sections. Moved creating ciao alias before check caldb. Added note that check_caldb requires the contrib package to be installed.|
|11 Dec 2014||Updated for CIAO 4.7. Version updates and minor text edits.|
|29 Jun 2015||added additional information for passive FTP mode.|
|15 Dec 2015||Updated for CIAO 4.8. Added information about new CALDB flags.|
|09 Feb 2016||Updated for CIAO 4.8.1.|
|24 Feb 2016||Updated for CIAO 4.8.1/4.8.2.|
|27 May 2016||updated for CALDB 4.7.2: version number of CALDB main|
|04 May 2017||updated for CALDB 4.7.4.|
|20 Jul 2017||updated for CALDB 18.104.22.168 and contrib 4.9.4.|
|18 Aug 2017||updated for CALDB 4.7.6.|
|13 Dec 2017||updated for CALDB 4.7.7.|
|09 Apr 2018||Updated for CIAO 4.10, CALDB 4.7.8.|
|11 Sep 2018||updated for CALDB 4.8.0.|
|25 Oct 2018||updated for CALDB 4.8.1.|
|03 Sep 2019||updated for CALDB 4.8.4.|
|08 Nov 2019||updated for CALDB 4.8.5.|