Search https://cxc.harvard.edu/ciao/
Contact the CXC HelpDesk
Last modified: 7 Mar 2023

URL: https://cxc.cfa.harvard.edu/ciao/threads/ciao_install_conda/
/ ciao / threads / ciao_install_conda /
Return to Threads Page: Top | All | Intro

Installing CIAO with conda

CIAO 4.15 Science Threads

Overview

Synopsis:

This thread shows how to install CIAO using the conda package manager. Users learn how to perform the basic steps and then can learn about the additional options they can use to customize their installation.

Related Links:

Last Update: 7 Mar 2023 - Tweak for force python 3.10. Added CONDA_SUBDIR info.

Contents

Quick Start

The following commands are all that is needed to install CIAO using an existing conda installation and verify that everything is working: 

$ conda create -n ciao-4.15 -c https://cxc.cfa.harvard.edu/conda/ciao \
  -c conda-forge ciao sherpa ds9 ciao-contrib caldb_main marx python=3.10
$ conda activate ciao-4.15
$ conda env config vars set MARX_ROOT=$CONDA_PREFIX
$ cd $ASCDS_INSTALL/test
$ make

CIAO 4.15 requires using the conda-forge channel which provides a newer version of libgfortran which is required for XSpec 12.12.0.

The rest of this thread discusses these steps in more detail and presents the user with additional options.

Detailed Instructions

Introduction to conda

CIAO 4.15.2 can be installed using conda.

conda is a package management and environment management system. It is similar to other package managers (such as yum, apt, macports, dnf) but allows for users to create multiple environments with multiple versions of packages which can co-exist on the same system. Users can easily install pre-compiled and pre-configured packages from curated channels (repositories) into various environments. Users then activate the environment to run the applications.

With conda , Linux and Mac users can choose to install CIAO 4.15.2 using either Python 3.10, 3.9, or 3.8. The default version is subject to change based on the availability of other packages on the conda-forge channel. Using the conda package manager users can also install additional third-party packages such as astropy, scipy, and jupyter notebook.

Install conda

Users must already have conda installed on their systems. This can be either Anaconda or miniconda. CIAO 4.15.2 is available for Linux and MacOSX. CIAO does not work with Windows, not even when running Windows Subsystem for Linux.

[NOTE]
Minimum conda version?

Some users, macOSX in particular, have reported problems with using old versions of conda. CIAO has successfully be installed using conda 4.7.6 and newer. Users should consider upgrading their base conda installation before installing CIAO.

Users can check if they have conda and the version using 

$ conda --version
conda: Command not found.

In this case, conda is not installed or otherwise not setup in the user's PATH.

[IMPORTANT]
Help installing conda

Users should consult the Anaconda troubleshooting guide if they encounter problems installing conda instead of the CXC helpdesk. Timeout's during download are frequent. Users are encouraged to simply try again.

Below are the basic instructions for installing miniconda. 

Linux$ curl -o get_conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh 

or 

macOS$ curl -o get_conda.sh https://repo.anaconda.com/miniconda/Miniconda3-latest-MacOSX-x86_64.sh

Users can then run the interactive installation script 

$ bash get_conda.sh 

... accept license and answer prompts ... 


Preparing transaction: done
Executing transaction: done
installation finished.
Do you wish the installer to initialize Miniconda3
by running conda init? [yes|no]
[no] >>>
[IMPORTANT]
tcsh shell users

Users who use the tcsh shell must not run the conda init command if you are using conda 4.7.4 or earlier. There is a serious bug that may prevent users from running commands or may even prevent users from logging into their user account.

This does not affect users using the bash shell. If unsure which you are using check the output from 

$ echo $0
bash

After the installer has finished users will want to be sure that the conda command is setup in their PATH environment. 

bash$ PATH="${PATH}:<<directory where miniconda was installed>>/bin"

or

tcsh% set path = ( ${path} <<directory where miniconda was installed>>/bin )

Install CIAO into a new conda environment

The CXC is hosting the CIAO 4.15.2 packages on our own conda channel. This is due to the size of the packages.

To create a new conda environment with CIAO installed users run the following command: 

$ conda create -n ciao-4.15 -c https://cxc.cfa.harvard.edu/conda/ciao \
  -c conda-forge ciao sherpa ds9 ciao-contrib caldb_main marx

This will create an environment named ciao-4.15 and perform a standard CIAO installation including the standard CALDB. Users who need the ACIS or HRC background event files should replace caldb_main with simply caldb.

Users can choose to install CIAO with Python 3.10, 3.9 , or 3.8. Use the following example to select a specific python version: 

$ conda create -n ciao-4.15_python38 \
  -c https://cxc.cfa.harvard.edu/conda/ciao \
  -c conda-forge \
  python=3.10 ciao sherpa ds9 ciao-contrib caldb_main marx
[IMPORTANT]
Installation on Apple Silicon Systems

CIAO is only available as Intel x86-64 binaries. Users of Apple Silicon-based systems (M1, M1 Pro/Max/Ultra, and M2) must use an Intel x86-64 MacOS version of python. Depending on the version/architecture of conda that is installed, some users may get an error message that looks like this: 

PackagesNotFoundError: The following packages are not available from current channels:

- sherpa
- ciao
- ds9
- marx

Current channels:

- https://cxc.cfa.harvard.edu/conda/ciao/osx-arm64
...

Note the reference to the osx-arm64. If this error occurs, then users can tell conda to install the Intel x86-64 version using the CONDA_SUBDIR environment variable 

$ env CONDA_SUBDIR=osx-64 conda create -n ciao-4.15 \
  -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge \
  ciao sherpa ds9 ciao-contrib caldb_main marx python=3.10

$ conda activate ciao-4.15
$ conda config --env --set subdir osx-64

The first command tells conda to install the Intel x86-64 binaries. The second and third commands ensure that any updates or additional packages added to that environment will continue to use the Intel x86-64 versions.

Note: Please be sure that you have also installed Apple's Rosetta2 software to be able to run Intel applications. 

$ softwareupdate --install-rosetta
[TIP]
Tip

Timeouts during download are frequent. Users are encouraged to simply try again.

[NOTE]
Blank sky background files

To install the ACIS and HRC blank sky background files users should replace caldb_main with caldb. 

$ conda create -n ciao-4.15 \
  -c https://cxc.cfa.harvard.edu/conda/ciao \
  -c conda-forge \
  ciao sherpa ds9 ciao-contrib caldb marx python=3.10
[TIP]
Problem with downloading XSPEC models or CALDB

Users experiencing problems downloading the CALDB files and/or the XSPEC model files can follow the Alternative download instructions.

Alternative: fixing the installed package versions to avoid possible conflicts

The default conda installation instructions above will select the latest version of the packages, including the packages that CIAO relies on, such as NumPy and matplotlib. As new versions of these packages are made available, it is possible for their to be problems. An alternative approach is to install a fixed set of versions, which fits under the umbrella of "reproducible research", and can be performed with a conda environment file.

Users can select a fixed set of packages using one of the two commands below - the difference is whether the CALDB packages (it selects the caldb_main packages) are included or not:

Linux

$ conda env create -n ciao-4.15 cxc/ciao-4.15-caldb-linux --dry-run
  -or-
$ conda env create -n ciao-4.15 cxc/ciao-4.15-linux --dry-run

macOS

$ conda env create -n ciao-4.15 cxc/ciao-4.15-caldb-macos --dry-run
  -or-
$ conda env create -n ciao-4.15 cxc/ciao-4.15-macos --dry-run

The --dry-run argument will list what packages are to be installed, and where they are from (the CXC channel, conda-forge, or conda). Once you have reviewed this list you can repeat the call without the --dry-run option.

Adding extra CALDB packages

Users can then later include the blank sky background files with: 

$ conda install -n ciao-4.15 -c https://cxc.cfa.harvard.edu/conda/ciao/ \
  -c conda-forge acis_bkg_evt hrc_bkg_evt

Activate CIAO environment

Once the CIAO environment is created, it has to be activated. This is equivalent to source'ing the CIAO setup scripts.

For bash shell users, make sure conda's activate script is in your PATH and then simply 

$ source activate ciao-4.15

where ciao-4.15 is the name of the environment you just created.

For tcsh shell users 

% alias conda source <<directory where miniconda was installed>>/etc/profile.d/conda.csh
% conda activate ciao-4.15

Users may wish to add the alias to their $HOME/.tcshrc startup files.

By default, users will notice the shell prompt has changed to indicate the current conda environment. 

$ source activate ciao-4.15
(ciao-4.15) $

MARX setup

If you have installed marx and are going to be using the simulate_psf script, the you should setup the MARX_ROOT environment variable in your conda environment 

$ conda env config vars set MARX_ROOT=$CONDA_PREFIX

You should deactivate and then activate the CIAO environment to pickup the new environment variable. Then check it is set using plist 

$ plist simulate_psf

Parameters for /export/miniconda/envs/ciao-4.15/param/simulate_psf.par

...
    (marx_root = ${MARX_ROOT} -> /export/miniconda/envs/ciao-4.15) Directory where MARX is installed

If you plan to run marx by itself, then you will need to copy the parameter file from the installation directory: $CONDA_PREFIX/share/marx/pfiles/

Run the smoke tests

Users are encouraged to run the CIAO smoke tests to ensure everything is installed and working correctly. 

(ciao-4.15)$ cd $ASCDS_INSTALL/test
(ciao-4.15)$ make

If users choose to omit packages (eg sherpa or ds9), then smoke tests associated with those packages will fail.

Install additional 3rd party packages

Additional packages can then be installed into the CIAO environment using the conda installer. 

(ciao-4.15)$ conda install -n ciao-4.15 -c conda-forge jupyter astropy scipy

Due to the constraints on python version and numpy version, users may not be able to install certain packages, or possibly the latest versions of certain packages such as astropy or scipy. Similarly, users may not be able to install CIAO into an existing environment with incompatible versions of python or numpy.

If installing additional CIAO packages, users need to remember to include the CXC conda channel. For example to add the ACIS blank sky background files to an existing CIAO installation the command would look like: 

(ciao-4.15)$ conda install -n ciao-4.15 -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge acis_bkg_evt

Users can also install packages using pip or other package managers 

(ciao-4.15)$ pip3 install bash_kernel

Leave CIAO environment

Use the deactivate command to leave the conda environment: 

(ciao-4.15)$ conda deactivate

or simply exit the terminal window. deactivate can be useful when users need to install additional package which may conflict with the CIAO environment or when updating CIAO components.

Updating CIAO environment

[IMPORTANT]
Install CIAO 4.15.2 in a new environment

CIAO 4.15.2 requires using the conda-forge channel to get a version of libgfortran that is compatible with XSpec 12.12. Because of this, users are encourage to create a new conda environment for CIAO 4.15.2 rather than attempting to update an existing environment. This section is provided as a reference for updating CIAO 4.15.2 components, such as a new CALDB release, a new contributed scripts release, or a CIAO 4.15.2 patch.

Users can install the latest CIAO and CALDB updates using the conda update command 

$ conda update -n ciao-4.15 \
  -c https://cxc.cfa.harvard.edu/conda/ciao \
  -c conda-forge \
  ciao sherpa ds9 ciao-contrib caldb_main marx --freeze-installed

Only the packages listed will be updated. To update all the packages in the environment use the --all option. We recommend using the --freeze-installed flag to prevent other packages such as numpy from updating which may could potentially cause a conflict with CIAO.

After updating, users may want to run the conda clean command to completely remove the previous versions. This is especially true when updating the CALDB since each CALDB update is a full release which each requires 5-12Gb of disk space. 

$ conda clean --all

Alternative Download Options

Users on slow internet or wireless connections may experience problems downloading the large data files associated with the Chandra CALDB and the XSPEC models. Often users will be able to simply retry again successfully; however, sometimes the server timeouts are persistent.

CALDB alternatives

Partial CALDB download

Users who only need to analyze a limited number of Chandra datasets (OBS_IDs), can use the download_obsid_caldb script to download only the calibrations files associated with individual observations (including optional blank sky background files). 

unix% conda create -n ciao_no_caldb -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge \
  ciao sherpa ciao-contrib ds9 python=3.10
...
unix% conda activate ciao_no_caldb
(ciao_no_caldb) unix% download_chandra_obsid 4425
...
(ciao_no_caldb) unix% download_obsid_caldb 4425
Output CALDB directory (${ASCDS_INSTALL}/CALDB -> /soft/miniconda/envs/ciao_sans_caldb/CALDB): 
download_obsid_caldb
          infile = 4425
          outdir = /soft/miniconda/envs/ciao_sans_caldb/CALDB
      background = no
         missing = no
         clobber = no
         verbose = 1
            mode = ql

Retrieving files for CALDB_VER = 4.9.6
Retrieving CALDB index files
Processing infile=4425/primary/acisf04425N004_evt2.fits.gz
Retrieving CALDB data files
    Filename:                               0------------------1
    telD1999-07-23geomN0006.fits            ####################
    telD1999-07-23aimptsN0002.fits          ####################
    telD1999-07-23tdetN0001.fits            ####################
    telD1999-07-23skyN0002.fits             ####################
    telD1999-07-23sgeomN0001.fits           ####################
    hrmaD1996-12-20axeffaN0008.fits         ####################
    hrmaD1996-12-20vignetN0003.fits         ####################
    acisD1997-04-17qeN0006.fits             ####################
    acisD2002-02-01qeuN0007.fits            ####################
    acisD2000-11-28badpixN0004.fits         ####################
    acisD1999-08-13contamN0014.fits         ####################
    acisD1996-11-01gradeN0004.fits          ####################
    acisD2000-01-29grdimgN0001.fits         ####################
    acisD2000-01-29gain_ctiN0008.fits       ####################
    acisD1996-11-01evtspltN0002.fits        ####################
    acisD2002-08-01ctiN0009.fits            ####################
    acisD2003-02-01t_gainN0008.fits         ####################
    acisD1999-07-22subpixN0001.fits         ####################
    acisD2000-01-29fef_pha_ctiN0004.fits    ####################
    acisD1999-09-16dead_areaN0001.fits      ####################
    hrmaD1996-12-20reefN0001.fits           ####################
    acisD2000-01-29p2_respN0008.fits        ####################
    hrmaD1996-11-01wpsfN0001.fits           ####################
    acisD2000-01-29osip_ctiN0006.fits       ####################
    acisD2002-11-15gtilimN0004.fits         ####################

Be sure to source the new setup scripts to
use these CALDB files.

(t)csh:
source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.unix

bash:
source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.sh

(ciao_no_caldb) unix% source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.unix
[TIP]
Setup CALDB environment variables

To avoid needing to setup these environment variable each time the conda environment is activated, users can permanently set them using the conda env config command: 

$ source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.sh
$ conda env config vars set CALDB=$CALDB
$ conda env config vars set CALDBCONFIG=$CALDBCONFIG
$ conda env config vars set CALDBALIAS=$CALDBALIAS

This only needs to be done once. Users will need to deactivate and then activate their CIAO environment to pick up the new environment variables.

The partial CALDB can be installed anywhere; the default is the same location as the full CALDB. When run again, the script will skip any CALDB files that have already been downloaded and will only download the missing files.

Install separately using the ciao-install tar files

Similar to above, users can download the entire CALDB and install it manually. We have a copy of the CALDB files in our Google Drive in the CALDB folder. The Google Drive downloads are less likely to time out. 

unix% conda create -n ciao_no_caldb -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge ciao sherpa ciao-contrib ds9
...
unix% conda activate ciao_no_caldb
(ciao_no_caldb) unix% cd $ASCDS_INSTALL
(ciao_no_caldb) unix% mkdir CALDB
(ciao_no_caldb) unix% cd CALDB
(ciao_no_caldb) unix% tar xfz ~/Downloads/caldb_4.9.6_main.tar.gz
(ciao_no_caldb) unix% tar xfz ~/Downloads/acis_bkgrnd_4.9.2.1.tar.gz
(ciao_no_caldb) unix% tar xfz ~/Downloads/hrc_bkgrnd_4.7.7.tar.gz

  -- For bash/dash/zsh users
(ciao_no_caldb) unix% export CALDB=$ASCDS_INSTALL/CALDB
(ciao_no_caldb) unix% export CALDBCONFIG=${CALDB}/software/tools/caldb.config
(ciao_no_caldb) unix% export CALDBALIAS=${CALDB}/software/tools/alias_config.fits

  -- For tcsh users
(ciao_no_caldb) unix% setenv CALDB $ASCDS_INSTALL/CALDB
(ciao_no_caldb) unix% setenv CALDBCONFIG ${CALDB}/software/tools/caldb.config
(ciao_no_caldb) unix% setenv CALDBALIAS ${CALDB}/software/tools/alias_config.fits
[TIP]
Setup CALDB environment variables

To avoid needing to setup these environment variable each time the conda environment is activated, users can permanently set them using the conda env config command: 

$ source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.sh
$ conda env config vars set CALDB=$CALDB
$ conda env config vars set CALDBCONFIG=$CALDBCONFIG
$ conda env config vars set CALDBALIAS=$CALDBALIAS

This only needs to be done once. Users will need to deactivate and then activate their CIAO environment to pick up the new environment variables.

Install individual conda tar files

Users can also install the individual conda tar files from our Google Drive in the conda/ciao/noarch folder. The Google Drive downloads are less likely to time out. 

unix% conda create -n ciao_no_caldb -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge ciao sherpa ciao-contrib ds9
...
unix% conda activate ciao_no_caldb
(ciao_no_caldb) unix% conda install ~/Downloads/caldb_main-4.9.6-0.tar.bz2
(ciao_no_caldb) unix% conda install ~/Downloads/acis_bkg_evt-4.9.2.1-1.tar.bz2   (optional)
(ciao_no_caldb) unix% conda install ~/Downloads/hrc_bkg_evt-4.7.7.1-1.tar.bz2    (optional)

The advantage of this method is that it will also install the activation scripts; however, when updates are requested conda may try to download the entire CALDB files again.

Install from a local channel

Users who have downloaded the conda tar file from the Google Drive (see above) can also setup a local conda repository and install from there. The advantage is that installing using the individual tar files does not check for dependencies; whereas installing from a local repository does resolve dependencies. 

unix% conda install conda-build
...
unix% mkdir -p /tmp/my_conda_dir/noarch
unix% cp ~/Downloads/caldb_main-4.9.6-0.tar.bz2 /tmp/my_conda_dir/noarch
unix% cp ~/Downloads/acis_bkg_evt-4.9.2.1-1.tar.bz2 /tmp/my_conda_dir/noarch   (optional)
unix% cp ~/Downloads/hrc_bkg_evt-4.7.7.1-1.tar.bz2 /tmp/my_conda_dir/noarch    (optional)

unix% conda index /tmp/my_conda_dir

unix% conda create -n ciao_my_caldb -c /tmp/my_conda_dir/ -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge ciao sherpa ciao-contrib ds9 caldb_main
...
  ca-certificates    conda-forge/linux-64::ca-certificates-2021.10.8-ha878542_0
  caldb_main         my_conda_dir/noarch::caldb_main-4.9.6-0
  cfitsio            ciao/linux-64::cfitsio-4.0.0-h9bf148f_2
...
unix% conda activate ciao_my_caldb

XSPEC alternatives

Given the platform specific dependencies, the only alternative method to download the sherpa XSPEC model files is to download the correct tar file from our Google Drive and install it using the local repository method described above 

unix% conda install conda-build
...

 -- For Linux
unix% mkdir -p /tmp/my_conda_dir/linux-64
unix% cp ~/Downloads/xspec-modelsonly-12.12.0-h6f05748_1.tar.bz2 /tmp/my_conda_dir/linux-64

 -- For macOS
unix% mkdir -p /tmp/my_conda_dir/osx-64
unix% cp ~/Downloads/xspec-modelsonly-12.12.0-hd4aef4f_1.tar.bz2 /tmp/my_conda_dir/osx-64


unix% conda index /tmp/my_conda_dir

unix% conda create -n ciao_my_xspec -c /tmp/my_conda_dir/ -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge ciao sherpa ciao-contrib ds9 caldb_main
...
  xerces-c           conda-forge/linux-64::xerces-c-3.2.3-hfe33f54_1
  xpa                ciao/linux-64::xpa-2.1.20-h9bf148f_0
  xspec-modelsonly   my_conda_dir/linux-64::xspec-modelsonly-12.12.0-h6f05748_1
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  zlib               conda-forge/linux-64::zlib-1.2.11-h36c2ea0_1013
...
unix% conda activate ciao_my_xspec

Caveats with Alternative Downloads

Please keep the following caveats in mind

  • Using a local channel will list the package's channel (when running "conda list") as the directory location
  • Using an explicit package will list the package's channel (when running "conda list") as "unknown"
  • If you later try to install from the official channel, you will get messages of the package getting superseded by a higher-priority channel.

Known Problems

Users should be aware of these known issues before using the conda ciao installation.

  1. The conda installation will seem to hang during the installation process when retrieving the largest segments: the XSPEC models and CALDB files. Do not kill the installation. If it times out then it will be able resume and only install the failed packages.

  2. Users who keep the CALDB separate from the CIAO installation must setup the following environment variables independently from the conda setup. 

    $ export CALDB=/Some/Other/Directory/CALDB
$ export CALDBCONFIG=${CALDB}/software/tools/caldb.config
$ export CALDBALIAS=${CALDB}/software/tools/alias_config.fits

    These can be setup in the users' login script ($HOME/.bash_profile), or setup each time the environment is activated. They can be setup either before or after the conda environment is activated.

    The CALDB version will not be reported by the ciaover command if it is not installed by conda

  3. Due to the constraints on python version and numpy version, users may not be able to install CIAO in the same environment as existing packages. Known incompatibilities include:

    • CIAO and StSCI's astroconda which fail due to version incompatibilities in the CFITSIO library.
    • There is a conflict between CIAO and HEASARC's XSPEC package. Users must

      1. First, initialize their HEASARC environment

        $ export HEADAS=/soft/lheasoft/headas/x86_64-pc-linux 
$ source $HEADAS/headas-init.sh

      2. Then activate their CIAO environment

        $ conda activate ciao-4.15

      3. Finally, users have to restore the HEADAS environment variable back to it's original value 

        $ export HEADAS=/soft/lheasoft/headas/x86_64-pc-linux

      Note: this may cause problems using XSpec models in sherpa. Users must not source the headas-init.sh script after activating the CIAO environment as it will cause incompatibilities in the suite of parameters tools: pset, pget, and plist.

  4. Users must not activate CIAO conda environment in the same terminal (shell) where the classic CIAO setup script (eg ciao.sh) has already been sourced. Similarly, users must not source the CIAO setup scripts in an activate CIAO conda environment. The results in both cases are corrupted environments which can lead to unexpected failures.

History

12 Nov 2020 Moved detailed conda instructions into own thread.
01 Mar 2021 Added new section on Alternative download options.
07 Mar 2021 Added some FAQs
08 Dec 2021 CIAO 4.14 updates: CIAO now requires using conda-forge channel.
10 Jan 2022 Added MARX information. Also added info about setting CALDB environment variables.
13 Apr 2022 The default version of python is subject to change without notice based on the availability of other packages on the conda-forge channel.
27 Jan 2023 Updated for CIAO 4.15.1 patch.
07 Mar 2023 Tweak for force python 3.10. Added CONDA_SUBDIR info.
Return to Threads Page: Top | All | Intro
/ ciao / threads / ciao_install_conda /