Using Python with CIAO

CIAO provides a number of Python modules that support data analysis using CIAO in Python, with the main ones being:

crates: The Python interface to the CIAO Data Model. This module supports reading and writing FITS and ASCII files. The transform module provides support for coordinate conversion (linear and WCS) for tables and images.
Sherpa: The CIAO modeling and fitting package.

More information on Python support in CIAO can be found in "ahelp python", and there is limited customization support as described in users may opt to run their own installation instead; see "ahelp ciaorc".

Note

We strongly suggest that users install Python packages into their CIAO environment, rather than trying to install CIAO into their Python environment.

What Python packages are provided with CIAO?

The Python packages installed with CIAO depends on how CIAO was installed - whether ciao-install or the conda package manager was used - as well as what packages were installed and, for the conda version, the version of Python.

conda

When using conda, the ciaover command lists the "base" CIAO packages that have been installed (in the example below the CIAO CALDB has not been installed):

unix% ciaover -v
The current environment is configured for:
  CIAO        : CIAO 4.16.0 Tuesday, December 05, 2023
  Contrib     : Package release 0  Tuesday, November 28, 2023
  bindir      : /soft/ciao-4.16/bin
  Python path : /soft/ciao-4.16/bin
  CALDB       : 4.11.0

System information:
Linux is.awesome.edu 4.18.0-477.13.1.el8_8.x86_64 #1 SMP Thu May 18 10:27:05 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

A full list of packages can be shown with conda list and pip3 list. The output depends on the OS in use (Linux or macOS), what version of Python was selected and what other packages are installed.

ciao-install

The pip3 tool can be used to report on the Python packages installed in CIAO. For the CIAO 4.16 Linux build this reports (where the major packages are shown in bold):

unix% pip3 list --format=columns | grep -E 'matplotlib|numpy|pip|python'
ipython                   8.18.1
matplotlib                3.8.2
matplotlib-inline         0.1.6
numpy                     1.26.2
pip                       23.3.1
python-dateutil           2.8.2
python-json-logger        2.0.7
types-python-dateutil     2.8.19.14

It is likely that you will see a note about pip being out of date, as it changes rapidly!

How do I install a Python package into CIAO?

The method for installing Python packages depends on how CIAO was installed. The recommended approach is to use a conda installation of CIAO.

Updating versions

CIAO depends on a number of libraries and modules (e.g. NumPy), and it has only been tested against the versions available at the time of release. The CXC does not guarantee that CIAO will work correctly if these packages are updated.

Please take care when installing software alongside CIAO as it can result in updating these dependencies.

conda

As you are within a standard conda environment, packages can be installed with conda or pip3.

Conflicts with astroconda

The CIAO installation provides a number of packages that may conflict with those from AstroConda, in particular DS9 and cfitsio. At the time of writing (December 2020), the difference in cfitsio package is known to cause problems. It is suggesttd that at this time CIAO not be combined with AstroConda in the same environment.

ciao-install

For users with CIAO installed via ciao-install, packages should be installed with the pip3 tool, and it requires that you have write access to the CIAO installation directory. If this fails you can try installing with the --user flag.

Ensure you have the original CIAO installation files

Installing Python packages into the CIAO installation area runs the risk of corrupting the CIAO system. The main concern is accidentally upgrading (or downgrading) a component that CIAO relies on (in particular NumPy. It is strongly suggested that you ensure you have the files that ciao-install downloaded when installing CIAO, so that CIAO can be re-installed if there is a problem.

Example: installing corner

As an example, the following set of routines will install the corner package, which is used to visualize multidimensional samples (in particular the results of Markov Chain Monte Carlo simulations). This package was picked as it is small, Python only, and does not require installing any extra code. Note that the screen output the exact output depends on the version available at the time of installation, and details of the CIAO installation, so may differ slightly from that shown here):

unix% pip3 install -c $ASCDS_INSTALL/constraints.txt corner
Collecting corner
  Using cached corner-2.1.0-py2.py3-none-any.whl (11 kB)
Requirement already satisfied: setuptools>=40.6.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from corner) (41.2.0)
Requirement already satisfied: matplotlib==3.2.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 25)) (3.2.0)
Collecting wheel
  Downloading wheel-0.36.2-py2.py3-none-any.whl (35 kB)
Collecting setuptools-scm
  Using cached setuptools_scm-5.0.1-py2.py3-none-any.whl (28 kB)
Requirement already satisfied: numpy==1.18.1 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 31)) (1.18.1)
Requirement already satisfied: python-dateutil==2.8.1 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 47)) (2.8.1)
Requirement already satisfied: kiwisolver==1.2.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 23)) (1.2.0)
Requirement already satisfied: cycler==0.10.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 6)) (0.10.0)
Requirement already satisfied: pyparsing==2.4.7 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 44)) (2.4.7)
Requirement already satisfied: six==1.14.0 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 53)) (1.14.0)
Installing collected packages: wheel, setuptools-scm, corner
Successfully installed corner-2.1.0 setuptools-scm-5.0.1 wheel-0.36.2

As a final check we see whether we can import the module (the name used in the import statement depends on how the package is designed and need not match the installation name):

unix% python -c 'import corner'

Example: installing AstroPy and SciPy

Following the previous example, we can install AstroPy and SciPy by saying:

unix% pip3 install -c $ASCDS_INSTALL/constraints.txt astropy scipy
Collecting astropy
  Downloading astropy-4.2-cp37-cp37m-manylinux1_x86_64.whl (9.7 MB)
     |################################| 9.7 MB 1.8 MB/s
Collecting scipy
  Downloading scipy-1.5.4-cp37-cp37m-manylinux1_x86_64.whl (25.9 MB)
     |################################| 25.9 MB 8.0 MB/s
Collecting pyerfa
  Downloading pyerfa-1.7.1.1-cp37-cp37m-manylinux2010_x86_64.whl (696 kB)
     |################################| 696 kB 16.5 MB/s
Requirement already satisfied: numpy==1.18.1 in /soft/ciao-4.13/ots/lib/python3.7/site-packages (from -c /soft/ciao-4.13/constraints.txt (line 31)) (1.18.1)
Installing collected packages: pyerfa, astropy, scipy
Successfully installed astropy-4.2 pyerfa-1.7.1.1 scipy-1.5.4

Problems when compiling code

The steps above work when the module is "pure Python", such as the corner package, or needs minimal compilation on the local machine (e.g. AstroPy). There can be problems when installing modules which require access to system-level - or CIAO-provided - code. Please contact the CXC Helpdesk as we may be able to help in these situations.

Have we broken anything?

It is possible to get into a state where incompatible packages have been installed, so it is worth periodically checking your CIAO installation with the following command:

unix% pip3 check
No broken requirements found.

If there is a problem then it may be possible to remove or change the version of the affected package, or packages, or it may require re-installing CIAO.

About Python

Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools and comes with extensive standard libraries. A list of key software features is available on the About Python webpage.

/ ciao / scripting /