Skip to the navigation links
Last modified: 13 March 2019

URL: http://cxc.harvard.edu/ciao/scripting/index.html

Using Python with CIAO


The CIAO software offers Python version 3.5.4 as the user scripting environment, as described in "ahelp python". The software package is distributed with CIAO, but users may opt to run their own installation instead; see "ahelp ciaorc" for information on how to change the Python version that is being used. Note that there is limited support for using an external Python system in CIAO, and it is suggested that users install external Python packages into CIAO rather than the other way around.

Although there have been a number of changes to support Python package installation in CIAO 4.11, not all use cases are supported. For example, the Conda and Anaconda package managers are not supported in CIAO 4.11.

CIAO provides a number of Python modules that support data analysis using CIAO in Python, with the main ones being:

crates

The Python interface to the CIAO Data Model. This module supports reading and writing FITS and ASCII files. The transform module provides support for coordinate conversion (linear and WCS) for tables and images.

Sherpa

The CIAO modeling and fitting package.

ChIPS

The CIAO imaging and plotting package. Note that as of CIAO 4.11, development of ChIPS has halted, and users are encouraged to move to using the Matplotlib plotting package (which is also provided as part of CIAO). The ChIPS site contains a ChIPS to Matplotlib conversion guide.


What Python packages are provided with CIAO?

The pip3 tool can be used to report on the Python packages installed in CIAO. For the CIAO 4.11 Linux build this reports (where the major packages are shown in bold):

unix% pip3 list --format=columns
Package            Version 
------------------ --------
attrs              17.3.0  
backcall           0.1.0   
bleach             2.1.4   
caldb4             0.1.0   
cycler             0.10.0  
Cython             0.25.2  
decorator          4.3.0   
defusedxml         0.5.0   
entrypoints        0.2.3   
html5lib           1.0.1   
ipykernel          4.9.0   
ipython            6.5.0   
ipython-genutils   0.2.0   
ipywidgets         7.4.1   
jedi               0.12.1  
Jinja2             2.10    
jsonschema         2.6.0   
jupyter            1.0.0   
jupyter-client     5.2.3   
jupyter-console    5.2.0   
jupyter-core       4.4.0   
kiwisolver         1.0.1   
MarkupSafe         1.0     
matplotlib         2.2.3   
mistune            0.8.3   
nbconvert          5.4.0   
nbformat           4.4.0   
notebook           5.6.0   
numpy              1.12.1  
pandocfilters      1.4.2   
parso              0.3.1   
pexpect            4.6.0   
pickleshare        0.7.4   
pip                9.0.1   
pixlib             1.0.0   
pluggy             0.6.0   
prometheus-client  0.3.1   
prompt-toolkit     1.0.15  
ptyprocess         0.6.0   
py                 1.6.0   
Pygments           2.2.0   
pyparsing          2.2.0   
pytest             3.3.1   
python-dateutil    2.7.3   
pytz               2018.5  
pyzmq              17.1.2  
qtconsole          4.4.1   
Send2Trash         1.5.0   
setuptools         28.8.0  
sherpa             ciao4.11
simplegeneric      0.8.1   
six                1.10.0  
terminado          0.8.1   
testpath           0.3.1   
tornado            5.1     
traitlets          4.3.2   
wcwidth            0.1.7   
webencodings       0.5.1   
widgetsnbextension 3.4.1   

Not shown in the above is the message about how the version of pip included in CIAO 4.11 is out of date and could be upgraded.


Do I need to add SSL support for CIAO?

The Python packaging tool pip3, which is used to add packages to CIAO, requires support for SSL, since the web sites it retrieves information from are "secure" (i.e. https rather than http).

If, after starting CIAO, the following command returns without any errors then the following text can be ignored and you can jump to the installation section.

unix% python -m ssl

If the command returns an error, then version 1.0 of the openssl library needs to be installed on your system, ideally using a package manager - such as apt-get, yum, macports, or brew - since the SSL libraries need to be installed in "system accessible" locations.


How do I install a Python package into CIAO?

New to CIAO 4.11 is support for installing Python packages into CIAO with the pip3 tool. Please contact the CXC Helpdesk if you have problems, as support for this should still be considered experimental. The following requires that you have write access to the CIAO installation directory. If this fails you can try installing with the --user flag.

Information on using a "Python virtual environment" with CIAO will be added to this guide soon, as this is an alternative way to use external Python packages with CIAO.

[WARNING]
Ensure you have the original CIAO installation files

Installing Python packages into the CIAO installation area runs the risk of corrupting the CIAO system. The main concern is accidentally upgrading (or downgrading) a component that CIAO relies on (in particular NumPy. It is strongly suggested that you ensure you have the files that ciao-install downloaded when installing CIAO, so that CIAO can be re-installed if there is a problem.

How to avoid updating NumPy

The version of NumPy provided with CIAO 4.11, that is version 1.12.1, is quite old. Installing recent versions of Python packages - such as AstroPy or PyMC3 - may try to upgrade the installed NumPy. This is currently unsupported, so we suggest that a constraints file is used with pip3 to ensure that NumPy remains at version 1.12.1.

It is safest to not update any Python package provided with CIAO, which can be achieved with the freeze option:

unix% pip3 freeze > $ASCDS_INSTALL/constraints.txt
unix% grep numpy $ASCDS_INSTALL/constraints.txt
numpy==1.12.1

or you can just create a file with this - e.g.

unix% echo "numpy==1.12.1" > $ASCDS_INSTALL/constraints.txt

The location of the file isn't that important, but placing it in $ASCDS_INSTALL means that you have it stored in a memorable location!

This file is then used in a pip3 install line with the -c flag, as will be shown below. Note that not all packages require this set of constraints, but it's hard to know before hand, so it is simplest to always use it.

Example: installing corner

As an example, the following set of routines will install the corner package, which is used to visualize multidimensional samples (in particular the results of Markov Chain Monte Carlo simulations). This package was picked as it is small, Python only, and does not require installing any extra code. Note that the screen output the exact output depends on the version available at the time of installation, and details of the CIAO installation, so may differ slightly from that shown here):

unix% pip3 install -c $ASCDS_INSTALL/constraints.txt corner
Collecting corner
  Downloading https://files.pythonhosted.org/packages/65/af/a7ba022f2d5787f51db91b5550cbe8e8c40a6eebd8f15119e743a09a9c19/corner-2.0.1.tar.gz
Requirement already satisfied: numpy==1.12.1 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.11/constraints.txt (line 29))
Requirement already satisfied: matplotlib==2.2.3 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.11/constraints.txt (line 24))
Requirement already satisfied: six==1.10.0 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.11/constraints.txt (line 50))
Requirement already satisfied: python-dateutil==2.7.3 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.11/constraints.txt (line 43))
Requirement already satisfied: cycler==0.10.0 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.11/constraints.txt (line 5))
Requirement already satisfied: kiwisolver==1.0.1 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.11/constraints.txt (line 22))
Requirement already satisfied: pytz==2018.5 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.11/constraints.txt (line 44))
Requirement already satisfied: pyparsing==2.2.0 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.11/constraints.txt (line 41))
Requirement already satisfied: setuptools in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from kiwisolver==1.0.1->-c /soft/ciao-4.11/constraints.txt (line 22))
Installing collected packages: corner
  Running setup.py install for corner ... done
Successfully installed corner-2.0.1

As a final check we see whether we can import the module (the name used in the import statement depends on how the package is designed and need not match the installation name):

unix% python -c 'import corner'

Example: installing AstroPy and SciPy

[NOTE]
Note

AstroPy version 3.1 requires NumPy version 1.13 but will install even if NumPy 1.12 is on your system. To avoid this, we have to restrict the AstroPy version to 3.0, which is the last version to support NumPy 1.12.

Following the previous example, we can install AstroPy and SciPy by saying:

unix% pip3 install -c $ASCDS_INSTALL/constraints.txt 'astropy<3.1' scipy
Collecting astropy<3.1
  Using cached https://files.pythonhosted.org/packages/36/f2/8ebc112247fec6e507127abbdd4e0734995337d289e6e47a36604806a1e8/astropy-3.0.5-cp35-cp35m-manylinux1_x86_64.whl
Collecting scipy
  Downloading https://files.pythonhosted.org/packages/f0/30/526bee2ce18c066f9ff13ba89603f6c2b96c9fd406b57a21a7ba14bf5679/scipy-1.2.1-cp35-cp35m-manylinux1_x86_64.whl (24.7MB)
Requirement already satisfied: numpy==1.12.1 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from -c /soft/ciao-4.11/constraints.txt (line 29))
Installing collected packages: astropy, scipy
Successfully installed astropy-3.0.5 scipy-1.2.1

Problems when compiling code

The steps above work when the module is "pure Python", such as the corner package, or needs minimal compilation on the local machine (e.g. AstroPy). There can be problems when installing modules which require access to system-level - or CIAO-provided - code. As an example, installing pygsl can fail as it is unable to find GSL include files, due to the presence of paths which begin with /proj/port_ots/. Please contact the CXC Helpdesk as we may be able to help in these situations.

Have we broken anything?

It is possible to get into a state where incompatible packages have been installed, so it is worth periodically checking your CIAO installation with the following command:

unix% pip3 check
No broken requirements found.

If there is a problem then it may be possible to remove or change the version of the affected package, or packages, or it may require re-installing CIAO.


About Python

Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools and comes with extensive standard libraries. A list of key software features is available on the About Python webpage.

Note that CIAO 4.11 is the first release to only provide a version with Python 3.5 support, since support for Python 2.7 in the scientific-software ecosystem is coming to an end.