Using Python with CIAO
- What Python packages are provided with CIAO? (13 Dec 2018)
- Do I need to add SSL support for CIAO? (13 Dec 2018)
- How do I install a Python package into CIAO? (13 Dec 2018)
- About Python
The CIAO software offers Python version 3.5.4 as the user scripting environment, as described in "ahelp python". The software package is distributed with CIAO, but users may opt to run their own installation instead; see "ahelp ciaorc" for information on how to change the Python version that is being used. Note that there is limited support for using an external Python system in CIAO, and it is suggested that users install external Python packages into CIAO rather than the other way around.
Although there have been a number of changes to support Python package installation in CIAO 4.11, not all use cases are supported. For example, the Conda and Anaconda package managers are not supported in CIAO 4.11.
CIAO provides a number of Python modules that support data analysis using CIAO in Python, with the main ones being:
The Python interface to the CIAO Data Model. This module supports reading and writing FITS and ASCII files. The transform module provides support for coordinate conversion (linear and WCS) for tables and images.
The CIAO modeling and fitting package.
The CIAO imaging and plotting package. Note that as of CIAO 4.11, development of ChIPS has halted, and users are encouraged to move to using the Matplotlib plotting package (which is also provided as part of CIAO). The ChIPS site contains a ChIPS to Matplotlib conversion guide.
What Python packages are provided with CIAO?
The pip3 tool can be used to report on the Python packages installed in CIAO. For the CIAO 4.11 Linux build this reports (where the major packages are shown in bold):
unix% pip3 list --format=columns Package Version ------------------ -------- attrs 17.3.0 backcall 0.1.0 bleach 2.1.4 caldb4 0.1.0 cycler 0.10.0 Cython 0.25.2 decorator 4.3.0 defusedxml 0.5.0 entrypoints 0.2.3 html5lib 1.0.1 ipykernel 4.9.0 ipython 6.5.0 ipython-genutils 0.2.0 ipywidgets 7.4.1 jedi 0.12.1 Jinja2 2.10 jsonschema 2.6.0 jupyter 1.0.0 jupyter-client 5.2.3 jupyter-console 5.2.0 jupyter-core 4.4.0 kiwisolver 1.0.1 MarkupSafe 1.0 matplotlib 2.2.3 mistune 0.8.3 nbconvert 5.4.0 nbformat 4.4.0 notebook 5.6.0 numpy 1.12.1 pandocfilters 1.4.2 parso 0.3.1 pexpect 4.6.0 pickleshare 0.7.4 pip 9.0.1 pixlib 1.0.0 pluggy 0.6.0 prometheus-client 0.3.1 prompt-toolkit 1.0.15 ptyprocess 0.6.0 py 1.6.0 Pygments 2.2.0 pyparsing 2.2.0 pytest 3.3.1 python-dateutil 2.7.3 pytz 2018.5 pyzmq 17.1.2 qtconsole 4.4.1 Send2Trash 1.5.0 setuptools 28.8.0 sherpa ciao4.11 simplegeneric 0.8.1 six 1.10.0 terminado 0.8.1 testpath 0.3.1 tornado 5.1 traitlets 4.3.2 wcwidth 0.1.7 webencodings 0.5.1 widgetsnbextension 3.4.1
Not shown in the above is the message about how the version of pip included in CIAO 4.11 is out of date and could be upgraded.
Do I need to add SSL support for CIAO?
The Python packaging tool pip3, which is used to add packages to CIAO, requires support for SSL, since the web sites it retrieves information from are "secure" (i.e. https rather than http).
If, after starting CIAO, the following command returns without any errors then the following text can be ignored and you can jump to the installation section.
unix% python -m ssl
If the command returns an error, then version 1.0 of the openssl library needs to be installed on your system, ideally using a package manager - such as apt-get, yum, macports, or brew - since the SSL libraries need to be installed in "system accessible" locations.
How do I install a Python package into CIAO?
New to CIAO 4.11 is support for installing Python packages into CIAO with the pip3 tool. Please contact the CXC Helpdesk if you have problems, as support for this should still be considered experimental. The following requires that you have write access to the CIAO installation directory. If this fails you can try installing with the --user flag.
Information on using a "Python virtual environment" with CIAO will be added to this guide soon, as this is an alternative way to use external Python packages with CIAO.
Installing Python packages into the CIAO installation area runs the risk of corrupting the CIAO system. The main concern is accidentally upgrading (or downgrading) a component that CIAO relies on (in particular NumPy. It is strongly suggested that you ensure you have the files that ciao-install downloaded when installing CIAO, so that CIAO can be re-installed if there is a problem.
As an example, the following set of routines will install the corner package, which is used to visualize multidimensional samples (in particular the results of Markov Chain Monte Carlo simulations). This package was picked as it is small, Pythoon only, and does not require installing any extra code.
First we check that there are no problems with the potential installation (that is, there are no version conflicts):
unix% pip3 check corner No broken requirements found.
As this was successful, we now install the package (note that the exact output depends on the version available at the time of installation):
unix% pip3 install corner Collecting corner Downloading https://files.pythonhosted.org/packages/65/af/a7ba022f2d5787f51db91b5550cbe8e8c40a6eebd8f15119e743a09a9c19/corner-2.0.1.tar.gz Requirement already satisfied: numpy in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from corner) Requirement already satisfied: matplotlib in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from corner) Requirement already satisfied: pytz in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from matplotlib->corner) Requirement already satisfied: six>=1.10 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from matplotlib->corner) Requirement already satisfied: python-dateutil>=2.1 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from matplotlib->corner) Requirement already satisfied: kiwisolver>=1.0.1 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from matplotlib->corner) Requirement already satisfied: cycler>=0.10 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from matplotlib->corner) Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from matplotlib->corner) Requirement already satisfied: setuptools in /soft/ciao-4.11/ots/lib/python3.5/site-packages (from kiwisolver>=1.0.1->matplotlib->corner) Installing collected packages: corner Running setup.py install for corner ... done Successfully installed corner-2.0.1
As a final check we see whether we can import the module (the name used in the import statement depends on how the package is designed and need not match the installation name):
unix% python -c 'import corner'
How to avoid updating NumPy
The version of NumPy provided with CIAO 4.11, that is version 1.12.1, is quite old. Installing recent versions of Python packages - such as AstroPy or PyMC3 - may try to upgrade the installed NumPy. This is currently unsupported, so we suggest that a constraints file is used with pip3 to ensure that NumPy remains at version 1.12.1.
It is safest to not update any Python package provided with CIAO, which can be achieved with the freeze option:
unix% pip3 freeze > constraints.txt unix% grep numpy constraints.txt numpy==1.12.1
or you can just create a file with this - e.g.
unix% echo "numpy==1.12.1" > constraints.txt
This file is then used in a pip3 install line by saying (here to install Astropy and SciPy):
unix% pip3 install -c constraints.txt astropy scipy
Problems when compiling code
The steps above work when the module is "pure Python", such as the corner package, or needs minimal compilation on the local machine (e.g. AstroPy). There can be problems when installing modules which require access to system-level - or CIAO-provided - code. As an example, installing pygsl can fail as it is unable to find GSL include files, due to the presence of paths which begin with /proj/port_ots/. Please contact the CXC Helpdesk as we may be able to help in these situations.
Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools and comes with extensive standard libraries. A list of key software features is available on the About Python webpage.
- Python Programming Language website
- Running CIAO tools from Python.
- AstroPython: Python for Astronomers
Note that CIAO 4.11 is the first release to only provide a version with Python 3.5 support, since support for Python 2.7 in the scientific-software ecosystem is coming to an end.