Chandra X-Ray Observatory
	(CXC)
Skip to the navigation links
Last modified: 28 Nov 2016

URL: http://cxc.harvard.edu/ciao/threads/ciao_build_linux/

Rebuilding CIAO 4.9 source with pre-built OTS binaries on Linux

CIAO 4.9 Science Threads


Overview

Synopsis:

CIAO depends on a number of external software components we refer to as OTS (off-the-shelf) software. The following instructions let you build CIAO 4.9 from source on Linux systems, but assume you are using pre-built binaries for the OTS software. With these directions, you can rebuild CIAO entirely, or rebuild selected tools in the source code.

Related Links:

Last Update: 28 Nov 2016 - Updated instructions for ciao 4.9 release


Contents


A few notes before you start

To prevent any old variables from affecting the CIAO build environment, it is advised to open a new terminal window before continuing.

Where should CIAO be installed?

CIAO can be installed in any location and does not need to be installed with any root or super-user privileges.


Off-the-shelf software

The CIAO off-the-shelf packages include adapted OTS and some dependencies on specific versions of OTS. For example, we use a modified version of VTK for CIAO, as well as a specific snapshot of gtkglext. As such, it is highly beneficial (and will save you a lot of headaches) to build CIAO with our OTS binaries. If you do need to build CIAO completely from source, contact the CXC Helpdesk.

You can view the list of the OTS and their versions here.


CIAO Source Build Quickstart

This section contains a quick start guide on how to build the CIAO system. The remainder of this document provides detailed instructions and explanations of the steps to build CIAO from source. If you are only rebuilding selected directories, skip step 3.

  1. Download GCC and gfortran, if not already installed.

  2. Download and unpack CIAO binaries and source code via ciao-install; cd into ciao-4.9

    % cd <path-to>/ciao-4.9/
    
  3. When re-building selected directories, rather than all of CIAO, then this step should be skipped.

    Delete the contents of the lib/ and binexe/ directories.

    % rm -rf binexe/* lib/*
    
  4. Set $PKG_CONFIG_PATH to include paths to `pwd`/ots/lib/pkgconfig and the path to your system pkgconfig:

    Bash

    For Redhat 6 based systems:

    % export PKG_CONFIG_PATH=`pwd`/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig
    

    For Ubuntu based systems:

    % export PKG_CONFIG_PATH=`pwd`/ots/lib/pkgconfig:/usr/lib/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:/usr/share/pkgconfig
    
    Tcsh/Csh

    For Redhat 6 based systems:

    % setenv PKG_CONFIG_PATH `pwd`/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig
    

    For Ubuntu based systems:

    % setenv  PKG_CONFIG_PATH `pwd`/ots/lib/pkgconfig:/usr/lib/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:/usr/share/pkgconfig
    
  5. Add ots/bin to your search path.

    Bash
    % export PATH=`pwd`/ots/bin:$PATH
    
    Tcsh/Csh
    % setenv PATH `pwd`/ots/bin:$PATH
    
  6. Run fixpc.sh and then configure the build.
    % bash src/config/fixpc.sh `pwd`
    % ./configure --with-fits --with-ascii
    
  7. Run make install; when re-building all of CIAO

    % make install |& tee build.log
    

    but when only rebuilding a certain tool, cd into that tool's directory and run make install there.

  8. Generate the indexes needed for the ahelp system. This requires running CIAO setup.

    Bash
    % . bin/ciao.sh
    
    Tcsh/Csh
    % source bin/ciao.csh
    

    Now the ahelp indexes can be created and the pre-compiled python modules:

    % ahelp -r
    % bash bin/ciao-python-fix
    
  9. Run smoke tests:

    % cd test
    
    Bash
    % make 2>&1 | tee ~/ciao_test.log
    
    Tcsh/Csh
    % make |& tee ~/ciao_test.log
    

Requirements

Operating Systems

Building CIAO with the pre-built OTS is supported on 64-bit Linux systems.

Note: we also provide instructions for building CIAO with the pre-built OTS on Mac OS X


Compilers

Building CIAO requires C, C++ and FORTRAN compilers. We strongly recommend using GCC and gfortran. Building CIAO on Linux system has been tested with GCC and gfortran versions 4.4.7, 4.8.x and 5.2 but should work with any current version of GCC.

[WARNING]
Issue building CALDB with GCC 6.2.1 on Fedora Core 25

There have been issues reported building the caldb code with GCC 6.2.1 packaged with Fedora Core 25. We are looking into this.

Check to see if you have gfortran and GCC on your system:

% gcc --version
% gfortran --version

If either gives you an error, you will have to download and install that compiler.


Source files + OTS binaries

Building CIAO from source using the pre-built binary OTS requires the ciao-4.9-src-*.tar.gz packages as well as the platform specific ciao-4.9-bin-Linux64.tar.gz binaries.

First, install CIAO using the ciao-install script. Go to the Downloading CIAO page. Under Custom Installation, choose your platform leave the other options at their default values (which is the "Standard CIAO installation"), and then change the "Source build" option to "Full Source Build". This will select all of the necessary binaries (which contain the OTS) and will download the source code. CALDB, the background event files and the science scripts package are optional; include or exclude these packages by checking the boxes next to them. Finally, click the "Download the ciao-install Script" button.

From a terminal, run the ciao-install script, which will unpack all the files for you [recommended].

% bash <path-to>/ciao-install

If you want to unpack the files yourself, use the --download-only switch:

% bash <path-to>/ciao-install --download-only

and then unpack the tar.gz packages. For example, the commands for unpacking the "Standard CIAO installation" Linux 64-bit packages are shown below.

% tar -xzf ciao-4.9-bin-chips-Linux64.tar.gz
% tar -xzf ciao-4.9-bin-core-Linux64.tar.gz
% tar -xzf ciao-4.9-bin-graphics-Linux64.tar.gz
% tar -xzf ciao-4.9-bin-obsvis-Linux64.tar.gz
% tar -xzf ciao-4.9-bin-prism-Linux64.tar.gz
% tar -xzf ciao-4.9-bin-sherpa-Linux64.tar.gz
% tar -xzf ciao-4.9-bin-tools-Linux64.tar.gz
% tar -xzf ciao-4.9-contrib-1.tar.gz
% tar -xzf ciao-4.9-src-chips.tar.gz
% tar -xzf ciao-4.9-src-core.tar.gz
% tar -xzf ciao-4.9-src-obsvis.tar.gz
% tar -xzf ciao-4.9-src-prism.tar.gz
% tar -xzf ciao-4.9-src-sherpa.tar.gz
% tar -xzf ciao-4.9-src-tools.tar.gz
% tar -xzf caldb_4.7.2_main.tar.gz

Please note that the list of files to untar depends on the selections made when downloading the ciao-install script.

If installing CALDB with --download-only switch

Unpacking caldb_4.7.2_main.tar.gz will create directories data/, docs/, and software/. You will need to create a CALDB directory and move data/, docs/ and software/ into CALDB/, then softlink CALDB to CIAO after the build.

In the directory containing caldb_4.7.2_main.tar.gz:

% mkdir CALDB
% mv docs/ data/ software/ CALDB/

Note: ciao-4.9-bin-obsvis-<plat>.tar.gz is NOT needed as no OTS are packaged in that binary distribution.

Note: If you are having trouble with the ciao-install script, refer to "Downloading the Software" in Manually Installing CIAO.


Build Preparation

  1. Untarring the ciao-4.9-*.tar.gz files creates a directory ciao-4.9; cd into it:

    % cd ciao-4.9
    

    You will make many references to the ciao-4.9 directory during the build configuration. Creating an environment variable for the path to ciao-4.9 will make the build commands easier:

    Bash
    % export ASCDS_INSTALL=`pwd`
    
    Tcsh/Csh
    % setenv ASCDS_INSTALL `pwd`
    
    % echo $ASCDS_INSTALL
    /<path-to>/ciao-4.9
    

    If you only want to rebuild certain directories of the CIAO source code, skip to step (3).

  2. To make sure that libraries and executables are newly created by the source build on your system, delete the contents of the lib/, and binexe/ directories:

    % rm -rf lib/* binexe/*
    
  3. CIAO uses GTK for chips and prism. The necessary GTK files are packaged with the CIAO graphics binary tarfile in the pkgconfig directory. We discovered that combining the system and CIAO GTK produces a more stable system than the CIAO GTK alone. Therefore, you must build with both the system and the CIAO GTK pkgconfig. To do this, set the following variable $PKG_CONFIG_PATH:

    Bash
    % export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig
    
    Tcsh/Csh
    % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig
    
    [NOTE]
    Note

    On Redhat based builds (CentOS 6), you need to add /usr/share/pkgconfig to $PKG_CONFIG_PATH:

    Bash
    % export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig
    
    Tcsh/Csh
    % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig
    

    On Ubuntu based builds, you need to add /usr/share/pkgconfig, /usr/lib/pkgconfig, and /usr/lib/x86_64-linux-gnu/pkgconfig to PKG_CONFIG_PATH:

    Bash
    % export PKG_CONFIG_PATH=${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:/usr/share/pkgconfig
    
    Tcsh/Csh
    % setenv PKG_CONFIG_PATH ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib/pkgconfig:/usr/lib/x86_64-linux-gnu/pkgconfig:/usr/share/pkgconfig
    
  4. Lastly, fixpc.sh must be run to fix the .pc configuration files in ${ASCDS_INSTALL}/ots/lib/pkgconfig. Also you must add ots/bin to your path so you will pick up the correct version of Python.

    % bash src/config/fixpc.sh `pwd`
    % export PATH=`pwd`/ots/bin:$PATH
    

Build Configuration

Before building CIAO, we must configure the build script for our system. This involves choosing which compilers to use, setting library locations, and adding other options to the build. All of this is handled by a configuration script.

Setting the compilers

Specifying a specific path to a compiler can be achieved by using the appropriate keyword:

  • CC - Name and location of the c compiler
  • CXX - Name and location of c++ compiler
  • FC - Name and location of FORTRAN compiler

Usage example:

% ./configure --with-ascii --with-fits \
  FC=<path-to>/bin/gfortran-4.8 \
  CC=<path-to>/bin/cc \
  CXX=<path-to>/bin/c++

To specify the FORTRAN compiler, the FC variable must be set.

The compiler must also be in your PATH. If <path-to-compiler> is not in $PATH, add it using

Bash
% export PATH=${PATH}:<path-to-compiler>
Tcsh/Csh
% setenv PATH ${PATH}:<path-to-compiler>

Configuration flags/options

Here is the list of configure options available for CIAO:

Option Description
--prefix=<location> Directory to install CIAO. Default location will be '.'
--x-includes=<DIR> directory DIR in which the X include files are located
--x-libraries=<DIR> directory DIR in which the X library files are located
--with-debug Compile with debug mode enabled
--with-opt=<level> Set optimization level (default=-O3)
--with-fits Include FITS kernel when building CXC DataModel.
Note: This switch is required to build the DataModel. It indicates that the setup is for a compile and not just a binary install.
--with-ascii Include the ASCII kernel when building the CXC DataModel.
Note: This switch is required if you need to read ASCII files into ChIPS.
--with-f95loc=<DIR> Location of libf95 (if using g95)
--with-gfortranloc=<DIR> Location of libgfortran (if using gfortran) Unnecessary if gfortran and libgfortran are in the same tree (e.g. if gfortran is in /usr/bin, and libgfortran is in /usr/lib, they’re in the same tree)
--with-imagerdir=<saord> Path to directory/tree containing DS9

The flags --with-ascii and --with-fits are REQUIRED for CIAO to build correctly. All other flags are optional.


Optimization and compilers

Optimization is turned on by default, -O3 for GCC. If you wish to use a different optimization level or turn off optimization, use the --with-opt= switch. To turn off optimization, use --with-opt=-O0. See GNU’s Optimization Options for more details.


Build Instructions

Now, the source tree should be properly set up to build. The directions here are for rebuilding the entire CIAO source code (Rebuilding CIAO), or for rebuilding a selected directory (Rebuilding selected source code).

Rebuilding CIAO

To build and install the code, type "make install". CIAO currently requires that "make install" be issued as a single command rather than as separate commands "make; make test; make install". This is due to build dependencies on installed libraries and packages.

% make install

Note that this will take a few minutes.


Rebuilding selected source code

There are a few tools that depend on local libraries that if not installed may require rebuilding first. Most src directories can be rebuilt after running configure by simply stepping into the directory and running "make install".

For example, say a user rebuilt libfftw to take advantage of their specific hardware. After placing the libfftw files in $ASCDS_INSTALL/ots/lib, they’d need to re-link the tools in $ASCDS_INSTALL/src/da to their system by doing

% cd ${ASCDS_INSTALL}/src/da
% ciaorun make install

to rebuild the da tools with their custom libfftw build.


Post-processing

After the build completes, run the script ciao-python-fix:

% bash bin/ciao-python-fix

This post-processes your configuration, to byte-compile included Python files for your installation, and to ensure all CIAO packages with Python interfaces can be built. Once this script is run, the CIAO installation cannot be moved or relocated to a different directory without repeating the configure and ciao-python-fix steps.

Linking to CALDB: if you downloaded CALDB and unpacked the tarfile by hand (e.g. ran ciao-install with the --download-only switch), you can link it to CIAO at this point by creating a softlink to the CALDB directory in $ASCDS_INSTALL:

% pwd
/<path-to>/ciao-4.9
% ln -s <path-to>/CALDB CALDB

AHELP Indexing

After installation you will need to generate an index file used by the CIAO help documentation application (ahelp). The ciao.*sh script needs to run to set up dynamic library paths used by ahelp.

Bash
% . bin/ciao.sh -o
Tcsh/Csh
% source bin/ciao.csh -o

Then run the ahelp command with the -r (re-index) option.

% ahelp -r

This will generate the help indexes. If this completes successfully, you can execute the smoke tests.


Smoke Tests

To run the tests, you should do something like (source-ing the CIAO start up file is only needed if it has not already been done in the terminal):

Bash
% cd test
% source ../bin/ciao.sh
% make -k test 2>&1 | tee ~/ciao_test.log
Tcsh/Csh
% cd test
% source ../bin/ciao.csh
% make -k test |& tee ~/ciao_test.log

A successful test will look like:

        [1/42] Running test FOO-smoke001 ... PASS

where 'FOO' is the name of an individual test. If you see a message that reports:

        [1/42] Running test FOO-smoke001 ... FAIL

then there is an unexpected problem with the tests. Note that if you see additional warnings/errors and the test reports PASS, then the test is good (tests sometimes do test error conditions).

All the test outputs are written to $ASCDS_WORK_PATH/smoke.$LOGNAME where $LOGNAME is your system login name. The tests need about 60 Mb to run. The test scripts clean-up the disk space upon successful completion. Tests which fail leave their test output so you may check the results. When all of the tests PASS, the smoke test scripts should remove $ASCDS_WORK_PATH/smoke.$LOGNAME and all its sub-directories.

Note

$ASCDS_WORK_PATH is set in the $ASCDS_INSTALL/bin/ciao.*sh setup scripts as /tmp by default. If the machine does not have a /tmp or you don't have write permission, you will need to edit the appropriate ciao.*sh script to change the $ASCDS_WORK_PATH location or set the $ASCDS_WORK_PATH environment variable. If this variable is set, ciao.*sh will not modify it.

If a test FAILS, you may want to double-check that you’re using the right versions of GCC and gfortran, and that your environment variables $PATH and $PKG_CONFIG_PATH are set correctly. Also check Bugs: Installation & Smoke Tests for other documented problems and solutions; if you run into any other problems, please contact the CXC Helpdesk. If you decide to re-run the build process, repeat the configuration and building process in a new terminal to prevent the old environment from affecting the new build.


Troubleshooting

Below are some common problems you may run into while building and installing CIAO. You can also check Bugs: Installation & Smoke Tests for other documented problems and solutions. If you run into any other problems, please contact the CXC Helpdesk.

Configure script

  1. Error "cfitsio is not found":

    checking for pkg-config... no
    configure: error: cfitsio is not found. Please specify the path using
        '--with-cfitsio=<<path>' or
        'setenv PKG_CONFIG_PATH <path>/lib/pkgconfig'
    

    This error shows that the $PKG_CONFIG_PATH variable is not set correctly. $PKG_CONFIG_PATH should contain

    ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig
    

    or - for CentOS 6:

    ${ASCDS_INSTALL}/ots/lib/pkgconfig:/usr/lib64/pkgconfig:/usr/share/pkgconfig   
    
  2. For other errors, check that:

    • your compilers are in your $PATH
    • $PKG_CONFIG_PATH is set correctly
    • ${ASCDS_INSTALL}/ots/bin is in your $PATH
    • If you need to change your $PKG_CONFIG_PATH you will also need to re-run configure.

Build

If the build fails at any point during "make install", check that

  1. you have compatible GCC and gfortran compilers (see Compilers for details)
  2. your compilers are in your $PATH.
  3. ${ASCDS_INSTALL}/ots/bin is in your $PATH.
  4. $PKG_CONFIG_PATH is set correctly

Then, run "make -k clean", then remove the contents of lib/ and binexe/ again,

% rm -rf binexe/* lib/*

re-configure the build with

Bash
% . bin/ciao.sh -o
Tcsh/Csh
% source bin/ciao.csh -o

and then

% ciaorun ./configure --with-fits --with-ascii

and run make install again. If it still fails, try opening-up a new terminal so that you have a clean environment to work with. Check that you have compatible compilers and that you’ve set $PKG_CONFIG_PATH and your $PATH variables correctly defined.


History

08 Dec 2014 First issue. Merges INSTALL_SOURCE text file into CIAO website.
15 Dec 2015 Updated instructions for ciao 4.8 release
07 Jun 2016 Added the fixpc command to Step 6.
28 Nov 2016 Updated instructions for ciao 4.9 release


Last modified: 28 Nov 2016
Smithsonian Institute Smithsonian Institute

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email:   cxchelp@head.cfa.harvard.edu Smithsonian Institution, Copyright © 1998-2017. All rights reserved.