Last modified: 27 Jan 2023

URL: https://cxc.cfa.harvard.edu/ciao/threads/ciao_build_mac/

Rebuilding CIAO 4.15.2 source with pre-built OTS binaries on mac

CIAO 4.15 Science Threads


Overview

Synopsis:

The following instructions let you build CIAO 4.15.2 from source on mac systems, but assume you are using pre-built binaries for the OTS (off-the-shelf) software. With these directions, you can rebuild CIAO entirely, or rebuild selected tools in the source code.

Related Links:

Last Update: 27 Jan 2023 - Updated for CIAO 4.15.1 patch.


Contents


A few notes before you start

[CAUTION]
Missing header files with newer versions of Xcode

Newer versions of Xcode, which are available on newer versions of macOS (including Catalina, BigSur, and Monterey), have changes to the command line compiler tools. These changes break many existing build scripts as the location of the standard header files, eg stdio.h, have moved (or are not installed at all). Users will need to ensure that they have a working set of command line compilers to build CIAO from source. On the Mac, CIAO has only been tested when built with the Xcode/clang C/C++ compilers; not with GNU gcc/g++.

While there is not single set of instructions that seems to work for all users, the following steps have been shown to help most users get their command line tool compilers to work:

  1. Sign Xcode agreement. Just start Xcode and agree to license agreement.
  2. Install Xcode command line tools.

    $ xcode-select -install
    
  3. Point to correct include files. Add the following to your profile startup script. For bash|zsh
    $ export SDKROOT=`xcrun --show-sdk-path`
    or
    $ export CPATH=`xcrun --show-sdk-path`/usr/include
    

    or for tcsh

    % setenv SDKROOT `xcrun --show-sdk-path`
    or
    % setenv CPATH `xcrun --show-sdk-path`/usr/include
    
[CAUTION]
No support for native M1|ARM architecture

At the current time, we do not support native builds on the Mac M1|ARM architecture. We only support source builds on Intel|AMD systems.

These instructions show how to build CIAO using the ciao-install edition, not the conda edition.

To prevent any old variables from affecting the CIAO build environment, it is advised to open a new terminal window before continuing.

Where should CIAO be installed?

CIAO can be installed in any location and does not need to be installed with any root or super-user privileges. Users are discouraged from installing CIAO as the root user.


Off-the-shelf software

The CIAO off-the-shelf packages include adapted OTS and some dependencies on specific versions of OTS. As such, it is highly beneficial (and will save you a lot of headaches) to build CIAO with our OTS binaries. If you do need to build CIAO completely from source, contact the CXC Helpdesk.

You can view the list of the OTS and their versions here.


CIAO Source Build Quickstart

This section contains a quick start guide on how to build the CIAO system. The remainder of this document provides detailed instructions and explanations of the steps to build CIAO from source.

For bash|zsh shell users:

$ bash ./ciao-install \
  --add tools_src --add core_src --add sherpa_src --add obsvis_src \
  --batch --download `pwd` --prefix `pwd` 

$ cd ciao-4.15
$ /bin/rm -rf bin binexe lib include data doc share param

$ PATH=`pwd`/ots/bin:$PATH
$ ./configure --with-fits --with-ascii --with-opt=-O3
$ make -k install 2>&1 | tee make.log

$ bash bin/ciao-python-fix

$ source bin/ciao.sh
$ ahelp -r
$ cd test
$ make

For tcsh shell users:

% bash ./ciao-install \
  --add tools_src --add core_src --add sherpa_src --add obsvis_src \
  --batch --download `pwd` --prefix `pwd` 
% cd ciao-4.15

% /bin/rm -rf bin binexe lib include data doc share param

% set path = ( `pwd`/ots/bin $path )
% ./configure --with-fits --with-ascii --with-opt=-O3
% make -k install |& tee make.log

% bash bin/ciao-python-fix

% source bin/ciao.csh
% ahelp -r
% cd test
% make

Briefly described:

  1. Make sure that you have Apple's Xcode installed and have agreed to it's license. In addition you will need to have flex or lexx and bison or yacc installed.

  2. Download and unpack CIAO binaries with the source code using ciao-install; cd into the ciao-4.15 directory

    $ bash ./ciao-install \
      --add tools_src --add core_src --add sherpa_src --add obsvis_src \
      --batch --download `pwd` --prefix `pwd` 
    $ cd ciao-4.15
    
  3. Delete the contents of the installation directories:

    $ /bin/rm -rf bin binexe lib include data doc share param
    

    When re-building selected directories, rather than all of CIAO, then this step should be skipped.

  4. Add the CIAO ots/bin directory to your path

    bash$ PATH=`pwd`/ots/bin:$PATH
    tcsh% % set path = ( `pwd`/ots/bin $path )
    
  5. Run the configure script

    $ ./configure --with-fits --with-ascii --with-opt=-O3
    
  6. Build CIAO using make

    bash$ make -k install 2>&1 | tee make.log
    tcsh% make -k install |& tee make.log
    

    Or if you only want to rebuild individual directories, cd into them and make install from there.

  7. (optional) You can precompile all the python modules using the ciao-python-fix script

    $ bash bin/ciao-python-fix 
    
  8. Recreate the ahelp index files:

    bash$ source bin/ciao.sh
    tcsh% source bin/ciao.csh
    
    $ ahelp -r
    
  9. Run the CIAO smoke tests

    $ cd test
    $ make
    

Requirements

Operating Systems

Building CIAO with the pre-built OTS is supported on 64-bit mac systems. We do not support native builds on M1/ARM architecture.

Note: we also provide instructions for building CIAO with the pre-built OTS on Linux


Compilers

Building CIAO on the mac requires the clang/Xcode compilers. We have not tested building CIAO with the GNU compilers (gcc,g++).

Check to see if you have GCC on your system:

% cc --version
% c++ --version

If either gives you an error, you will have to download and install that compiler.

CIAO is built using Makefiles, so the make utility must be installed.

CIAO also requires flex|lex and yacc|bison to compile various parsers. These can be installed using your system's package manager.


Source files + OTS binaries

Building CIAO from source using the pre-built binary OTS requires the ciao-4.15-src-*.tar.gz packages as well as the platform specific ciao-4.15-bin-macOS.tar.gz binaries.

First, install CIAO using the ciao-install script. Go to the Custom Installation, choose your platform leave the other options at their default values (which is the "Standard CIAO installation"), and then change the "Source build" option to "Full Source Build". This will select all of the necessary binaries (which contain the OTS) and will download the source code. CALDB, the background event files and the science scripts package are optional; include or exclude these packages by checking the boxes next to them. Finally, click the "Download the ciao-install Script" button.

From a terminal, run the ciao-install script, which will unpack all the files for you [recommended].

% bash <path-to>/ciao-install

If you want to unpack the files yourself, use the --download-only switch:

% bash <path-to>/ciao-install --download-only

and then unpack the tar.gz packages. For example, the commands for unpacking the "Standard CIAO installation" mac 64-bit packages are shown below.

% tar -xzf ciao-4.15-bin-core-macOS.tar.gz
% tar -xzf ciao-4.15-bin-obsvis-macOS.tar.gz
% tar -xzf ciao-4.15-bin-sherpa-macOS.tar.gz
% tar -xzf ciao-4.15-bin-tools-macOS.tar.gz
% tar -xzf ciao-4.15-contrib-3.tar.gz
% tar -xzf ciao-4.15-src-core.tar.gz
% tar -xzf ciao-4.15-src-obsvis.tar.gz
% tar -xzf ciao-4.15-src-sherpa.tar.gz
% tar -xzf ciao-4.15-src-tools.tar.gz
% tar -xzf caldb_4.10.4_main.tar.gz

Please note that the list of files to untar depends on the selections made when downloading the ciao-install script.

If installing CALDB with --download-only switch

Unpacking caldb_4.10.4_main.tar.gz will create directories data/, docs/, and software/. You will need to create a CALDB directory and move data/, docs/ and software/ into CALDB/, then softlink CALDB to CIAO after the build.

In the directory containing caldb_4.10.4_main.tar.gz:

% mkdir CALDB
% mv docs/ data/ software/ CALDB/

Note: ciao-4.15-bin-obsvis-<plat>.tar.gz is not required as no OTS are packaged in that binary distribution.

Note: If you are having trouble with the ciao-install script, refer to "Downloading the Software" in Manually Installing CIAO.


Build Configuration

Before building CIAO, we must configure the build script for our system. This involves choosing which compilers to use, setting library locations, and adding other options to the build. All of this is handled by a configuration script.

Setting the compilers

Specifying a specific path to a compiler can be achieved by using the appropriate keyword:

  • CC - Name and location of the c compiler
  • CXX - Name and location of c++ compiler

Usage example:

% ./configure --with-ascii --with-fits \
  CC=<path-to>/bin/cc \
  CXX=<path-to>/bin/c++

The compiler must also be in your PATH. If <path-to-compiler> is not in $PATH, add it using

Bash
% export PATH=${PATH}:<path-to-compiler>
Tcsh/Csh
% setenv PATH ${PATH}:<path-to-compiler>

Configuration flags/options

Here is the list of configure options available for CIAO:

Option Description
--prefix=<location> Directory to install CIAO. Default location will be '.'
--x-includes=<DIR> directory DIR in which the X include files are located
--x-libraries=<DIR> directory DIR in which the X library files are located
--with-debug Compile with debug mode enabled
--with-opt=<level> Set optimization level. The default should be -O3 however, this does not currently seem to be the case.
--with-fits Include FITS kernel when building CXC DataModel.
Note: This switch is required to build the DataModel. It indicates that the setup is for a compile and not just a binary install.
--with-ascii Include the ASCII kernel when building the CXC DataModel.
--with-imagerdir=<saord> Path to directory/tree containing DS9

The flags --with-ascii and --with-fits are REQUIRED for CIAO to build correctly. All other flags are optional.


Optimization and compilers

Optimization is turned on by default in the binary distribution but not when building from source. Users should use the default value, --with-opt=-O3. If you wish to use a different optimization level or turn off optimization, use the --with-opt= switch. To turn off optimization, use --with-opt=-O0.


Build Instructions

Now, the source tree should be properly set up to build. The directions here are for rebuilding the entire CIAO source code (Rebuilding CIAO), or for rebuilding a selected directory (Rebuilding selected source code).

Rebuilding CIAO

To build and install the code, type "make install". CIAO currently requires that "make install" be issued as a single command rather than as separate commands "make; make test; make install". This is due to build dependencies on installed libraries and packages.

% make install

Note that this will take a few minutes.


Rebuilding selected source code

There are a few tools that depend on local libraries that if not installed may require rebuilding first. Most src directories can be rebuilt after running configure by simply stepping into the directory and running "make install".

For example, say a user rebuilt libfftw to take advantage of their specific hardware. After placing the libfftw files in ots/lib, they'd need to re-link the tools in src/da to their system by doing

% cd src/da
% make install

to rebuild the da tools with their custom libfftw build.


Post-processing

After the build completes, run the script ciao-python-fix:

% bash bin/ciao-python-fix

This post-processes your configuration, to byte-compile included Python files for your installation, and to ensure all CIAO packages with Python interfaces can be built. Once this script is run, the CIAO installation cannot be moved or relocated to a different directory without repeating the configure and ciao-python-fix steps.

Linking to CALDB: if you downloaded CALDB and unpacked the tarfile by hand (e.g. ran ciao-install with the --download-only switch), you can link it to CIAO at this point by creating a softlink to the CALDB directory in $ASCDS_INSTALL:

% pwd
/<path-to>/ciao-4.15
% ln -s <path-to>/CALDB CALDB

AHELP Indexing

After installation you will need to generate an index file used by the CIAO help documentation application (ahelp). The ciao.*sh script needs to run to set up dynamic library paths used by ahelp.

Bash
% source bin/ciao.sh -o
Tcsh/Csh
% source bin/ciao.csh -o

Then run the ahelp command with the -r (re-index) option.

% ahelp -r

This will generate the help indexes. If this completes successfully, you can execute the smoke tests.


Smoke Tests

To run the tests, you should do something like (source-ing the CIAO start up file is only needed if it has not already been done in the terminal):

Bash
% cd test
% source ../bin/ciao.sh
% make -k test 2>&1 | tee ~/ciao_test.log
Tcsh/Csh
% cd test
% source ../bin/ciao.csh
% make -k test |& tee ~/ciao_test.log

A successful test will look like:

        [1/42] Running test FOO-smoke001 ... PASS

where 'FOO' is the name of an individual test. If you see a message that reports:

        [1/42] Running test FOO-smoke001 ... FAIL

then there is an unexpected problem with the tests. Note that if you see additional warnings/errors and the test reports PASS, then the test is good (tests sometimes do test error conditions).

All the test outputs are written to $ASCDS_WORK_PATH/smoke.$LOGNAME where $LOGNAME is your system login name. The tests need about 60 Mb to run. The test scripts clean-up the disk space upon successful completion. Tests which fail leave their test output so you may check the results. When all of the tests PASS, the smoke test scripts should remove $ASCDS_WORK_PATH/smoke.$LOGNAME and all its sub-directories.

Note

$ASCDS_WORK_PATH is set in the $ASCDS_INSTALL/bin/ciao.*sh setup scripts as /tmp by default. If the machine does not have a /tmp or you don't have write permission, you will need to edit the appropriate ciao.*sh script to change the $ASCDS_WORK_PATH location or set the $ASCDS_WORK_PATH environment variable. If this variable is set, ciao.*sh will not modify it.

If a test FAILS, you may want to double-check that you are using the correct versions of GCC and gfortran, and that your environment variables $PATH is set correctly. Also check Bugs: Installation & Smoke Tests for other documented problems and solutions; if you run into any other problems, please contact the CXC Helpdesk. If you decide to re-run the build process, repeat the configuration and building process in a new terminal to prevent the old environment from affecting the new build.


Troubleshooting

Below are some common problems you may run into while building and installing CIAO. You can also check Bugs: Installation & Smoke Tests for other documented problems and solutions. If you run into any other problems, please contact the CXC Helpdesk.

Configure script

  1. For other errors, check that:

    • your compilers are in your $PATH
    • ${ASCDS_INSTALL}/ots/bin is in your $PATH

Build

If the build fails at any point during "make install", check that

  1. you have compatible C/C++ compilers (see Compilers for details)
  2. your compilers are in your $PATH.
  3. ${ASCDS_INSTALL}/ots/bin is in your $PATH.

Then, run "make -k clean", then remove the contents of lib/ and binexe/ again,

% rm -rf binexe/* lib/*

re-configure the build with

Bash
% source bin/ciao.sh -o
Tcsh/Csh
% source bin/ciao.csh -o

and then

% ./configure --with-fits --with-ascii

and run make install again. If it still fails, try opening-up a new terminal so that you have a clean environment to work with. Check that you have compatible compilers and that you have set your $PATH variable correctly defined.


History

08 Dec 2014 First issue. Merges INSTALL_SOURCE text file into CIAO website.
15 Dec 2015 Updated instructions for ciao 4.8 release
07 Jun 2016 Added the fixpc command to Step 6.
28 Nov 2016 Updated instructions for ciao 4.9 release
09 Apr 2018 Updates for CIAO 4.10.
03 Apr 2018 Updates for CIAO 4.11
09 Dec 2019 Updates for CIAO 4.12
30 Nov 2021 Updates for CIAO 4.14. With GTK removed, the instructions are greatly simplified.
27 Jan 2023 Updated for CIAO 4.15.1 patch.