Chandra X-Ray Observatory
	(CXC)
Skip to the navigation links
Last modified: 16 December 2016

URL: http://cxc.harvard.edu/ciao/releasenotes/ciao_4.9_release.html

CIAO 4.9 Release Notes


CIAO 4.9 is distributed for the following platforms:

  • Linux 64 bit (CentOS 6.8 / Red Hat Enterprise 6)
  • Linux 64 bit (Ubuntu 14.04)
  • Apple OS X 10.9 (Mavericks)
  • Apple OS X 10.10 (Yosemite)
  • Apple OS X 10.11 (El Capitan)
  • Apple macOS 10.12 (Sierra)

Users can install CIAO with either Python 2.7 or with Python 3.5. Both Python 2.7 and Python 3.5 versions can be installed but must be installed into separate directories.

CIAO is no longer available for 32bit Linux operating systems or for older 64bit Linux machines (CentOS 5 era). CIAO is also no longer available for older versions of OSX (notably 10.6, 10.7, nor 10.8). CIAO 4.7 is still available for users who are unable to upgrade. More details on the Platform Support page.

Notable changes and improvements in CIAO 4.9:

  • CIAO pre-built binaries are now provided for a larger number of operating systems: Linux Fedora based systems (such as RedHat, CentOS, Scientific Linux), Linux Ubuntu based systems (such as LinuxMint), and individually for each Apple release: Mavericks, Yosemite, ElCapitan, and Sierra.
  • Beta support for Python 3.5. CIAO have been updated to work with either Python 2.7 or with Python 3.5.

  • This is primarily a maintenance release: bug fixes, supporting new compilers and OTS upgrades.

  • CIAO includes version 7.5 of SAOImage ds9. Users are reminded that they now need to change the Edit mode setting in order to select or create regions in recent versions of ds9. Please see the watchout page for more information on this.

  • Sherpa has seen: improvements to the WStat statistic, save_all function, and internal documentation (the Python docstrings); minor bug fixes; and updated to support Python 3.5. The XSPEC models have been updated to version 12.9.0o. As a reminder, Sherpa is also available as a stand alone system, accessible from the Sherpa GitHub repository or from the standalone Sherpa page.

  • Several tools have been updated. This includes adding support for table as inputs to dmellipse and expanded comparisons using dmdiff. There have also been various bug fixes.

  • There have been several new scripts released since the CIAO 4.8 release, including: blanksky, blanksky_image, correct_periscope_drift, and simulate_psf.



How CALDB 4.7.3 Affects Your Analysis

CALDB 4.7.3 Release Notes (release 15 December 2016)

ACIS Imaging and Grating Data

  • ACIS QE Contamination Model vN0010

    The ACIS QE contamination model has been upgraded to N0010:

    acisD1999-08-13contamN0010.fits 
    

    This version of the file is necessary due to recent monitoring and modeling of the optical depth of the contaminant on the optical blocking filter, requiring a short-term update for the ACIS contamination model—with updated contamination spatial and temporal models—included in the CalDB.

    The new contamination model provides improved fits to standard extended source spectra with stable photoabsorption and and other fitted parameters of their systematic models.

    While the new model is applicable to all ACIS observations throughout the mission, intended to replace all previous versions of the contamination model, the new model will have no significant effect on fitting results for observations preceeding 2009 and there is minimal change between the N0009 and N0010 models before 2005. However, the contamination is particularly noticeable near and below the Oxygen K-edge at 0.535 keV—and essentially gone above 2.0 keV—so the contamination will affect modeling results for most any low-energy source observed after 2005.

    It is therefore suggested that users doing—or having done—spectral analysis of observations with earlier ACIS contamination models, where a significant portion of the spectra are below 1.0 keV, may have an interest in repeating studies with the new contamination model applied to the ARFs/GARFs.

    More information about the N0010 contamination model, and prior versions, can be found in the ACIS QE contamination model Why document. A detailed presentation of the derivation of the new model—and the resulting changes in the estimated time-dependent effective area of ACIS-I and ACIS-S—is in the technical details section of the CALDB 4.7.3 Release Notes.

    These CIAO response tools automatically apply the contamination file when creating ACIS response files:

    As well as the scripts which use them:

  • Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

    The new, time-dependent ACIS Gain (T_GAIN) files for February-April 2016 (Epoch 65) and May-July 2016 (Epoch 66) are introduced and updated for Epoch 64 in this release, and therefore, affects observations taken since 2015 November 01. Other observations are unaffected by these new calibration files.

    Users working with ACIS data taken since 2015 November 01 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.

    See the time-dependent ACIS T_GAIN why page for more information.

    The gain corrections are at the nominal and expected levels of less than 2% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.

    Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.

HRC Imaging Data

  • HRC-I Gain Maps

    The latest time-dependent HRC-I Gain Map (GMAP) and the corresponding PI background spectrum—to be applied to observations after 2016 September 20, have been released.

    Note that HRC-I gain maps are only useful to observers doing hardness ratio or PI spectral analysis studies with HRC-I.

    As the HRC gain continues to slowly decline, the gain maps are updated annually. The change in gain correction is less than 1% on-axis and roughly 5-10% off-axis. Towards the edge of the microchannel plate, the change is up to 15%. Technical details on the gain maps are described in the gain maps section of the HRC calibration website and the in the HRC-I Gain Map Why Document.

    The corresponding background spectrum applicable to the same HRC-I GMAP time period have been generated.

    Users working with HRC-I data may wish run chandra_repro to reprocess the data and improve the gain calibration.


Installation

Users should be aware of these installation items before installing CIAO 4.9. Additional problems which are seen less frequently are listed on the Installation & Smoke Tests bug page.

Supported Platforms

  • CIAO 4.9 is supported on:

    • Linux 64 bit (CentOS 6.8 / Red Hat Enterprise 6)
    • Linux 64 bit (Ubuntu 14.04)
    • Apple OS X 10.9 (Mavericks)
    • Apple OS X 10.10 (Yosemite)
    • Apple OS X 10.11 (El Capitan)
    • Apple macOS 10.12 (Sierra)

    Users can install CIAO with either Python 2.7 or with Python 3.5; however, you must not install both into the same directory.

Updates to the ciao-install Installation Script

  • Users can select the Python3.5 version of CIAO using the new --python3 flag

    $ bash ./Downloads/ciao-install --python3
    
    There have been numerous updates to the install script to handle the new operating systems as well to support both Python 2.7 and 3.5 installations.

  • Some OSX users reported problems downloading the CIAO tar files. This was frequently tracked to the ciao-install script using an incompatible version of ftp. The script now uses /usr/bin/ftp to avoid this problem.

IPython Settings

  • Users will be prompted to update their IPython profiles the first time they run sherpa and chips.

    Users will also be prompted to update their IPython profiles whenever they switch between the Python2.7 and Python 3.5 builds.

Remove old parameter files

  • With every new CIAO release, some parameter files are changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:

    unix% rm ~/cxcds_param4/*
    

Changes to Dynamic/Shared Libraries

  • CIAO tools and applications are built using many different software libraries. As these libraries are common to most of the tools, they are compiled such that different programs can dynamically load the libraries; the libraries are shared by the programs.

    The mechanism used by the programs to locate the shared libraries has changed in CIAO 4.9. The programs are now compiled with the relative location of the libraries built into the executable itself. This means that the programs no longer require the use of the LD_LIBRARY_PATH (Linux) or DYLD_LIBRARY_PATH (OSX) environment variable to locate these libraries. This change was required for supporting OSX 10.11 (El Capitan) and macOS 10.12 (Sierra) but has now been made to each of the CIAO builds.

    Users who have purposefully overwritten the (DY)LD_LIBRARY_PATH environment variable or who have programs that link against the CIAO libraries (including libraries in the CIAO off-the-shelf, OTS, directory) may need to make changes to their configuration.


Python 3

CIAO 4.9 includes changes that allow the CIAO packages and scripts to run with Python 2.7 and Python 3.5. Users can install either or both versions of CIAO for the specific platform. If installing both, then they must be installed into separate directories.

Most of the Python packages and scripts have been modified to accommodate this upgrade.

The list of applications updated includes

The list of libraries and modules updated includes


Tools

acis_find_afterglow

  • Internal cleanup to support different compilers.

acis_process_events

  • Update for TDET coordinate calculations in CC mode to set to NULL value when times are invalid.

  • Updated PHA_RO (read-out) values for data which has not been CTI corrected.

acis_streak_map

  • Now can exclude sources along the read-out streak from the region using the new ssigma parameter.

  • Fix bug when infile has 0 events

axbary

  • Internal cleanup to support different compilers.

dates

  • Added 2016 leap second.

dmdiff

  • Fix problem when comments include a "%" character

  • Better checking of range() tolerance values

  • Better checking for vector (ie virtual) columns (eg "sky"). Now compares the individual physical column names and reports differences in ranges for each physical column separately.

  • Image units are now compared

  • Multiple transforms (WCS) on a column are now compared separately.

  • Fix bug when comparing vector and physical columns

  • The string representation of regions are now compared

  • Improved checking for structural differences when comparing tables. Including tables with 0 rows and different number of columns (including 0 columns).

  • Corrects display of byte data-type value differances

  • The ignorepath tolerance option has been extended to also ignore comments

  • Other internal code changes.

dmellipse

  • Now supports using tables as input. The new xcol, ycol, and zcol parameters specify the column names in the table for the coordinates (x,y) and optionally the per-row weighting (analogous to the pixel value in an image).

  • Various internal cleanup.

dmextract

  • Fix for binning on columns with long names (> 16 characters) and columns with long descriptions/comments.

  • Catch and report problems when exposure (exp) values are invalid (missing columns or non-existent files).

dmhistory

  • Show history for partial tool name matches

  • Update to handling of ARDLIB and PIXLIB history records.

dmhedit

  • Fix bug when comments were uninitialized

dmimgcalc

  • return error status when directory/file does not exist.

dmmakereg

  • Corrects the string identifying the units of the rotation angle; now correctly written as "deg".

dmtype2split

  • Bug fix for invalid output file names.

glvary

  • Fix problem when TSTART equals TSTOP. Now a 0 row output file is created.

mkwarf

  • Adjusts low energy cutoff used when no FEF file is supplied. (Needed to match HRC RMF energy grid).

tgextract2

  • Corrects the problem where the HRC+LETG CALDB bow-tie regionfile was not being used.

wcs_update

  • Can now use be used to update an arbitrary FITS extension in the infile.

  • The quaternion values in the aspect solution file are now also updated.

  • Error message handling improvements.


Parameter Files

A summary of parameter files changes is provided in this section. Refer to the Tools and Analysis Scripts sections of these release notes for complete details.

We recommend deleting all the old parameter files or renaming the parameter-file directory before running any new version of CIAO to avoid conflicts. More information is available in the FAQ.

acis_streak_map

  • A new ssigma parameter has been added to specify a limit for excluding sources from a readout streak.

dmellipse

  • dmellipse can now work with tables as well as images. The new xcol, ycol, and zcol parameters specify the column names to use for the coordinates and weighting (ie pixel value).


ChIPS

The following changes are in addition to changes to support Python 3.5:

  • Fixed crash on OSX when adding data with transforms associated with it. For example adding an image with a wcs to chips would cause it to crahs. The issue is specific to OSX.

  • Fixed a bug that would cause a crash after adding a frame to an existing plot when the gui window is displayed. This is specific to OSX and the gui window must be displayed.

  • A user's preferences are now loaded when running python as well as the chips application. Updated ChIPS smoke tests so that the user preferences don't interfere with the smoke test results


Sherpa

Sherpa is developed as a community project, and is available on GitHub. Comments, fixes, and additions from the community are welcomed. There are additional releases for this version throughout the year, and can be installed either directly from GitHub or using the conda package manager, as described on the standalone Sherpa page.

Statistics

  • There have been a number of fixes to the statistic and fitting code that address issues with the WStat statistic, including:

    • Fix problems that could arise when the data was filtered or grouped.

    • The BACKSCAL value was not being handled correctly when it was an array, rather than a scalar, as can happen with grating data.

    • The calc_stat_info function was failing when the WStat statistic was selected.

    • The statistics information returned by calc_stat_info and get_stat_info did not include information about the reduced statistic (rstat) or Q value (qval) when wstat was selected.

    • The calc_stat function was not exiting with an error message when a likelihood statistic - such as Cash, CStat, or WStat - is selected and the background has been subtracted.

Documentation

  • There have been a number of documentation updates, either improving or adding to existing docstrings. These include:

    • Documentation for the models has been added. So after

      sherpa> set_source(xsphabs.gal * xsvapec.src)
      

      help on the model components can be found using

      sherpa> help(gal)
      sherpa> help(src)
      
    • Use the correct tolerance value when discussing the optimisation routines (e.g. LevMar).

    • Miscellaneous updates to correct missing or incorrect information, including group_snr, group_adapt_snr, set_rmf, and set_quality.

    [NOTE]
    Sherpa ahelp in CIAO 4.9

    The ahelp documentation for Sherpa has not been updated to match changes in CIAO 4.9, so it is suggested that the Python help system be used; that is use the help command in Sherpa rather than ahelp (this command is not available from the Unix command line, only from Sherpa, ChIPS, or ipython prompts).

XSpec

  • The XSPEC models have been updated to version 12.9.0o. The nlapec model is not available, but it can be emulated with the xsapec model by setting the XSPEC XSET variable APECNOLINES to off; for example:

    sherpa> set_source(xsphabs.gal * xsapec.src)
    sherpa> set_xsxset('APECNOLINES', 'on')
    

    although note that this is a global setting, and so will affect all APEC model instances in the Sherpa session.

Python 3.5 support

  • The Sherpa code base is now compatible with both Python 2.7 and 3.5. This work included code clean ups to replace deprecated functionality from packages such as NumPy. There are several known differences with the Python 3.5 version in this release:

    1. Numbers displayed in the Python 3.5 version will generally show more decimal places than the Python 2.7 version.

    2. Several functions, such as image_getregion and image_xpaget, return byte strings rather than strings in Python 3.5. The decode method can be used to convert them to strings, as shown below:

      sherpa> image_open()
      sherpa> image_xpaget('version')
              b'sherpa 7.5\n'
      sherpa> image_xpaget('version').decode()
              'sherpa 7.5\n'
      
    3. The order of datasets may be different to the Python 2.7 version; for example, when load_pha is given a PHA2 file, or when fitting multiple data sets. This does not affect the fit results, but may be surprising if the results from Sherpa are expected to be in a certain order.

Bug fixes

  • The updates in this version of Sherpa are listed below.

    Improvements and fixes to the save_all function, including:

    • The source expression should now be saved; in previous versions it could sometimes be missed from the output file. Similarly, improvements to storing the background model associated with a PHA data set have been made.

    • There have been improvements to the handling of grouping and quality arrays of PHA data sets: they are now written out as integer values and the values for background data sets are now also stored.

    • Data sets created with the load_arrays function are now written out. This is intended for "small" data sets. It is suggested that FITS files are used for large arrays of if the numeric values may be significantly affected by conversion to text format (i.e. a potential loss in precision).

    • Sessions which use the load_psf function can now be saved when multiple data sets are in use; in previous versions some information may have been lost.

    • When possible, user models created with load_user_model and add_user_pars are now saved to the file. This may not be possible, in which case a place-holder function is used which will display a warning message when the file is evaluated.

      It may still be necessary to edit the file to include necessary imports or set up code for the user model.

    • The script now uses dataset-appropriate functions: for example load_pha or load_image rather than always use load_data, and create_model_component is preferred to eval. Optional arguments to functions are now explicitly named rather than relying on position.

    • The clobber argument now behaves as it does with CIAO tools, in that an existing output file is deleted. Previously it would be appended to.

    • The output file now explicitly imports the sherpa.astro.ui module so that the file can be run from IPython sessions (such as the sherpa environment) with the %run command:

      sherpa> %run bestfit.py
      

      or directly from python, e.g.

      unix% python bestfit.py
      
    • The outfile argument has been added to the save_all function. It can accept any file-like argument such as a file handle or a StringIO object.

    The function load_xstable_model should be used to create an XSPEC table model for additive and multiplicative models (atable and mtable). Support for these models is still available in load_table_model but is now deprecated.

    The covar_matrix argument has been added to the get_draws function, which allows a user-provided covariance matrix to be used.

    Minor change to the axis labels created by the plot_source function for PHA data sets, to better match the other plot types.

    The plot_photon_flux function no-longer errors out with the message:

    TypeError: scales option must be iterable of length ...
    

    Fix the issue where the image_data function would fail with the message

    DS9Err: Could not display image
    

    when the working directory contained a file called x or y.


Graphical User Interfaces

There are no changes to prism, peg, nor to taskmonitor in this release beyond those needed to support Python 3.5.

Analysis Scripts

Many of the scripts have been updated to work with Python 3.5. The list includes

Also included in this list are new scripts added to the contributed package after the release of CIAO 4.8.

specextract

  • The behavior has been changed when bkgresp=yes so that the calculated background ARF will always be weighted. The background RMF type is still determined by the unweight_rmf parameter.

blanksky

  • [New] Script to create an unscaled blanksky background file compatible with a given observation events file. The background scaling factors for each chip are calculated and stored in the BKGSCALE/BKGSCALn header keywords.

blanksky_image

  • [New] Script to create scaled background and background-subtracted images given an observation-specific blanksky background file and a reference image.

correct_periscope_drift

  • [New] Script to correct small (~0.1 arcsec) intra-observation alignment drifts that can be seen in recent, long (>50 ks) observations. Requires a bright, on-axis (<2 arcmin) point source with minimal pileup.

simulate_psf

  • [New] A simplified interface to MARX, to perform both raytrace PSF simulations or projecting ChaRT/SAOTrace rayfiles to the detector-plane for an existing observation.

Python Modules

CIAO modules are now compabile with both Python v 2.7.11 and v3.5. Additional updates are noted below.

crates

  • Add a new add_comments routine.
  • when adding image object to IMAGECreate, also add subspace records for X and Y axes.
  • generate warning for data columns of unknown type. no CrateData object is created for these columns.
  • returning single row bit array column as 2D
  • Enable writing of NULL primary image with uint8 type. Improved error message when write_block encounters None value pointer. NOTE: Converts to int2 FITS type (BITPIX=16)

pypixlib

  • Improve search logic for obtaining underlying C library cannot rely on (DY)LD_LIBRARY_PATH since this has been deprecated in Mac ElCapitan build.

pyTransform

  • wrapper updates for string* elements, removes cross-interpretation as char**

Libraries

cxcparam

  • Additional error checking for long string values

  • Internal cleanup to support different compilers.

datamodel

  • Fixes output physical transform definition when filtering images that have no specific physical coordinate system defined.

  • Introduces changes to better handle the case where a region filter is entirely outside the boundary of the image.

mtllib

  • Internal cleanup

pixlib

  • Internal cleanup

region

  • Introduces changes to how region bounds are computed when the region is outside the boundary of the dataset.

Environment

Off-the-Shelf (OTS) Package Versions

  • The following OTS packages are included with CIAO 4.9. For more information on how the OTS packages are built for use with CIAO, refer to the INSTALL_SOURCE file distributed with the software.

    Executables

    • ds9 7.5

    Python

    • either Python 2.7.11 or, as a beta release, Python 3.5.1
      Frameworks on OSX have been disabled to avoid a problem with users picking up the wrong version of NumPy.
    • NumPy 1.11.0
    • ipython 4.2.0
    • six 1.10.0

    Libraries

    • cfitsio v3.380
    • readline 6.3
    • wcssubs 3.9.0
    • XSPEC v12.9.0o (models only)
    • VTK 5.10.1
    • CCfits 2.5
    • fontconfig 2.8.0
    • freetype 2.4.4
    • gsl 2.1
    • vte 0.25.1
    • fftw 3.3.4
    • xpa 2.1.15
    • jpeg 9a
  • The ciaover command will now report the version of the Calibration Database as well as the version of the Analysis Scripts package. In addition the full path to the Python executable is shown (even when using the precompiled CIAO version).

Compiler versions

  • The following versions of the gcc compiler were used to compile the binary CIAO releases:
    System Build System Version
    Linux64 CentOS 6 gcc 4.4.7
    LinuxU Ubuntu gcc 4.8.4
    osxm Mavericks llvm 6.0
    osxy Yosemite llvm 7.0.2
    osxElCap ElCapitan llvm 7.0.3
    osxSierra Sierra llvm 8.0.0

Documentation

Many pages have been updated to reflect the changes in this release. Several analysis threads are still under review.


Last modified: 16 December 2016
Smithsonian Institute Smithsonian Institute

The Chandra X-Ray Center (CXC) is operated for NASA by the Smithsonian Astrophysical Observatory. 60 Garden Street, Cambridge, MA 02138 USA.   Email:   cxcweb@head.cfa.harvard.edu Smithsonian Institution, Copyright © 1998-2016. All rights reserved.