Skip to the navigation links
Last modified: 8 December 2021

URL: https://cxc.cfa.harvard.edu/sherpa/index.html

CIAO's modeling and fitting package


Sherpa is the CIAO modeling and fitting application. It enables the user to construct complex models from simple definitions and fit those models to data, using a variety of statistics and optimization methods (see the Gallery of Examples).

CIAO 4.14

Sherpa version for CIAO 4.14 was released on December 14, 2021. Sherpa in CIAO runs under Python 3.9, 3.8, 3.7 (when installed using the conda package manager) and Python 3.8 (when installed with ciao-install). The full list of the Sherpa updates is given in the 4.14.0 and 4.13.1 release notes on GitHub. The major updates in this release include:

XSPEC

Updated to version 12.12.0 of the XSPEC model library, from version 12.10.1s in CIAO 4.13. Added support for the new models: additive xsgrbjet, xswdem, xsvwdem, xsvvwdem and multiplicative xszxipab.

There are changes to the hard limits of some models, the default parameter values and units, including adding and removing units. Also, XSPEC model parameters now use the same limits for soft and hard (the hard limit from the model.dat file) and the hard limits can be changed for some models which support a value outside the normal parameter range. It is strongly suggested that any parameter set to a value outside of the original limits is also frozen.

New abundance tables (lpgp, lpgs) from Lodders et al, (2009) are added and can be selected with set_xsabund

XSPEC ETABLE table model files can be read by load_xstable_model by setting the etable parameter.

Changed the default XSPEC chatter level from 0 to 10. This may result in screen output the first time an XSPEC model is evaluated. set_xschatter can be used to get back to the previous behavior.

Filtering and grouping

Filtering and grouping area for binned (1D) spectral data has been improved with changes to the default behavior and many bug fixes resulting in changes to the statistics, degrees-of-freedom and energy flux in comparison to the previous version for the same data with the same filter. Highlights include:

Energy and wavelength filters (with notice and ignore) for PHA data are now treated as lo <= x < hi in all cases, and channel filters are lo <= x <= hi (where the channel values must be integers). For Data1DInt cases the notice and ignore filters are treated as lo <= x < hi. This can result in slightly different fit results because the data used in the fit can be changed and hence changing the number of degrees of freedom (by one or two depending on whether the first and last bins have been removed).

Filter expressions for Data1DInt and DataPHA cases now report the start and end value of each set of grouped data rather than use the end points. For PHA data sets users now see the same ranges displayed whether the data is grouped or not. The actual filter remains the same even if the filter expressions has changed slightly.

Updated the image filters: "|" is used now instead of & to combined the regions. This affects the output of get_filter as input to calc_data_sum2d and definitions of very-complex spatial filters. Restoring Sherpa sessions could be impacted.

There are also many bug fixes in this area.

Plotting improvements

CIAO 4.14 continued the improvements to the plotting functionality, and includes the following:

  • plot_model_component and get_model_component add the response for PHA data sets, if there is one. Model expressions which contain a response are not changed.
  • Corrected the order of the points, x, and y values when displaying a CDFPlot object, improve support for a list argument.
  • Bug fixes to plot_source, reg_proj, reg_unc and multi-plots.
  • Internal changes to the plotting code and documentation.
Flux calculations

There are changes to sample_flux which now:

  • uses the id argument rather than always using the data from the default dataset.
  • returns a statistic value for each iteration, even if those rows are not used in the reported flux distribution.
  • returns correct information for the clip column (that is, it matches the clipping done by this routine). There may be changes to the reported error ranges because of this change.
  • no-longer excludes samples at the parameter bounds (soft) when calculating the flux distribution. This could lead to an over-estimation of the flux for upper limits.

The How do we calculate a flux in Sherpa Jupyter notebook has been updated.

Optimization methods

Updated on how the optimization minim will behave if the free parameter is beyond the limit. The default parameter reflect is True (will reflect by an equal amount about the limit). If reflect is set to False then model function will return DBL_MAX (~1e308) and therefore it will not be included in the simplex.

Updated the internal documentation of the methods.

Documentation

The Sherpa ahelp files have been updated to match the Python docstrings. Each command now has its own ahelp file, rather than combining multiple commands into a single file.

Sherpa lets you:
  • fit 1-D data sets (simultaneously or individually), including:
    spectra, surface brightness profiles, light curves, general ASCII arrays;

  • fit 2-D images/surfaces in the Poisson/Gaussian regime;

  • visualize the data with Matplotlib and DS9;

  • access the internal data arrays;

  • build complex model expressions;

  • import and use your own models;

  • choose appropriate statistics for modeling Poisson or Gaussian data;

  • import new statistics, with priors if required by analysis;

  • visualize a parameter space with simulations or using 1-D/2-D cuts of
    the parameter space;

  • calculate confidence levels on the best-fit model parameters;

  • choose a robust optimization method for the fit: Levenberg-Marquardt,
    Nelder-Mead Simplex or Monte Carlo/Differential Evolution;

  • perform Bayesian analysis with Poisson Likelihood and priors, using
    Metropolis or Metropolis-Hastings algorithm in the MCMC (Markov-Chain Monte Carlo);

  • and use Python to create complex analysis and modeling functions,
    build the batch mode analysis or extend the provided functionality
    to meet the required needs.

[] []
[] []
[] []

The Sherpa infrastructure greatly enhances the default Sherpa functions, and provides users with an environment for developing complex and sophisticated analysis.

Sherpa is designed for use in a variety of modes: as a user-interactive application and in batch mode. Sherpa is an importable module for the Python the scripting language. In addition, users may write their own Python scripts for use in Sherpa.

The About Sherpa page outlines key features of the software, and the Latest Updates page describes new functionality and recent changes. See also the SciPy2009 conference proceedings for detailed information about Sherpa's capabilities.

If you have ideas about how to enhance or improve Sherpa, please contribute ideas (and code) to the Sherpa GitHub repository.

Please send feedback and questions on Sherpa to the CXC Helpdesk or the Sherpa Issues list on GitHub.


Citing Sherpa in a Publication

New in CIAO 4.13 is the sherpa.citation method which will return infromation from the latest release on Zenodo:

sherpa> sherpa.citation('latest')

If you are writing a paper and would like to cite Sherpa, we recommend the following papers and presentations:

Sherpa: a mission-independent data analysis application (ADS)
P. E. Freeman, S. Doe, A. Siemiginowska
SPIE Proceedings, Vol. 4477, p.76, 2001

\bibitem[Freeman et al.(2001)]{2001SPIE.4477...76F} Freeman, P., Doe, S., 
\& Siemiginowska, A.\ 2001, \procspie, 4477, 76

The specific version of CIAO and CALDB (if applicable) used for the analysis should be mentioned as well.

A reference for the Python interface to Sherpa is also available:

Developing Sherpa with Python (ADS)
S. Doe, et al.
Astronomical Data Analysis Software and Systems XVI, 376, 543

\bibitem[Doe et al.(2007)]{2007ASPC..376..543D} Doe, S., et al.\ 2007, 
Astronomical Data Analysis Software and Systems XVI, 376, 543

Sherpa: 1D/2D modeling and fitting in Python (SciPy 2009)
B. Refsdal, S. Doe, D. Nguyen, A. Siemiginowska, N. Bonaventura, D. Burke, I. Evans, J. Evans, A. Fruscione, E. Galle, J. Houck, M. Karovska, N. Lee, M. Nowak
Proceedings of the 8th Python in Science Conference (SciPy 2009), G. Varoquaux, S. van der Walt, J. Millman (Eds.), pp. 51-57 2009

Fitting and Estimating Parameter Confidence Limits with Sherpa (SciPy 2011),
B. Refsdal, S. Doe, D. Nguyen, A. Siemiginowska, V. Kashyap
Proceedings of the 19th Python in Science Conference (SciPy 2011), S. van der Walt, J. Millman (Eds.), pp. 4-10 2011

Further guidelines are available from the Acknowledgment of Use of Chandra Resources.