CalDB 4.9.2 Public Release Notes
Public Release Date: 09 July 2020 (with CalDB 126.96.36.199)
SDP Installation Date: 2020-07-06T17:01:07 (UTC)
CalDB 4.9.2 is an upgrade to the Chandra CalDB, which includes the following items:
- A FULL UPGRADE to the ACIS T_GAIN LIBRARY
- ACIS FP_TEMP Boundary Condition Updates (Effective 01 July 2020)
- HRC-S Time-dependent Gain Map (T_GMAP) Version 3
- HRC-S QE Version 15
- HRC-I QE Version 12
For the CIAO 4.12 / CalDB 4.9.2 release notes see How CalDB 188.8.131.52 Affects Your Analysis.
II. SUMMARY OF CHANGES
where "YYYY-MM-DD" in the N0008 and N0008B filenames above are the
Calibration Validity Start Dates (CVSD*) values in the Block 2 FITS
HDUs of each file. There are 73 pairs of these files extending from
2000-01-29 (the beginning of normal -120C ACIS operations) until
2019-05-02 (ACIS T_GAIN combined epochs 77 and 78).
With the release of ACIS T_GAIN Epoch 75-76 (CalDB 4.8.3, May 2019) the ACIS Calibration Team introduced and new algorithm designed optimally for averaging of data with diminished statistics. The results were a marked improvement in T_GAIN correction factor derivation. With the advent of the improved ACIS DET_GAIN N0008 file (CalDB 4.9.1, April 2020), ACIS Cal began to review the earlier ACIS T_GAIN epochs with the new gains applied to the data.
The resulting new T_GAIN correction data showed a marked improvement over the previously-released T_GAIN files. See Item A of the Technical Details Section below for details. The ACIS Cal Team has decided to upgrade T_GAIN files for all -120C epochs (specifically epochs 1-74). It was also found that recompiling the T_GAIN data with (CHIPX x CHIPY) of (32 x 128) could reduce the size of the files significantly without compromising T_GAIN correction detailing. Hence all 148 active files are being upgraded (all of epochs 1-80) in this release.
ACIS Level 1 pipeline tool acis_process_events.
As the Chandra spacecraft ages, it becomes increasingly difficult to maintain the ACIS focal plane temperature (FP_TEMP in the software and FITS data sets) firmly at -120C for the entire observing period of each orbit. This is particularly true during periods approaching and exiting the Earth's radiation zone. The CXC Mission Planning team has been considering allowing science observations to be scheduled during these warmer periods, specifically allowing ACIS-S/HETG and certain restricted imaging observations with reduced statistics to be taken with FP_TEMP up to -109C. The warmer temperatures were approved in advance of the 06 July 2020 observing load.
The files listed above in the ACIS branch of CalDB have a boundary condition on FP_TEMP that is likely too restrictive for planned -109C observations. The boundary condition is currently set as 'FP_TEMP(151.16 - 164.16)K' (-121.99C - -108.99C). With this release, the boundary condition has been changed to 'FP_TEMP(148.15-168.16)K', or (-125.0 - -104.99)C, effective 2020-07-01T00:00:00 for all of the above CalDB files, except for the two provisional T_GAIN files, wherein the CBD20001 keyword has simply been changed without changing the effective date. (Since these are provisional files, they will be upstaged in the near future regardless, and so there was no need to add any further time-dependence to their listings.)
For all of the above sixteen files excluding the T_GAINs, an extra CalDB keyword set applicable to science observations has been added, including CCNM* (codename or product), CDES* (alibration description), CVSD* (cal validity start date), CVST* (cal validity start time), and boundary conditions including the new FP_TEMP expanded range as required for proper file selection for dates after CVSD*=2020-07-01T00:00:00. Here "*" indicates the version number of the added keyword set, sufficient to distinguish it from pre-existing ones, and hence must be '0002' or higher version. Details are given in the Technical Details seciton below. These added keyword sets produce a new line for each relevant FITS file/Extension in the ACIS branch CalDB index file, whence it may be selected in the software for new, including warmer, observations.
The rapid loss of HRC-S gain since the year 2015 has precipitated problems with filtering background events from LETG/HRC-S observation spectra, which is the basic purpose of setting the gain calibration on HRC-S. The HRC-S/LETG calibration team, specifically Brad Wargelin, has engaged in a 2-year project to develop a new time-dependent gain map method, to be applied in two separate T_GMAPs, one for before the 2012-03-29 HV change, and one for after that time. The new T_GMAPs require a different branch of software to apply them in hrc_process_events, which was released in CIAO 4.12 (and with DS 10.8.2), but which will only become active with the installation of this CalDB release.
See the HRC-S T_GMAP why page for more information.
HRC Level 1 Processing Tool (SDP): hrc_process_events
CIAO reprocessing script: chandra_repro, which uses hrc_process_events.
Regular calibration observations have shown that the HRC-S QE is declining at long wavelengths (>60Å, or below 200 eV) for some time, decaying linearly since the 2012-03-29 high-voltage adjustment onboard. This spectral variation in the QE loss will, with this CalDB release, be accounted for using a series of time-displaced QE files in CalDB, which are annually adjusted each year beginning in 2013.
See the HRC-S QE/QEU why page for summary information on the current status of the HRC-S QE and QEU calibrations. See the technical details section below for more specific information on this new calibration and its derivation.
SDP HRC-S Level 2 python script: tg_mkresponse.py generates +/- first order grating ARFs for LETG/HRC-S OBS_IDs.
CIAO contributed tool: mktgresp will generate first and higher order as specified in a Type II PHA file, or according to a user specification of the "orders" parameter.
CiAO thread using the HRC-S QE: LETG/HRC-S Grating ARFs
The HRC-I Calibration team has updated the QE files for HRC-I following on from the 2018 version (v11) which introduced a time-dependence into the HRC-I QE. In this version, the team extends the time dependence to include the latest epoch of observation. During recent calibration observations, a steady decline has been seen in the observed count rates and estimated fluxes of the soft source white dwarf system HZ43 and the hard source SNR G21.5-0.9. This upgrade maintains consistency with the HRC-S instrumental QE, particularly at low energies (below 626 eV), where the shape of the QE(E) is maintained in common.
CIAO contributed script: fluximage
CIAO thread: HRC-I Exposure Map and Exposure-corrected Image
III. TECHNICAL DETAILS
The ACIS combined T_GAIN epochs 1 - 80, covering the bi-epochal period of 29 Jan 2000 - 31 Jan 2020, have been recompiled using an improved statistical averaging technique for data with decreasing counting statistics, which has resulted in considerable improvement in the resulting noise level in the corrected PHA/ENERGY data in the events lists. In addition, the tiling of the "corrgain", and hence CalDB T_GAIN, datasets has been somewhat optimized to maintain CHIPX to 32X detailing, while increasing the CHIPY tiling to 128Y, a direction less sensitive to variations along the chips. The corrgain datasets are those compiled by R. Nick Durham of the ACIS calibration team by analyzing ACIS External Calibration Source (ECS) data during Chandra' radiation zone passes.
This new statistical averaging software was built by Nick Durham, and used to generate the ACIS T_GAIN Epochs 75+76 for release in CalDB 4.8.3, in May 2019. Once certain kinks were worked out in deriving new "corrgain" calibration interface product files, the method could be applied to any ECS epochal data. With the advent of the new ACIS DET_GAIN version N0008 file, released in CalDB 4.9.1 as of April 2020, the Cal Team decided to check on the older ECS corrgain-selected data sets, to see what the result would be.
The selected ECS data from each quarterly epoch of ACIS since the start of -120C operations on 29 Jan 2000 have been reanalyzed with the improved method for all ACIS chips except for chips 4 and 9, the outermost chips on the "S" array. Hence, all of the ACIS-I chips have been redone in the "corrgain" files that are used to generate the T_GAIN files for CalDB. In addition, because of the re-tiling, even the most recent T_GAIN files have been recompiled, vetted, tested in the software against real observations, and cleared for installation in this CalDB release.
The results, particularly for earlier T_GAIN epochs, are somewhat striking, particularly for the higher energy lines of the ECS. See Figures 1-2 a and b below for Epochs 20 and 63 on ACIS-I3. The improvement for Epoch 20 (late 2004 ECS data) is obvious for the Mn K-alpha1 line, with some improvement also visible in the lower-energy lines. The improvement is less obvious in this case for epoch 63 ECS data, taken in mid-2015. While the improvements in the T_GAIN are limited to the 0.2% level overall, the improvement is sufficient to warrant the full upgrade of the ACIS T_GAIN library, which currently consists of 148 files in the CalDB.
Figures 3-4 a and b below give a similar illustration, but for FI chip ACIS-6 (S2).
Figures 5-6 a and b below give the same illustration, but for BI chip ACIS-7 (S3).
The new 32x128 tiling of the T_GAIN is introducing some stepping (the downward ramps) in the newest results above. This effect can be eliminated by an appropriate interpolation between the correction values at a given CHIPX, and has been requested as an enhancement in the acis_process_events tool for a future CIAO release.
As the spacecraft ages and continues to warm, the need to accommodate warmer ACIS operations for real science observations has increased, to the point where it has become desirable for mission planners to allow certain low-count imaging, as well as some ACIS-S/HETG observations, to be done at limited warmer focal plane temperatures (FP_TEMP in the CalDB and software). The decision has been made to allow a limited number of ORs to be done at temperatures up to -109C (164.15K) nominally. The FP_TEMP cannot quite be controlled to that accuracy, and so -108 C is probably a better real upper limit for these operations.
The ACIS branch of the CalDB currently uses a limiting FP_TEMP boundary condition of (151.16-164.16)K, which is slightly to restrictive for the planned warmer ops, so it becomes necessary to expand this range. It has also been suggested that some ACIS observations might be started with -122C FP_TEMP, to allow the temperature to sweep through the optimal calibration range in some cases, or for calibration purposes. Hence the FP_TEMP CalDB boundary condition should simply be expanded at both ends, to provide some safety. If the FP_TEMP value of an OR is outside the current boundary range in CalDB, standard processing and CIAO routines will fail to obtain DET_GAIN, T_GAIN, P2_RESP, QEU, OSIP, FEF_PHA, and BADPIX files.
Hence, we have updated the FP_TEMP boundary condition limits to (148.15 - 168.16)K, or -125 to -104.99 C, which gives plenty of temperature space at both ends for proposed operations. These changes are to be made only to the currently effective ACIS CalDB files with FP_TEMP boundary settings, and to be effective as of the UTC date 2020-07-01T00:00:00, so that this change cannot affect reprocessing of older observations, regardless of their FP_TEMP values.
This has been accomplished by adding an additional keyword set to each of 18 currently active ACIS CalDB files, with the expanded boundary conditions, and with a Calibration Validity Start Date (CVSD000*) of 2020-07-01T00:00:00.
The rapid loss of HRC-S gain since the year 2015 has precipitated problems with filtering background events from LETG/HRC-S observation spectra, which is the basic purpose of setting the gain calibration on HRC-S. The HRC-S/LETG calibration team, specifically Brad Wargelin, has engaged in a 2-year project to develop a new time-dependent gain map method, to be applied in two separate T_GMAPs, one for before the 2012-03-29 HV change, and one for after that time.
Details of the actual calibration are given on two new HRC-S Calibrations web pages: HRC-S Gain Map and LETG Background Filter (2020 Long version) with a shorter version here: HRC-S Gain Map and LETG Background Filter (2020, Main Page)
From the Introduction of the longer version web page:
The new calibration benefits from several improvements over the 2008 calibration:
- Gains for observations taken after the March 2012 HV change (released in CALDB 4.5.1) were based on rescaled extrapolation of pre-change trends; the new calibration uses observational data.
- The lab-based spatial gain calibration is applied before the time-dependent gain calibration, which minimizes the effects of aim point drift combined with 2D spatial gain variation.
- LETG/HRC-S event tg coordinates are adjusted using dewiggle, detilt, and symmetrize, and spectral and background data are then extracted using narrower regions that reduce background contamination.
- Periods of even slightly enhanced background are now removed.
- Data are analyzed for each 1/3 of a CRSV tap instead of few-tap bins.
- Taps are divided into thirds based on counts rather than pixel values, i.e., RAWY(modulo 256)=0:88/89:166/167:255 rather than 0:84/85:170/171:255.
- Corrections for higher-order diffraction are more accurate: they use a better pulse-height vs model and convolve each order's flux with the dither pattern.
- There are new observations of novae, which have very bright spectra without higher order contamination.
- Several off-axis Capella observations were used to map gain near the aim point.
The primary practical result of the gain calibration is that pulse height filtering can be applied to LETG spectra as a function of dispersed wavelength in order to reduce background by over half longward of 20 Å, compared with standard Level 2 processing.
The following text is adapted from the public twicky web page generated by Peter Ratzlaff, available at the link:
HRC-S QE N0015
The figures referred to below are not included in this ECR, but are available at the link above. From this posted page:
From regular HRC-S/LETG calibration observations of the hot white dwarf HZ 431 , it is known that the HRC-S QE has declined steadily at wavelengths longer than 60 Å (200 eV). This drop has been linear since the high tension voltage change in March, 2012, at a yearly loss rate of 2.39 (+/- 0.04)% (more precisely, the QE decline is modeled as linear versus time with a slope of [1-0.0239] units per annum) in dispersed orders (Fig. 1 at link above). While in the past decade there have been increasingly large addiional position-dependent corrections necessary for the outer plates, broadly speaking the linear QE decline can be characterized as "grey," i.e., independent of wavelength.
Despite the lack of a standard candle that illuminates the entire detector, there is still one useful source available for relative calibration of the central plate: the isolated neutron star RX J1856.5-3754. This source, although somewhat faint ( 0.11 counts s ~ -1 for positive first order in the 20-60 Å band), yields measurable flux for wavelengths longer than 20 Å on S2, and has been observed with the HRC-S/LETG configuration in 2001, 2013 and 2019. Comparison of fluxes from those epochs, derived with the current CalDB, which extends the 2.39% QE loss per year to all dispersed wavelengths, is consistent with the assumption of grey decline (Fig. 2 at link above). The calculated fluxes in both 2013 and 2019 are in agreement to the approximately 10% level with those from 2001.
For higher energies, we are left with extragalactic sources such as the blazar Markarian 421, which is highly variable, displaying variations of >50% over timescales of 10s of ksec. For the purpose of cross calibration, it is often observed in a sequence of consecutive interleaved instrument/grating (ACIS-S with HETGS and LETGS, HRC-S with LETGS) configurations. We then model the flux light curve as a continuous auto-regressive model (CARMA; Kelly et al. 2014), adjusting the relative QE calibration of HRC-S to ensure the smoothest curve through the instrument changes to and from ACIS-S. Examples of these interleaved observations are shown in Fig. 3 at link above, for the past three years, in the 3-6 Å band. The left side panels show flux curves in the years 2017-2019, using the current CalDB HRC-S QE version N0014. As can be seen in the lower left plot, the discontinuity between HRC-S/LETG (blue) and ACIS-S/LETG (black) has become especially pronounced in recent years. Figs. 4-7 (at link above) show similar trends for other wavelength bands, going out to 18 Å.
Calibrating the HRC-S to ACIS-S via Mkn 421 while assuming the ACIS-S QE remains constant requires, at the shortest wavelengths, raising the HRC-S QE increasingly over time. Though these data are at low signal-to-noise, some trends in the required QE correction are apparent. The discrepancy between ACIS-S and HRC-S is roughly correlated with wavelength: shorter wavelengths require more modification. Similarly, the disagreement between detectors is generally growing over time, and this appears to have begun around the same time as the HV change in 2012.
We address the flux discrepancy by raising the HRC-S QE relative to the grey correction the most at shortest wavelengths, linearly ramping down that excess correction to unity (i.e., no additional correction beyond the grey secular one) at 20 Å. Additionally, the QE will be linearly increased over time, post HV change, in order to compensate for the grey secular correction (Fig. 8 at the above link).
Beginning in 2013, QE N0015 contains one calibration file per year. At the shortest wavelength (0 Å), the QE in S2 rises at the rate of 2.1% per year (a linear increase versus time with a slope of 1.021 units per year relative to 2012.5), which was the value found to most closely match the factor which must be applied to the highest energy HRC-S fluxes so that they agree with ACIS-S. This upwards correction shorter than 20 Å on S2 is counter to the grey decline implemented in the QEU. Together, the QEU correction and new QE counter-correction amount to a net "real" decline in HRC-S QE of 0.34% per year at the highest energies. This wavelength-dependent correction of the HRC-S QE, in addition to linearly ramping up with time, also linearly ramps down with wavelength until it is zeroed out relative to the grey correction at 20 Å.
The right side panels in Figs. 4-7 (at the above link) show how the HRC-S fluxes change with this QE update. They are generally in better agreement with ACIS-S, although the 6-9 Å and 12-15 Å bands still require additional scrutiny, particularly in the latest 2019 observations. The upcoming interleaved calibration observations in July, 2020 will shine further light on the situation.
The plerionic supernova remnant G21.5-0.9 can provide a consistency check on our corrections. The plerion has a highly-absorbed spectrum, with X-ray emission almost entirely at energies greater than 1 keV (Fig. 9 at link above). As shown in Fig. 10 (at link above), the latest HRC-S imaging plerion flux measurements using QE N0015 more closely match historical values than did those computed from the N0014 version.
From Vinay Kashyap of the HRC-I Calibration Team on 16 June 2020, with figures available at the link Chandra/HRC-I Quantum Efficiency vN0012.
We have updated the QE files for the HRC-I, following on from the 2018 version (v11) which introduced a time-dependence into the HRC-I QE. In this version, we extend the time dependence to include the latest epoch of observation.
During recent calibration observations, a steady decline has been seen in the observed count rates and estimated fluxes of the soft source white dwarf system HZ43 (Figure 1) and the hard source SNR G21.5-00.9 (Figure 2).
We continue to tie the HRC-I low energy (E<626 eV) QE to the shape of the HRC-S QE (in this case v14), while being normalized to match the zeroth-order aimpoint count rates of HZ43. The HZ43 rates are loess-smoothed with a 2nd-degree polynomial over +/-5 nearest observations, in order to avoid overfitting the QE corrections to the measured rates. This implies a retroactive change even at older epochs of ~0.5%, while newly predicted rates match the observed rates with an rms of 1% (Figure 3).
Additionally, we also impose a grey decline of 1.5% percent/year in the QE post-2016 before applying the HRC-S shape correction in order to account for the QE drop visible at high energies. This has the effect of correcting the drop in flux for G21.5-00.9 (Figure 4).
The update (Figure 5) applies to all previous epochs, and as before, QE files are provided for each epoch when a HZ 43 calibration observation is made.