Grant Tremblay, Ralph Kraft, Paul Nulsen, Dan Patnaude, Tom Gauron, Almus Kenter, Ken Gage, Brad Bissell, Josh Model, Gerald Austin, Tom Aldcroft, & Sabina Hurley
In our last Newsletter update, we told the harrowing tale of the 2020 HRC anomalies, and the long road to successful recovery of the instrument on redundant (B-side) electronics. Indeed, since the HRC returned to science observations on November 18, 2020, the instrument continued to enable world-class Chandra science. It shed light on how magnetic cataclysmic variables transfer mass to white dwarf companions, advanced our understanding of how stellar dynamos shut down, and embarked on a groundbreaking search for X-ray emission from Saturn as it passed through Jupiter’s magnetotail.
We therefore regrettably report that, after 527 days of nominal and stable B-side operations, the HRC suffered from another electronic anomaly whose signature was almost identical to those that afflicted the A-side in 2020. At 13:17:32.084 UT (8:17am ET) on February 9th, 2022, the +/- 15 V voltages on the redundant B-side power supply bus collapsed to off-nominal values of +3.36 V and -0.16 V, accompanied by a temperature rise and drop in the central and forward electronics assemblies (respectively). All secondary science data was corrupted with the loss of the 15 V bus, although the anticoincidence shield continued to operate nominally. The anomaly was noticed at the start of the 12:15pm ET comm pass, and we immediately shut the instrument down to place it in a safe state. An intensive (and continuing) B-side anomaly investigation began at that moment, drawing on the huge body of work already done for the 2020 A-side anomalies.
We show the major signatures of all three anomalies in Figure 1, with the latest 2022 occurrence plotted in purple. Indeed, the three anomalies are barely distinguishable from one another, even though two completely independent electronics boards are involved.
Figure 1: The three HRC anomalies that have occurred since August 2020. The two A-side anomalies from 2020 are shown in blue and red, while the newest B-side 2022 anomaly is shown in purple. The top panel shows bus voltages collapsing to similar off-nominal values. The bottom panel shows similar warming and cooling trends in the central and forward electronics assemblies (respectively) at the onset of each anomaly. Note that the second 2020 anomaly occurred 11h 31min after we had restarted the instrument from a cold (off) state, so its temperature signature is offset. The flicker in the FEA temperature decline is understood (it’s effectively electronics noise).
The HRC Science Operations, Flight Operations, and Instrument Principal Investigator teams have since been hard at work with fault tree analysis (including laboratory testing and even X-ray computed tomography (CT) imaging of flight spare components!), discussion with Chandra’s original hardware vendors, and complex Command Action Procedures with the instrument to determine paths forward toward eventual resumption of science for the HRC. A parallel effort is also underway to enable the Spacecraft’s On-board Computer (OBC) to directly work with telemetry from our (still healthy and operational) anticoincidence shield, and the ACIS team has undertaken a major effort to resume science operations with a modified radiation monitoring algorithm while the HRC is shut down. This is another demonstration of the Chandra team at its very best.
We are encouraged to report that, while we unsuccessfully attempted to recover the B-side on March 11, 2022, our recent intervention to swap back to the A-side on May 19 saw the recovery of entirely nominal and stable telemetry for over two hours during our test (Figure 2).
Figure 2: A successful restart attempt on Side A. This spacecraft Command Action Procedure (CAP), which we executed on May 11, 2022, successfully recovered nominal voltages in the A-side power supply bus. Telemetry remained nominal and stable for the duration of our ~2 hour test. We intentionally shut the instrument down at its conclusion.
Recall that, when we reset the Side A power supplies following the first 2020 Anomaly, we recovered nominal voltages for 11 hours and 31 minutes (about 40 ks). That the instrument operated nominally for a time during the both the 2020 and 2022 Side A restarts indeed suggests a possible path forward for a new operational paradigm with the HRC. We look forward to keeping the community up-to-date as we work tirelessly to return this phenomenal and powerful instrument to science.