Update ARD to flag chip and node boundaries as bad pixels
Update analysis reference data for ACIS to flag chip and node boundary and neighboring pixels (as appropriate based on READMODE and DATAMODE) as bad so that they do not result in false source detections.
Use SIM position directly from aspect solution instead of aspect offsets
Revise pipelines to extract SIM dy, dz positions directly from the aspect solution file rather than from the aspect offsets file. This improves the aspect solution by extracting the actual SIM position for each aspect record. (The SIM positions in the aspect offsets file are updated only when the deltas from the previous recorded values exceed a predefined threshold.)
Bias image FITS headers not updated
Several keywords (e.g., REVISION, CALDBVER) are missing from the FITS headers of level 1 bias images (produced for FAINT_BIAS datamode only).
ACIS PI bins calculated incorrectly
ACIS PI (pulse height invariant) bins are erroneously shifted by -1.
Use TAN-P projection for DET coordinate system
Compute DET (detector) coordinates using the TAN-P (polar projection tangent plane) coordinate system rather than the TAN coordinate system. The TAN-P transform uses longitude (LONG) and North polar angle (NPOL) to map the tangent plane, with NPOL = 0 at the North pole.
Record one dimensional sky position (SKY_1D) for ACIS CC mode events
Record the one dimensional sky position of each event recorded in ACIS CC mode relative to the position of the target (ie, RA_TARG, DEC_TARG) in the column SKY_1D. This enables the user to construct the one dimensional projection of the sky image along the CHIPX axis.
Counts per pixel in level 2 images sometimes overflows short integer
The number of counts in a single pixel in the images created in level 2 sometimes overflows the range of a short integer. This typically occurs in the full field image, where the events from several detector pixels are summed into a single image pixel. The datatypes of these images should be changed from 16 to 32 bits. All level 2 pipelines are revised.
Add source detect region to background region exposure ratio to output
A new source output column EXPO_RATIO that lists the ratio of the source's detect region exposure to the background region exposure is added to the level 2 src data products. The celldetect parameter expratio suppresses generation of ASCII regions for sources that have EXPO_RATIO less than the value specified by this parameter.
Revise source list columns to support CIAO region file format
The source list columns are revised to incorporate the information needed for compatibility with the CIAO (DataModel) region file format. This allows them to be used directly to specify regions for spatial filtering by dmextract.
Produce observation summary pages in level 2
Create PostScript and HTML observation summary pages at the end of level 2 processing. These pages will be distributed to the user as paper products and summarize the observation configuration and provide a quick look at the data.
Compound filtering with multiple good time interval components incorrect
Compound filters - such as separate good time intervals for each ACIS chip - do not perform correctly. This results in the level 2 events (which should be filtered according to good time intervals on a per CCD basis) being filtered instead on the logical AND of the filters for all of the CCD chips. For example, if the GTIs for chips 1-5 are 100-1000 and the GTI for chip 6 is 200-1200 then the events for all chips will be filtered using the interval 200-1000. The ONTIMEs recorded in the FITS headers are computed correctly per CCD. The filtering also works correctly for the case of a single GTI list (eg, for HRC).
Add calibration database version (CALDBVER) to FITS headers
Add CALDBVER (CALibration DataBase VERsion number) to the FITS headers of the level 0.5/level 1/level 2 files.
Continuous clocking ACIS FITS headers should not include CYCLE keyword
The CYCLE keyword should not be included in the FITS headers for ACIS CONTINUOUS clocking observations. CYCLE is only relevant for TIMED exposure data.
HISTORY keywords in FITS headers do not comply with ASC-FITS-2.0
The HISTORY keywords in the headers of level 0 FITS files do not comply with the requirements of ASC-FITS-2.0. Specifically, there is an extra space inserted prior to the ASC-FITS HISTORY keyword label specifier.
All level 1 and above ACIS FITS headers should include READMODE
ACIS READMODE is erroneously not propogated to the FITS headers of some level 1 and above ACIS data files.
Add background light curve to mission time line
Add the background light curve (count rate in counts per second) to the mission time line. The background region is determined based on analysis of the histogram of the event data.
Data not cut correctly when science run spans multiple obsids
If an ACIS science run spans multiple obsids, then the level 1 data products incorrectly span the entire duration of the science run, rather than just the relevant observation interval (the intervals outside the ObI are marked as bad time). This means that data that are potentially proprietary to another observer could be distributed as part of the dataset (has never happened). Solution is to cut the input data to level 1 at the ObI boundaries.
Detect ACA hot pixels and exclude from aspect centroid determination
Dynamically detect ACA (Aspect Camera Assembly) pixels with high dark current ("hot pixels"). Exclude detected pixels when computing the centroid of the star/fid light image. In earlier releases, the aspect solution could be degraded because of intermittently hot pixels in one or more star/fid light images.
AIPROPS average pointing coordinates are incorrect for some intervals
The average pointing coordinates (avg_ra, avg_dec, avg_roll) are incorrect in the level 0.5 AIPROPS (Aspect Interval PROPertieS) files for some aspect intervals. In these cases, the values recorded in the AIPROPS file are the actual offsets from the target pointing rather than the actual pointing (ie, target pointing + actual offsets).
Compute initial quaternion when possible even if some data bad
Modify pipeline to determine initial pointing quaternion from telemetry data when all of the individual telemetry elements necessary for the computation are present, even if some of the ancillary data elements (such as those used to calculate estimated errors) have bad data quality. In earlier releases, if any data element was bad, then the quaternion was not calculated even though all the information needed to do so was good. This results in a large solution error at the beginning of the aspect interval.
PCAD telemetry data at end of file erroneously ignored
PCAD engineering telemetry that is sampled less frequently than aspect camera data is not identified as matching the latter in the interval from the TIME of the last record in a file to TSTOP.
Update ARD to calibrate PI column for HRC-S
Update analysis reference data for HRC to provide a calibration of the PI (Pulse-height Invariant) column for HRC-S data. The PI column is a mapping of the observed pulse height corrected for variation of gain across the detector. Prior to this, PI was set to 0 for all HRC data (still true for HRC-I data).
Data not cut correctly when HRC mode changes
The beginning or end of a data interval is determined incorrectly when the HRC data mode changes between OBSERVING and NEXT_IN_LINE in HRC level 0.5 files. This may result in invalid secondary science data products being fed into higher level processing if a mode change occurs within an observation interval. Secondary science data should always be cut on major frame boundaries for OBSERVING mode, and should start with a minor frame 4 record and end with a minor frame 0 record for NEXT_IN_LINE mode. Event data should be cut at corresponding times.
Correct event fine positions using new tap ringing algorithm
Under certain circumstances (the amplifier scale factor is switched to the least sensitive scale, an even number of taps on the axis have signals that are above a set threshold, and the event occurs on the negative side of the tap), tap amplifier ringing in the HRC hardware corrupts the fine positions of events. An empirical algorithm (tap ringing correction) has been developed to correct the event positions. In order to apply the tap ringing correction, the following conditions have to be satisfied:
Revise amp saturation test to support ARDs as a function of amp_sf
Revise the HRC amplifier saturation test algorithm to support different limits in the analysis reference data depending on the value of the amplifier scale factor (amp_sf) used by the HRC electronics for the event. Previously only a single set of limits could be used for the amplifier saturation test, independent of the amp_sf.
Revise hyperbolic test to use lower cutoff for center tap fraction
Revise the HRC hyperbolic test algorithm to include a lower limit for the fractional center tap ADC signal (= center tap ADC value / sum of tap ADC values). If the center tap fraction is smaller than the cutoff ARD value, then the event status is flagged as bad.
Update OBI_NUM, STARTMJF, STOPMJF, STARTMNF, and STOPMNF in FITS headers
OBI_NUM (observation interval number), and start and stop major frame and minor frame numbers are not correctly updated in the FITS headers of HRC level 0.5 files when the data are cut by the pipeline due to HRC detector or data mode changes.
Add smoothed mission time line product with limit violations flagged
Add the level 1 smoothed mission time line data product (std_mtl1) to the data product set. This data product is the actual mission time line product that is used as input to the good time interval filters in level 1 used to calculate the GTIs. The principal HDU (block MTL) contains the mission timeline values smoothed according to the MTL specifications, as well as a limit_status bit-array that records for each row whether the corresponding limit was violated (1) or not (0). The subsequent HDU (block LIMITS) lists the actual gti_limits applied to determine whether the limits were violated or not. This permits the user to modify and reapply the GTI limit set.
Filtering on no good time intervals erroneously includes all data
All data should be filtered out when creating level 2 files if there are no good time intervals (ie, when the good time interval limits are violated for the entire observation). When the data subspace is empty, data model filtering is erroneously including all data rather than excluding all data.
end_ut_time not recorded correctly in SIM strip files
The SIM stripper is not recording the end data time in the strip files. This results in bad TSTOP values during telemetry extraction.
SIM stripper drops data from first major frame of each diagnostic record
The SIM stripper is dropping the 4 bytes of diagnostic data recorded in the first major frame of each 32 major frame SIM diagnostics record.
Bin size for ACIS/LETG specified incorrectly
The bin size for ACIS/LETG is incorrectly specified as 1.25e-03 angstrom. It should be 1.25e-02 angstrom. Also set default number of bins for ACIS/LETG in tool to 8192.
Support independent spectral binning scales for HEG and MEG
Modify the grating extraction tool (tgextract) to allow the spectral binning scales to be independently specified for HEG and MEG spectra of the HETG. Previously, the binning parameters were interpreted as MEG parameters, and corresponding parameters for HEG were scaled from MEG - resulting in non-"simple" bin sizes. (Partial implementation.)
Use celldetect's encircled energy threshold in tgdetect
When running celldetect as part of tgdetect, use a value of 0.925 for eenergy (the fraction of PSF energy to be encircled by the detect cell). Empirical studies indicate that this is a better default value than the previous 0.9.
Support enhanced specification of background regions
The grating region extraction tool (tgextract) is modified to allow (a) independent specification of rectangular background regions above and below the spectrum in the cross-dispersion direction via tool parameters, or (b) specification of non-rectangular background regions via a region file. For case (a), new parameters {min|max}_{up|down}bkg_tg_d specify the minimum and maximum tg_d values to include in the up and down background regions, and for case (b) the region file is specified by the parameter inregion_file. Option (b) provides support for bowtie shaped regions for background extraction of HRC-S/LETG spectra (however, as of this release the level 2 pipeline does not fully support the new tool, and no ARD defining such a background is defined).
Add support for new format response matrix files (RMFs)
New format response matrix files are supported. The new RMFs energy data depend spatially on chip coordinates, and in addition have energy limits identified as absolute values, rather than as differences from the peak. The old format is still supported, and at release time only ARDs in the old format are available.
Detect/filter persistent events from cosmic-ray/charged particle afterglow
Add a tool (acis_detect_afterglow) to the level 1 pipeline that detects persistent events due to detector afterglow (residual charge) caused by cosmic-ray or charged particle impacts on the detector. The tool is applied only to datasets obtained in TIMED mode that do not use alternating (interleaved) exposures. The number of consecutive ACIS frames in which events are present at the same detector pixel (in chip coordinates) and have monotonically decreasing pulse heights (PHA(n+1) <= PHA(n)) are counted and the chain length (number of frames) minus one is encoded in bits 16-19 of the event STATUS bit flags. The standard event filter in the level 2 pipeline filters out events with STATUS != 0, so chains of length 2 or greater will be removed from the level 2 event files in the standard data processing pipeline. As a byproduct of this update, the length of the STATUS bit flag column was extended from 16 to 32 bits.
Ignore bogus bias parity errors from ACIS flight software
Mark pixel (1023, 1) as bad because of bogus bias parity errors that are the result of an ACIS flight software bug (VFAINT datamode only).
Too many pixels near node boundaries flagged as bad for VFAINT datamode
"Chip edge" bad pixel flags should be set only for one pixel on each side of the node boundaries in the bad pixel (bpix1) data product. Starting with R4CU5UPD7, two pixels on each side of the node boundaries are flagged for VFAINT datamode.
Coordinate system (detector, sky) limits TLMIN/TLMAX incorrect
Due to pixel randomization, the TLMIN/TLMAX limits recorded in the FITS headers for the DET and SKY coordinates should be, respectively, 0.5 pixels smaller/larger. The correct values for ACIS are 0.5 to 8192.5.
Coordinate system (detector, sky) precision changed
Change the precision with which DET and SKY coordinates are recorded in the FITS file event lists from double precision to single precision. This results in significant space savings, and analysis indicates that single precision meets all of the requirements for coordinate accuracy. The coordinates are still computed internally in double precision.
Bad pixels in 1st row at top of CCD not identified in bpix1 file
For continuous clocking (CC) mode observations, bad pixels in the 1st row at the top of the each CCD are incorrectly not identified in the level 1 bad pixel (bpix1) file.
Columns removed from level 2 event files
The following columns are removed from the level 2 event files: PHAS, CORN_PHA. They remain in the level 1 event files, but are not appropriate for higher level data products. This results in significant space savings.
Some ACIS level 1 tools fail if target position not located on detector
Some tools in the ACIS level 1 pipeline that must identify which CCD the target is located on do not function correctly if the target position falls off the detector. These include the background light curve calculation, and the CC mode sky coordinate computation. If the target position (RA_TARG, DEC_TARG) does not fall on the detector, then instead use the pointing position (RA_PNT, DEC_PNT). If this also does not fall on the detector, then use defaults in detector space, which usually correspond to the center of the nominal aimpoint chip.
Correct calibration database version (CALDBVER) to FITS headers
The value of CALDBVER (CALibration DataBase VERsion number) populated in the FITS headers of the level 0.5/level 1/level 1.5/level 2 files is incorrectly truncated after the integer part. For example, version "1.2" is incorrectly recorded as "1.".
Coordinate system (detector, sky) limits TLMIN/TLMAX incorrect
Due to pixel randomization, the TLMIN/TLMAX limits recorded in the FITS headers for the DET and SKY coordinates should be, respectively, 0.5 pixels smaller/larger. The correct values for are 0.5 to 32768.5 for HRC-I, and 0.5 to 65536.5 for HRC-S.
Columns removed from level 2 event files
The following columns are removed from the level 2 event files: CRSU, CRSV, AMP_SF, AU1, AU2, AU3, AV1, AV2, AV3, RAWX, RAWY, SUMAMPS. They remain in the level 1 event files, but are not appropriate for higher level data products. This results in significant space savings.
CCD node id column naming inconsistent with other data products
The CCD node id column is named "ccd_node" in the level 1.5 grating data products, but "node_id" in the level 1 and other data products. Rename "ccd_node" to "node_id" in the level 1.5 products.
Revise definition of EXPOSURE keyword in the level 2 pha data product
Revise the EXPOSURE keyword in the FITS header of the level 2 pha data product to be the average of the EXPOSURn keywords. This changes the definition of the EXPOSURE keyword IN THE pha2 FILE ONLY to be the mean value of the exposures for the individual CCDs contributing to the data, and is required for proper integration of the CIAO RMF tools and threads. For all other data products, the EXPOSURE keyword remains unchanged (ie, equal to the EXPOSURn for the CCD on which the target position is located).
Coordinate system (grating coordinates) precision changed
Change the precision with which GAC (tg_r, tg_d), tg_lam, and tg_mlam coordinates are recorded in the FITS file event lists from double precision to single precision. This results in significant space savings, and analysis indicates that single precision meets all of the requirements for grating coordinate accuracy. The coordinates are still computed internally in double precision.
Columns removed from level 2 event files
The following columns are removed from the level 2 event files: GDPX, GDPY. They remain in the level 1 event files, but are not appropriate for higher level data products. This results in significant space savings.
Add column TG_PART identifying grating part to region extension
A new column TG_PART is added to the output region extension from tgextract identifying the grating part for each FITS region. Valid values for TG_PART are "HEG", "HETG", "LETG", or "MEG".
Record the background to source region area scale factors in the FITS header
Record the ratio of the background region area to the source region area for tgextract in the BACKSCAL keyword in the FITS header. If separate up and down background regions are used, record the individual area ratios in the BACKSCUP and BACKSCDN keywords.
The POISSERR keyword in the FITS header should be of type boolean
The POISSERR keyword in the FITS header from tgextract is a boolean value (true/false) and should be recorded as a FITS boolean header keyword rather than as a FITS string header keyword.
Support enhanced specification of background regions
Update the level 2 pipeline to support independent specification of rectangular background regions above and below the spectrum in the cross-dispersion direction and specification of non-rectangular background regions via a region file. As of this release no ARD defining such a background is defined.
Revise the Rowland spacing value from 8632.48 mm to 8637.0 mm
Revise the Rowland spacing value from 8632.48 mm to 8637.0 mm per input from the calibration group.
Proposal information that includes special LaTeX characters not handled
Proposal information such as proposal title, target name, etc. that include special LaTeX characters are not handled correctly when generating summary pages. These characters need to be escaped for LaTeX.
Data quality checking performed on wrong row
When processing a row of engineering and HRC housekeeping data, the data quality check is erroneously being performed on the data for the subsequent row. This can result in bad data incorrectly not being identified and therefore used improperly.
Update DATE-OBS, DATE-END based on TSTART, TSTOP for OVERRIDE obspar
Revise obidet to automatically determine the correct values of DATE-OBS and DATE-END from the TSTART and TSTOP values recorded in an OVERRIDE obspar. Previously, the user would be required to compute matching values for override processing.
Modify the HRC level 2 status filter to support new operating mode
Per request from the Science Operations Team HRC scientist, the event status filter is modified to ignore the values of status bits 21 (veto status - V axis width exceeded) and 22 (veto status - U axis center blank event) to allow for a modification in the way the HRC is to be operated in the future.
HISTORY keywords in FITS headers do not comply with ASC-FITS-2.0
The HISTORY keywords in the headers of level 0.5 and level 1 Aspect FITS files do not comply with the requirements of ASC-FITS-2.0. Specifically, the sequence number field is missing. (Fixed for all except aspect quality aqual1 data product.)
REVISION keyword in FITS headers is not updated correctly
The REVISION keyword in the headers of level 0.5 and level 1 Aspect FITS files is not updated with the actual data product revision number from the processing system. Instead, the "dummy" value from the observation parameter file (usually 1) is being used.
Keyword obspar_type is not updated to 'ACTUAL' for OVERRIDE obspar
The obspar keyword obspar_type should be set to 'ACTUAL' for the output (actual) obspar even in OVERRIDE mode. This keyword is a flag to support database/archive ingestion of the obspar.
Modify the HRC level 2 status filter to support new operating mode
Per request from the Science Operations Team HRC scientist, the event status filter is modified as follows: (a) ignore the values of status bits 20 (veto status - U axis width exceeded) and 21 (veto status - V axis width exceeded), and (b) check that the values of status bits 24 (event status - U axis not triggered) and 25 (event status - V axis not triggered) are clear. This backs out and corrects the change that was applied in R4CU5UPD10 (which was specified incorrectly).
Enhance aspect solution to differentiate between gyro rollover and large data gap
An enhanced algorithm is added that allows the gyro count processing in the aspect solution to differentiate between a gyro count rollover and a large change in the gyro counts resulting from a data gap. In earlier versions the inability to distinguish these cases could result in an invalid aspect solution for an aspect interval that included a large data gap.
Aspect quality target_offset incorrect when target quaternion is bad
The target_offset column in the aspect quality data product is computed incorrectly when one or more target quaternions in the PCAD engineering data has bad data quality or is gapped. The asp_make_qualint tool should identify and ignore data flagged as bad or gapped data.
Aspect solution not computed for short aspect intervals
(1) The aspect pipeline cannot compute an aspect solution for short time intervals with start and stop times that fall between single major frame timestamps (ie, every 32.8 seconds). (2) For short intervals, the number of FID light images present in the FIDPROPS file may be smaller than the expected number, causing the pipeline to fail. The pipeline crashes and an aspect solution is not determined for such intervals.
The observation mean pointing is calculated incorrectly in some cases
The actual mean pointing (RA_PNT, DEC_PNT) during an observation interval (ObI) is calculated incorrectly if the obi boundaries fall entirely within a single KALMAN constant aspect interval. If the ObI is the first one for a given Obs_Id, then the nominal pointing (RA_NOM, DEC_NOM) will also be incorrect. This is a rare occurrence.
ACA-CAL stripper does not time out on end of data
The ACA-CAL stripper (Aspect Camera Assembly CALibration - used only for ACA calibration data, not normal science operations) does not time out after the end of data is reached. As a result, the ACA-CAL telemetry extractor is not automatically kicked of to process the data.
HRC status filter update inadvertently not applied to secondary mode pipelines
The update to the HRC level 2 status bit filter mask that was included in R4CU5UPD11.1 was inadvertently not applied to the secondary mode pipelines (ie, for data with obsmode 'secondary' --- not-on-the-sky calibrations). The status mask change is required for these pipelines also.
Grating position should not be considered in GTI for secondary mode
For data with obsmode 'secondary' (ie, not-on-the-sky calibrations) the position of the grating is not relevant for determining good time, and should be removed from the MTL.
Actual obspar detnam 'HRC-SI' should match predicted obspar detnam 'HRC-S'
If the HRC-S is operated in 'imaging' mode (ie, triggering event data from the central MCP only), then the observation interval determination pipeline will set the detector name (detnam) in the actual obspar to 'HRC-SI'. However, the predicted obspar will have detnam = 'HRC-S' because the ocat does not recognize HRC-SI. To support HRC-SI observations, the dynamic GTI limits output to the olims0a data product created by the observation interval determination pipeline should consider 'HRC-SI' to be identical to 'HRC-S'. The use use HRC-SI is temporary pending implementation of HRC window masks.
Potential problem with calibration database version (CALDBVER) 2.0
There is a potential problem when the value of CALDBVER (CALibration DataBase VERsion number) populated in the FITS headers of the level 0.5/level 1/level 1.5/level 2 files is updated to 2.0. The current code would result in CALDBVER being recorded incorrectly as 2.7.
ACA image stripper records same VCDU counter for successive images
Under some circumstances, the ACA image stripper would record the same VCDU counter (major/minor frame number) for two images in succession. This problem could only occur if the first minor frame of a major frame was gapped.
Support different observation interval selection criteria for secondary mode
For data with obsmode 'secondary' (ie, not-on-the-sky calibrations) the criteria for selecting observation interval elements that match the predicted configuration may be different from those for on-the-sky observations (eg, grating position should not be considered for secondary mode). The tolerances file should be retrieved from the CALDB with the obsmode as a boundary condition. The selected CALDB file should be recorded in HISTORY.
The BACKSCAL, BACKSCUP, and BACKSCDN keyword values are erroneously inverted
The background area to source region area scale factors in the FITS FITS headers (BACKSCAL, BACKSCUP, and BACKSCDN) are erroneously being recorded as the inverse (1 / x) of the correct values.
CALDB queries for detnam 'HRC-SI' should return data for 'HRC-S'
If the HRC-S is operated in 'imaging' mode (ie, triggering event data from the central MCP only), then the detector name (detnam) will be set to 'HRC-SI'. However, the calibration database (CALDB) index does not recognize detnam = 'HRC-SI', and so incorrect analysis reference data (ARDs) are retrieved for this configuration. To support HRC-SI observations, the pipelines should issue requests to the CALDB for 'HRC-S' ARDs if 'HRC-SI' is used. The use use HRC-SI is temporary pending implementation of HRC window masks.
Tools using <detnam>.par parameter files should use hrc-s.par if detnam = 'HRC-SI'
If the HRC-S is operated in 'imaging' mode (ie, triggering event data from the central MCP only), then the detector name (detnam) will be set to 'HRC-SI'. The level 1.5 Transmission grating pipeline attempts to extract parameters for the file 'hrc-si.par' and fails. To support HRC-SI observations, the pipelines should use the gile 'hrc-s.par' instead. The use use HRC-SI is temporary pending implementation of HRC window masks.
Grating position check should not be included for secondary mode
For data with obsmode 'secondary' (ie, not-on-the-sky calibrations) the criteria for selecting observation interval elements that match the predicted configuration may be different from those for on-the-sky observations (eg, grating position should not be considered for secondary mode). The dynamic observation GTI limits file should not include a check of the Transmission grating position in this case, as the grating position is irrelevant.
acis_build_badpix crashes for continuous clocking data with bad bias pixels
Under some circumstances, the acis_build_badpix tool that identifies bad pixels crashes when processing continuous clocking data with bad pixels identified in the bias frame.