The data products generated by the Automated Processing (AP) pipelines of the CXC Data System are initially cached and subsequently ingested into the archive, usually within a day. Before any products can be accessed by users, these products must pass a Verification and Validation (V&V) test. If the quality of the data products is good enough to be released, they are made available to the user(s). The V&V inspection is especially important for proprietary data: since release of the data to the GO starts the clock on the proprietary period, V&V provides a mechanism that prevents that clock's starting on the release of data products of poor quality.
If a previous version of the products is already in the archive, the new products enter the archive as ``pending" and replace the old products as the default version upon V&V approval. Normally, only the current default version is available to users. Older versions can be made available by the Archive Operations group (arcops) upon special, reasoned request. The version of any product is encoded in the FITS headers as the keyword REVISION and in the file names through the substring of three digits that is preceded by the capital letter `N'. Note that version numbers are incremented by processing level but do not carry over to subsequent levels. For instance, Level 1 Version 2 may be used to produce Level 2 Version 3 and Version 4. The version of the software used for creating each data product is encoded in the FITS keyword ASCDSVER.
An approved proposed observation is given a six-digit Sequence Number (SeqNum). Observations are then given an instrument configuration and an Observation Id (ObsId). Although usually there is a one-to-one relation between SeqNum and ObsId, it is possible for a SeqNum to contain more than one ObsId -- for instance when the user requests a change of instrument configuration for a partially complete ObsId. And an ObsId may be broken up into multiple Observation Intervals (ObI). This may happen when an observation is not quite completed or gets scheduled in more than one part. ObIs are processed individually through Level 1 or Level 1.5. The data for all ObIs from a single ObsId are combined in the Level 2 products. If we have an ObsId that initially has a single ObI, Level 2 products will be generated containing just that one ObI. When the second ObI comes along, a second version of the Level 2 products is generated containing data from both ObIs. Be aware that, as a result, Level 2 products may appear to cover a long period that is in no way representative of the observation time included.
The standard data distribution contains primary and secondary data products. The CDA web pages list the products in each category. Upon request, Arcops can make additional products available. When approved by V&V, the data for each ObsId are packaged into a tar file and the PI receives notice how and where to pick up that file. CXCDS operations will put the data also on a physical medium, if requested by the proposer and mail that medium to the PI. If AP has difficulty with a certain dataset, the duty scientist may be able to nudge the pipeline processing. This is commonly known as Special Automated Processing (SAP). SAP products are handled in the same way as all AP products. If that fails, the user may request Custom Processing (CP) by CXCDS Operations; this requires approval by the Chandra Director's Office (CDO). CP data products are only shipped via ftp. They are not archived and not distributed on physical media.
All observations other than calibration observations may have a proprietary period associated with them. For observations proposed through the regular NRAs the default is one year, but may be less, at the proposer's discretion. For Director's Discretionary Time (DDT) observations it may be zero to three months. Only science data associated with proprietary observations are protected during the proprietary period. The clock for the proprietary period starts when the PI receives notice of data for at least 80% of a SeqNum's approved exposure time being available for download. Note that this does not go by ObI, or by ObsId, but by SeqNum. The public release date for each ObsId may be found through the processing status tool, accessible from the CDA web pages.
At this time we are still using our provisional retrieval interface. Although it is fairly flexible in regard to the products users may select for retrieval, it is not very user-friendly when it comes to browsing the observations. We have just started testing a new generation interface for browsing and retrieval that will be far more powerful. This new interface will come on-line for the users in the Spring of 2001. It will be able to handle public as well as proprietary data retrieval. For proprietary data all PIs will be given a password-protected archive account for their personal use, as well as an account for each proposal. The latter will allow PIs to share data access with their CoIs, while the former provides PIs with a single account access to all their data.
- Arnold Rots