Fetch API documentation

Fetch values from the Ska engineering telemetry archive.

Classes

class Ska.engarchive.fetch.MSID(msid, start='1999:204', stop=None, filter_bad=False, stat=None)

Bases: object

Fetch data from the engineering telemetry archive into an MSID object.

The input msid is case-insensitive and can include linux file “glob” patterns, for instance orb*1*_x (ORBITEPHEM1_X) or *pcadmd (AOPCADMD). For derived parameters the initial DP_ is optional, for instance dpa_pow* (DP_DPA_POWER).

Parameters:

• msid – name of MSID (case-insensitive)
• start – start date of telemetry (Chandra.Time compatible)
• stop – stop date of telemetry (current time if not supplied)
• filter_bad – automatically filter out bad values
• stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

Returns:

MSID instance

fetch = <module 'Ska.engarchive.fetch' from '/proj/sot/ska/dev/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/Ska.engarchive-0.36-py2.7.egg/Ska/engarchive/fetch.pyc'>

filter_bad(bads=None, copy=False)

Filter out any bad values.

After applying this method the “bads” column will be set to None to indicate that there are no bad values.

Parameters:

• bads – Bad values mask. If not supplied then self.bads is used.
• copy – return a copy of MSID object with bad values filtered

filter_bad_times(start=None, stop=None, table=None, copy=False)

Filter out intervals of bad data in the MSID object.

There are three usage options:

• Supply no arguments. This will use the global list of bad times read in with fetch.read_bad_times().
• Supply both start and stop values where each is a single value in a valid DateTime format.
• Supply an table parameter in the form of a 2-column table of start and stop dates (space-delimited) or the name of a file with data in the same format.

The table parameter must be supplied as a table or the name of a table file, for example:

bad_times = ['2008:292:00:00:00 2008:297:00:00:00',
             '2008:305:00:12:00 2008:305:00:12:03',
             '2010:101:00:01:12 2010:101:00:01:25']
msid.filter_bad_times(table=bad_times)
msid.filter_bad_times(table='msid_bad_times.dat')

Parameters:

• start – Start of time interval to exclude (any DateTime format)
• stop – End of time interval to exclude (any DateTime format)
• table – Two-column table (start, stop) of bad time intervals
• copy – return a copy of MSID object with bad times filtered

interpolate(dt=None, start=None, stop=None, times=None)

Perform nearest-neighbor interpolation of the MSID to the specified time sequence.

The time sequence steps uniformly by dt seconds starting at the start time and ending at the stop time. If not provided the times default to the first and last times for the MSID.

The MSID times attribute is set to the common time sequence. In addition a new attribute times0 is defined that stores the nearest neighbor interpolated time, providing the original timestamps of each new interpolated value for that MSID.

If times is provided then this gets used instead of the default linear progression from start and dt.

Parameters:

• dt – time step (sec, default=328.0)
• start – start of interpolation period (DateTime format)
• stop – end of interpolation period (DateTime format)
• times – array of times for interpolation (default=None)

iplot(fmt='-b', fmt_minmax='-c', **plot_kwargs)

Make an interactive plot for exploring the MSID data.

This method opens a new plot figure (or clears the current figure) and plots the MSID vals versus times. This plot can be panned or zoomed arbitrarily and the data values will be fetched from the archive as needed. Depending on the time scale, iplot displays either full resolution, 5-minute, or daily values. For 5-minute and daily values the min and max values are also plotted.

Once the plot is displayed and the window is selected by clicking in it, the following key commands are recognized:

a: autoscale for full data range in x and y
m: toggle plotting of min/max values
p: pan at cursor x
y: toggle autoscaling of y-axis
z: zoom at cursor x
?: print help

Example:

dat = fetch.Msid('aoattqt1', '2011:001', '2012:001', stat='5min')
dat.iplot()
dat.iplot('.b', '.c', markersize=0.5)

Caveat: the iplot() method is not meant for use within scripts, and may give unexpected results if used in combination with other plotting commands directed at the same plot figure.

Parameters:

• fmt – plot format for values (default=”-b”)
• fmt_minmax – plot format for mins and maxes (default=”-c”)
• plot_kwargs – additional plotting keyword args

logical_intervals(op, val, complete_intervals=True, max_gap=None)

Determine contiguous intervals during which the logical comparison expression “MSID.vals op val” is True. Allowed values for op are:

==  !=  >  <  >=  <=

If complete_intervals is True (default) then the intervals are guaranteed to be complete so that the all reported intervals had a transition before and after within the telemetry interval.

If max_gap is specified then any time gaps longer than max_gap are filled with a fictitious False value to create an artificial interval boundary at max_gap / 2 seconds from the nearest data value.

Returns a structured array table with a row for each interval. Columns are:

• datestart: date of interval start
• datestop: date of interval stop
• duration: duration of interval (sec)
• tstart: time of interval start (CXC sec)
• tstop: time of interval stop (CXC sec)

Examples:

>>> dat = fetch.MSID('aomanend', '2010:001', '2010:005')
>>> manvs = dat.logical_intervals('==', 'NEND')

>>> dat = fetch.MSID('61PSTS02', '1999:200', '2000:001')
>>> safe_suns = dat.logical_intervals('==', 'SSM', complete_intervals=False, max_gap=66)

Parameters:

• op – logical operator, one of == != > < >= <=
• val – comparison value
• complete_intervals – return only complete intervals (default=True)
• max_gap – max allowed gap between time stamps (sec, default=None)

Returns:

structured array table of intervals

plot(*args, **kwargs)

Plot the MSID vals using Ska.Matplotlib.plot_cxctime()

This is a convenience function for plotting the MSID values. It is equivalent to:

plot_cxctime(self.times, self.vals, *args, **kwargs)

where *args are additional arguments and **kwargs are additional keyword arguments that are accepted by plot_cxctime().

Example:

dat = fetch.Msid('tephin', '2011:001', '2012:001', stat='5min')
dat.plot('-r', linewidth=2)

raw_vals

Raw counts corresponding to the string state-code values that are stored in self.vals

remove_intervals(intervals, copy=False)

Remove telemetry points that occur within the specified intervals

This method is the converse of select_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to removing intervals, and that copy is returned. The default is to remove intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then removed and the standard deviation is recomputed. This filters out the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.remove_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
-0.0003688639491030978
>>> figure(2)
>>> aorate2.plot(',')

Parameters:

• intervals – EventQuery or iterable (N x 2) with start, stop dates/times
• copy – return a copy of MSID object with intervals removed

select_intervals(intervals, copy=False)

Select telemetry points that occur within the specified intervals

This method is the converse of remove_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to selecting intervals, and that copy is returned. The default is to selecte intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then selected and the mean is recomputed. This highlights the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.select_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
24.764309542605481
>>> figure(2)
>>> aorate2.plot(',')

Parameters:

• intervals – EventQuery or iterable (N x 2) with start, stop dates/times
• copy – return a copy of MSID object with intervals selected

state_codes

List of state codes tuples (raw_count, state_code) for state-valued MSIDs

state_intervals()

Determine contiguous intervals during which the MSID value is unchanged.

Returns a structured array table with a row for each interval. Columns are:

• datestart: date of interval start
• datestop: date of interval stop
• duration: duration of interval (sec)
• tstart: time of interval start (CXC sec)
• tstop: time of interval stop (CXC sec)
• val: MSID value during the interval

Example:

dat = fetch.MSID('cobsrqid', '2010:001', '2010:005')
obsids = dat.state_intervals()

Parameters:val – state value for which intervals are returned. Returns:structured array table of intervals

tdb

Access the Telemetry database entries for this MSID

write_zip(filename, append=False)

Write MSID to a zip file named filename

Within the zip archive the data for this MSID will be stored in csv format with the name <msid_name>.csv.

Parameters:

• filename – output zipfile name
• append – append to an existing zipfile

class Ska.engarchive.fetch.MSIDset(msids, start='1999:204', stop=None, filter_bad=False, stat=None)

Bases: collections.OrderedDict

Fetch a set of MSIDs from the engineering telemetry archive.

Each input msid is case-insensitive and can include linux file “glob” patterns, for instance orb*1*_? (ORBITEPHEM1_X, Y and Z) or aoattqt[1234] (AOATTQT1, 2, 3, and 4). For derived parameters the initial DP_ is optional, for instance dpa_pow* (DP_DPA_POWER).

Parameters:

• msids – list of MSID names (case-insensitive)
• start – start date of telemetry (Chandra.Time compatible)
• stop – stop date of telemetry (current time if not supplied)
• filter_bad – automatically filter out bad values
• stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

Returns:

Dict-like object containing MSID instances keyed by MSID name

class MSID(msid, start='1999:204', stop=None, filter_bad=False, stat=None)

Bases: object

Fetch data from the engineering telemetry archive into an MSID object.

The input msid is case-insensitive and can include linux file “glob” patterns, for instance orb*1*_x (ORBITEPHEM1_X) or *pcadmd (AOPCADMD). For derived parameters the initial DP_ is optional, for instance dpa_pow* (DP_DPA_POWER).

Parameters:

• msid – name of MSID (case-insensitive)
• start – start date of telemetry (Chandra.Time compatible)
• stop – stop date of telemetry (current time if not supplied)
• filter_bad – automatically filter out bad values
• stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

Returns:

MSID instance

fetch = <module 'Ska.engarchive.fetch' from '/proj/sot/ska/dev/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/Ska.engarchive-0.36-py2.7.egg/Ska/engarchive/fetch.pyc'>

filter_bad(bads=None, copy=False)

Filter out any bad values.

After applying this method the “bads” column will be set to None to indicate that there are no bad values.

Parameters:

• bads – Bad values mask. If not supplied then self.bads is used.
• copy – return a copy of MSID object with bad values filtered

filter_bad_times(start=None, stop=None, table=None, copy=False)

Filter out intervals of bad data in the MSID object.

There are three usage options:

• Supply no arguments. This will use the global list of bad times read in with fetch.read_bad_times().
• Supply both start and stop values where each is a single value in a valid DateTime format.
• Supply an table parameter in the form of a 2-column table of start and stop dates (space-delimited) or the name of a file with data in the same format.

The table parameter must be supplied as a table or the name of a table file, for example:

bad_times = ['2008:292:00:00:00 2008:297:00:00:00',
             '2008:305:00:12:00 2008:305:00:12:03',
             '2010:101:00:01:12 2010:101:00:01:25']
msid.filter_bad_times(table=bad_times)
msid.filter_bad_times(table='msid_bad_times.dat')

Parameters:

• start – Start of time interval to exclude (any DateTime format)
• stop – End of time interval to exclude (any DateTime format)
• table – Two-column table (start, stop) of bad time intervals
• copy – return a copy of MSID object with bad times filtered

interpolate(dt=None, start=None, stop=None, times=None)

Perform nearest-neighbor interpolation of the MSID to the specified time sequence.

The time sequence steps uniformly by dt seconds starting at the start time and ending at the stop time. If not provided the times default to the first and last times for the MSID.

The MSID times attribute is set to the common time sequence. In addition a new attribute times0 is defined that stores the nearest neighbor interpolated time, providing the original timestamps of each new interpolated value for that MSID.

If times is provided then this gets used instead of the default linear progression from start and dt.

Parameters:

• dt – time step (sec, default=328.0)
• start – start of interpolation period (DateTime format)
• stop – end of interpolation period (DateTime format)
• times – array of times for interpolation (default=None)

iplot(fmt='-b', fmt_minmax='-c', **plot_kwargs)

Make an interactive plot for exploring the MSID data.

This method opens a new plot figure (or clears the current figure) and plots the MSID vals versus times. This plot can be panned or zoomed arbitrarily and the data values will be fetched from the archive as needed. Depending on the time scale, iplot displays either full resolution, 5-minute, or daily values. For 5-minute and daily values the min and max values are also plotted.

Once the plot is displayed and the window is selected by clicking in it, the following key commands are recognized:

a: autoscale for full data range in x and y
m: toggle plotting of min/max values
p: pan at cursor x
y: toggle autoscaling of y-axis
z: zoom at cursor x
?: print help

Example:

dat = fetch.Msid('aoattqt1', '2011:001', '2012:001', stat='5min')
dat.iplot()
dat.iplot('.b', '.c', markersize=0.5)

Caveat: the iplot() method is not meant for use within scripts, and may give unexpected results if used in combination with other plotting commands directed at the same plot figure.

Parameters:

• fmt – plot format for values (default=”-b”)
• fmt_minmax – plot format for mins and maxes (default=”-c”)
• plot_kwargs – additional plotting keyword args

logical_intervals(op, val, complete_intervals=True, max_gap=None)

Determine contiguous intervals during which the logical comparison expression “MSID.vals op val” is True. Allowed values for op are:

==  !=  >  <  >=  <=

If complete_intervals is True (default) then the intervals are guaranteed to be complete so that the all reported intervals had a transition before and after within the telemetry interval.

If max_gap is specified then any time gaps longer than max_gap are filled with a fictitious False value to create an artificial interval boundary at max_gap / 2 seconds from the nearest data value.

Returns a structured array table with a row for each interval. Columns are:

• datestart: date of interval start
• datestop: date of interval stop
• duration: duration of interval (sec)
• tstart: time of interval start (CXC sec)
• tstop: time of interval stop (CXC sec)

Examples:

>>> dat = fetch.MSID('aomanend', '2010:001', '2010:005')
>>> manvs = dat.logical_intervals('==', 'NEND')

>>> dat = fetch.MSID('61PSTS02', '1999:200', '2000:001')
>>> safe_suns = dat.logical_intervals('==', 'SSM', complete_intervals=False, max_gap=66)

Parameters:

• op – logical operator, one of == != > < >= <=
• val – comparison value
• complete_intervals – return only complete intervals (default=True)
• max_gap – max allowed gap between time stamps (sec, default=None)

Returns:

structured array table of intervals

plot(*args, **kwargs)

Plot the MSID vals using Ska.Matplotlib.plot_cxctime()

This is a convenience function for plotting the MSID values. It is equivalent to:

plot_cxctime(self.times, self.vals, *args, **kwargs)

where *args are additional arguments and **kwargs are additional keyword arguments that are accepted by plot_cxctime().

Example:

dat = fetch.Msid('tephin', '2011:001', '2012:001', stat='5min')
dat.plot('-r', linewidth=2)

raw_vals

Raw counts corresponding to the string state-code values that are stored in self.vals

remove_intervals(intervals, copy=False)

Remove telemetry points that occur within the specified intervals

This method is the converse of select_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to removing intervals, and that copy is returned. The default is to remove intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then removed and the standard deviation is recomputed. This filters out the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.remove_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
-0.0003688639491030978
>>> figure(2)
>>> aorate2.plot(',')

Parameters:

• intervals – EventQuery or iterable (N x 2) with start, stop dates/times
• copy – return a copy of MSID object with intervals removed

select_intervals(intervals, copy=False)

Select telemetry points that occur within the specified intervals

This method is the converse of remove_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to selecting intervals, and that copy is returned. The default is to selecte intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then selected and the mean is recomputed. This highlights the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.select_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
24.764309542605481
>>> figure(2)
>>> aorate2.plot(',')

Parameters:

• intervals – EventQuery or iterable (N x 2) with start, stop dates/times
• copy – return a copy of MSID object with intervals selected

state_codes

List of state codes tuples (raw_count, state_code) for state-valued MSIDs

state_intervals()

Determine contiguous intervals during which the MSID value is unchanged.

Returns a structured array table with a row for each interval. Columns are:

• datestart: date of interval start
• datestop: date of interval stop
• duration: duration of interval (sec)
• tstart: time of interval start (CXC sec)
• tstop: time of interval stop (CXC sec)
• val: MSID value during the interval

Example:

dat = fetch.MSID('cobsrqid', '2010:001', '2010:005')
obsids = dat.state_intervals()

Parameters:val – state value for which intervals are returned. Returns:structured array table of intervals

tdb

Access the Telemetry database entries for this MSID

write_zip(filename, append=False)

Write MSID to a zip file named filename

Within the zip archive the data for this MSID will be stored in csv format with the name <msid_name>.csv.

Parameters:

• filename – output zipfile name
• append – append to an existing zipfile

MSIDset.filter_bad(copy=False)

Filter bad values for the MSID set.

This function applies the union (logical-OR) of bad value masks for all MSIDs in the set with the same content type. The result is that the filtered MSID samples are valid for all MSIDs within the content type and the arrays all match up.

For example:

msids = fetch.MSIDset(['aorate1', 'aorate2', 'aogyrct1', 'aogyrct2'],
                      '2009:001', '2009:002')
msids.filter_bad()

Since aorate1 and aorate2 both have content type of pcad3eng they will be filtered as a group and will remain with the same sampling. This will allow something like:

plot(msids['aorate1'].vals, msids['aorate2'].vals)

Likewise the two gyro count MSIDs would be filtered as a group. If this group-filtering is not the desired behavior one can always call the individual MSID.filter_bad() function for each MSID in the set:

for msid in msids.values():
    msid.filter_bad()

Parameters:copy – return a copy of MSID object with intervals selected

MSIDset.filter_bad_times(start=None, stop=None, table=None, copy=False)

Filter out intervals of bad data in the MSIDset object.

There are three usage options:

• Supply no arguments. This will use the global list of bad times read in with fetch.read_bad_times().
• Supply both start and stop values where each is a single value in a valid DateTime format.
• Supply an table parameter in the form of a 2-column table of start and stop dates (space-delimited) or the name of a file with data in the same format.

The table parameter must be supplied as a table or the name of a table file, for example:

msidset.filter_bad_times()
bad_times = ['2008:292:00:00:00 2008:297:00:00:00',
             '2008:305:00:12:00 2008:305:00:12:03',
             '2010:101:00:01:12 2010:101:00:01:25']
msidset.filter_bad_times(table=bad_times)
msidset.filter_bad_times(table='msid_bad_times.dat')

Parameters:

• start – Start of time interval to exclude (any DateTime format)
• stop – End of time interval to exclude (any DateTime format)
• table – Two-column table (start, stop) of bad time intervals
• copy – return a copy of MSID object with intervals selected

MSIDset.interpolate(dt=None, start=None, stop=None, filter_bad=True, times=None, bad_union=False, copy=False)

Perform nearest-neighbor interpolation of all MSID values in the set to a common time sequence. The values are updated in-place.

Times

The time sequence steps uniformly by dt seconds starting at the start time and ending at the stop time. If not provided the times default to the start and stop times for the MSID set.

If times is provided then this gets used instead of the default linear progression from start and dt.

For each MSID in the set the times attribute is set to the common time sequence. In addition a new attribute times0 is defined that stores the nearest neighbor interpolated time, providing the original timestamps of each new interpolated value for that MSID.

Filtering and bad values

If filter_bad is True (default) then bad values are filtered from the interpolated MSID set. There are two strategies for doing this:

1. bad_union = False

   Remove the bad values in each MSID prior to interpolating the set to a common time series. This essentially says to use all the available data individually. Each MSID has bad data filtered individually before interpolation so that the nearest neighbor interpolation only finds good data. This strategy is done when filter_union = False, which is the default setting.

2. bad_union = True

Mark every MSID in the set as bad at the interpolated time if any of them are bad at that time. This stricter version is required when it is important that the MSIDs be truly correlated in time. For instance this is needed for attitude quaternions since all four values must be from the exact same telemetry sample. If you are not sure, this is the safer option.

Parameters:

• dt – time step (sec, default=328.0)
• start – start of interpolation period (DateTime format)
• stop – end of interpolation period (DateTime format)
• filter_bad – filter bad values
• times – array of times for interpolation (default=None)
• bad_union – filter union of bad values after interpolating
• copy – return a new copy instead of in-place update (default=False)

MSIDset.write_zip(filename)

Write MSIDset to a zip file named filename

Within the zip archive the data for each MSID in the set will be stored in csv format with the name <msid_name>.csv.

Parameters:filename – output zipfile name

class Ska.engarchive.fetch.Msid(msid, start='1999:204', stop=None, filter_bad=True, stat=None)

Bases: Ska.engarchive.fetch.MSID

Fetch data from the engineering telemetry archive into an MSID object. Same as MSID class but with filter_bad=True by default.

Parameters:

• msid – name of MSID (case-insensitive)
• start – start date of telemetry (Chandra.Time compatible)
• stop – stop date of telemetry (current time if not supplied)
• filter_bad – automatically filter out bad values
• stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)
• unit_system – Unit system (cxc|eng|sci, default=current units)

Returns:

MSID instance

class Ska.engarchive.fetch.Msidset(msids, start='1999:204', stop=None, filter_bad=True, stat=None)

Bases: Ska.engarchive.fetch.MSIDset

Fetch a set of MSIDs from the engineering telemetry archive. Same as MSIDset class but with filter_bad=True by default.

Parameters:

• msids – list of MSID names (case-insensitive)
• start – start date of telemetry (Chandra.Time compatible)
• stop – stop date of telemetry (current time if not supplied)
• filter_bad – automatically filter out bad values
• stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)
• unit_system – Unit system (cxc|eng|sci, default=current units)

Returns:

Dict-like object containing MSID instances keyed by MSID name

class MSID(msid, start='1999:204', stop=None, filter_bad=False, stat=None)

Bases: object

Fetch data from the engineering telemetry archive into an MSID object.

The input msid is case-insensitive and can include linux file “glob” patterns, for instance orb*1*_x (ORBITEPHEM1_X) or *pcadmd (AOPCADMD). For derived parameters the initial DP_ is optional, for instance dpa_pow* (DP_DPA_POWER).

Parameters:

• msid – name of MSID (case-insensitive)
• start – start date of telemetry (Chandra.Time compatible)
• stop – stop date of telemetry (current time if not supplied)
• filter_bad – automatically filter out bad values
• stat – return 5-minute or daily statistics (‘5min’ or ‘daily’)

Returns:

MSID instance

fetch = <module 'Ska.engarchive.fetch' from '/proj/sot/ska/dev/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/Ska.engarchive-0.36-py2.7.egg/Ska/engarchive/fetch.pyc'>

filter_bad(bads=None, copy=False)

Filter out any bad values.

After applying this method the “bads” column will be set to None to indicate that there are no bad values.

Parameters:

• bads – Bad values mask. If not supplied then self.bads is used.
• copy – return a copy of MSID object with bad values filtered

filter_bad_times(start=None, stop=None, table=None, copy=False)

Filter out intervals of bad data in the MSID object.

There are three usage options:

• Supply no arguments. This will use the global list of bad times read in with fetch.read_bad_times().
• Supply both start and stop values where each is a single value in a valid DateTime format.
• Supply an table parameter in the form of a 2-column table of start and stop dates (space-delimited) or the name of a file with data in the same format.

The table parameter must be supplied as a table or the name of a table file, for example:

bad_times = ['2008:292:00:00:00 2008:297:00:00:00',
             '2008:305:00:12:00 2008:305:00:12:03',
             '2010:101:00:01:12 2010:101:00:01:25']
msid.filter_bad_times(table=bad_times)
msid.filter_bad_times(table='msid_bad_times.dat')

Parameters:

• start – Start of time interval to exclude (any DateTime format)
• stop – End of time interval to exclude (any DateTime format)
• table – Two-column table (start, stop) of bad time intervals
• copy – return a copy of MSID object with bad times filtered

interpolate(dt=None, start=None, stop=None, times=None)

Perform nearest-neighbor interpolation of the MSID to the specified time sequence.

The time sequence steps uniformly by dt seconds starting at the start time and ending at the stop time. If not provided the times default to the first and last times for the MSID.

The MSID times attribute is set to the common time sequence. In addition a new attribute times0 is defined that stores the nearest neighbor interpolated time, providing the original timestamps of each new interpolated value for that MSID.

If times is provided then this gets used instead of the default linear progression from start and dt.

Parameters:

• dt – time step (sec, default=328.0)
• start – start of interpolation period (DateTime format)
• stop – end of interpolation period (DateTime format)
• times – array of times for interpolation (default=None)

iplot(fmt='-b', fmt_minmax='-c', **plot_kwargs)

Make an interactive plot for exploring the MSID data.

This method opens a new plot figure (or clears the current figure) and plots the MSID vals versus times. This plot can be panned or zoomed arbitrarily and the data values will be fetched from the archive as needed. Depending on the time scale, iplot displays either full resolution, 5-minute, or daily values. For 5-minute and daily values the min and max values are also plotted.

Once the plot is displayed and the window is selected by clicking in it, the following key commands are recognized:

a: autoscale for full data range in x and y
m: toggle plotting of min/max values
p: pan at cursor x
y: toggle autoscaling of y-axis
z: zoom at cursor x
?: print help

Example:

dat = fetch.Msid('aoattqt1', '2011:001', '2012:001', stat='5min')
dat.iplot()
dat.iplot('.b', '.c', markersize=0.5)

Caveat: the iplot() method is not meant for use within scripts, and may give unexpected results if used in combination with other plotting commands directed at the same plot figure.

Parameters:

• fmt – plot format for values (default=”-b”)
• fmt_minmax – plot format for mins and maxes (default=”-c”)
• plot_kwargs – additional plotting keyword args

logical_intervals(op, val, complete_intervals=True, max_gap=None)

Determine contiguous intervals during which the logical comparison expression “MSID.vals op val” is True. Allowed values for op are:

==  !=  >  <  >=  <=

If complete_intervals is True (default) then the intervals are guaranteed to be complete so that the all reported intervals had a transition before and after within the telemetry interval.

If max_gap is specified then any time gaps longer than max_gap are filled with a fictitious False value to create an artificial interval boundary at max_gap / 2 seconds from the nearest data value.

Returns a structured array table with a row for each interval. Columns are:

• datestart: date of interval start
• datestop: date of interval stop
• duration: duration of interval (sec)
• tstart: time of interval start (CXC sec)
• tstop: time of interval stop (CXC sec)

Examples:

>>> dat = fetch.MSID('aomanend', '2010:001', '2010:005')
>>> manvs = dat.logical_intervals('==', 'NEND')

>>> dat = fetch.MSID('61PSTS02', '1999:200', '2000:001')
>>> safe_suns = dat.logical_intervals('==', 'SSM', complete_intervals=False, max_gap=66)

Parameters:

• op – logical operator, one of == != > < >= <=
• val – comparison value
• complete_intervals – return only complete intervals (default=True)
• max_gap – max allowed gap between time stamps (sec, default=None)

Returns:

structured array table of intervals

plot(*args, **kwargs)

Plot the MSID vals using Ska.Matplotlib.plot_cxctime()

This is a convenience function for plotting the MSID values. It is equivalent to:

plot_cxctime(self.times, self.vals, *args, **kwargs)

where *args are additional arguments and **kwargs are additional keyword arguments that are accepted by plot_cxctime().

Example:

dat = fetch.Msid('tephin', '2011:001', '2012:001', stat='5min')
dat.plot('-r', linewidth=2)

raw_vals

Raw counts corresponding to the string state-code values that are stored in self.vals

remove_intervals(intervals, copy=False)

Remove telemetry points that occur within the specified intervals

This method is the converse of select_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to removing intervals, and that copy is returned. The default is to remove intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then removed and the standard deviation is recomputed. This filters out the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.remove_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
-0.0003688639491030978
>>> figure(2)
>>> aorate2.plot(',')

Parameters:

• intervals – EventQuery or iterable (N x 2) with start, stop dates/times
• copy – return a copy of MSID object with intervals removed

select_intervals(intervals, copy=False)

Select telemetry points that occur within the specified intervals

This method is the converse of remove_intervals().

The intervals argument can be either a list of (start, stop) tuples or an EventQuery object from kadi.

If copy is set to True then a copy of the MSID object is made prior to selecting intervals, and that copy is returned. The default is to selecte intervals in place.

This example shows fetching the pitch component of the spacecraft rate. After examining the rates, the samples during maneuvers are then selected and the mean is recomputed. This highlights the large rates during maneuvers:

>>> aorate2 = fetch.Msid('aorate2', '2011:001', '2011:002')
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
3.9969393528801782
>>> figure(1)
>>> aorate2.plot(',')

>>> from kadi import events
>>> aorate2.select_intervals(events.manvrs)
>>> aorate2.vals.mean() * 3600 * 180 / np.pi  # rate in arcsec/sec
24.764309542605481
>>> figure(2)
>>> aorate2.plot(',')

Parameters:

• intervals – EventQuery or iterable (N x 2) with start, stop dates/times
• copy – return a copy of MSID object with intervals selected

state_codes

List of state codes tuples (raw_count, state_code) for state-valued MSIDs

state_intervals()

Determine contiguous intervals during which the MSID value is unchanged.

Returns a structured array table with a row for each interval. Columns are:

• datestart: date of interval start
• datestop: date of interval stop
• duration: duration of interval (sec)
• tstart: time of interval start (CXC sec)
• tstop: time of interval stop (CXC sec)
• val: MSID value during the interval

Example:

dat = fetch.MSID('cobsrqid', '2010:001', '2010:005')
obsids = dat.state_intervals()

Parameters:val – state value for which intervals are returned. Returns:structured array table of intervals

tdb

Access the Telemetry database entries for this MSID

write_zip(filename, append=False)

Write MSID to a zip file named filename

Within the zip archive the data for this MSID will be stored in csv format with the name <msid_name>.csv.

Parameters:

• filename – output zipfile name
• append – append to an existing zipfile

Functions

Ska.engarchive.fetch.get_telem(msids, start=None, stop=None, sampling='full', unit_system='eng', interpolate_dt=None, remove_events=None, select_events=None, time_format=None, outfile=None, quiet=False, max_fetch_Mb=1000, max_output_Mb=100)

High-level routine to get telemetry for one or more MSIDs and perform common processing functions:

• Fetch a set of MSIDs over a time range, specifying the sampling as either full-resolution, 5-minute, or daily data.
• Filter out bad or missing data.
• Interpolate (resample) all MSID values to a common uniformly-spaced time sequence.
• Remove or select time intervals corresponding to specified Kadi event types.
• Change the time format from CXC seconds (seconds since 1998.0) to something more convenient like GRETA time.
• Write the MSID telemetry data to a zipfile.

Parameters:

• msids – MSID(s) to fetch (string or list of strings)’)
• start – Start time for data fetch (default=<stop> - 30 days)
• stop – Stop time for data fetch (default=NOW)
• sampling – Data sampling (full | 5min | daily) (default=full)
• unit_system – Unit system for data (eng | sci | cxc) (default=eng)
• interpolate_dt – Interpolate to uniform time steps (secs, default=None)
• remove_events – Remove kadi events expression (default=None)
• select_events – Select kadi events expression (default=None)
• time_format – Output time format (secs|date|greta|jd|..., default=secs)
• outfile – Output file name (default=None)
• quiet – Suppress run-time logging output (default=False)
• max_fetch_Mb – Max allowed memory (Mb) for fetching (default=1000)
• max_output_Mb – Max allowed memory (Mb) for file output (default=100)

Returns:

MSIDset object

Ska.engarchive.fetch.get_time_range(msid, format=None)

Get the time range for the given msid.

Parameters:

• msid – MSID name
• format – Output format (DateTime format, e.g. ‘secs’, ‘date’, ‘greta’)

Returns:

(tstart, tstop) in CXC seconds

Ska.engarchive.fetch.get_units()

Get the unit system currently being used for conversions.

Ska.engarchive.fetch.msid_glob(msid)

Get the archive MSIDs matching msid.

The function returns a tuple of (msids, MSIDs) where msids is a list of MSIDs that is all lower case and (where possible) matches the input msid. The output MSIDs is all upper case and corresponds to the exact MSID names stored in the archive HDF5 files.

Parameters:msid – input MSID glob Returns:tuple (msids, MSIDs)

Ska.engarchive.fetch.read_bad_times(table)

Include a list of bad times from table in the fetch module bad_times registry. This routine can be called multiple times with different tables and the bad times will be appended to the registry. The table can include any number of bad time interval specifications, one per line. A bad time interval line has three columns separated by whitespace, e.g.:

aogbias1  2008:292:00:00:00  2008:297:00:00:00

The MSID name is not case sensitive and the time values can be in any DateTime format. Blank lines and any line starting with the # character are ignored.

Ska.engarchive.fetch.set_units(unit_system)

Set the unit system used for output telemetry values. The default is “cxc”. Allowed values for unit_system are:

cxc	FITS standard units used in CXC archive files (basically MKS)
sci	Same as “cxc” but with temperatures in degC instead of Kelvins
eng	OCC engineering units (TDB P009, e.g. degF, ft-lb-sec, PSI)

Parameters:unit_system – system of units (cxc, sci, eng)

Time and Date formats

The Ska engineering telemetry archive tools support a wide range of formats for representing date-time stamps. Note that within this and other documents for this tool suite, the words ‘time’ and ‘date’ are used interchangably to mean a date-time stamp.

The available formats are listed in the table below:

Format	Description	System
secs	Elapsed seconds since 1998-01-01T00:00:00	tt
numday	DDDD:hh:mm:ss.ss... Elapsed days and time	utc
jd*	Julian Day	utc
mjd*	Modified Julian Day = JD - 2400000.5	utc
date	YYYY:DDD:hh:mm:ss.ss..	utc
caldate	YYYYMonDD at hh:mm:ss.ss..	utc
fits	FITS date/time format YYYY-MM-DDThh:mm:ss.ss..	tt
unix*	Unix time (since 1970.0)	utc
greta	YYYYDDD.hhmmss[sss]	utc

* Ambiguous for input parsing and only available as output formats.

For the date format one can supply only YYYY:DDD in which case 12:00:00.000 is implied.

The default time “System” for the different formats is either tt (Terrestrial Time) or utc (UTC). Since TT differs from UTC by around 64 seconds it is important to be consistent in specifying the time format.