.. _test_suite:

Using the ``acis_thermal_check`` Regression Testing Framework
-------------------------------------------------------------

``acis_thermal_check`` includes a regression test framework which allows one to
develop tests for a given thermal model against a set of "gold standard" model 
outputs for a number of load weeks. This section describes the test suite, how 
to run it, how to add new loads for testing, and how to update the gold standard
model answers.

An Overview of ``acis_thermal_check`` Regression Testing
========================================================

When an ``acis_thermal_check`` model is run, it produces numerical outputs for 
model prediction and validation in addition to the plots and tables on the 
webpages. The ``acis_thermal_check`` regression testing framework compares output
from model runs against a set of "gold standard" stored outputs. The idea is that 
code developments should not change the values compared to those stored in the 
gold standard, or if they do, that the reasons for the changes are understood and 
deemed necessary (e.g., you found a bug, you added a feature to a model, etc.). 
This allows us to track the effect of code changes in a systematic way and flag 
those changes which are not expected to change results but do, so that bugs can 
be identified and fixed before merging the new code into master. 

The different types of tests will now be detailed:

ACIS State Builder Tests
++++++++++++++++++++++++

The ACIS state builder constructs a history of commands and states prior to the
load using the ``backstop_history`` package. There are two basic kinds of tests 
that are run for the ACIS state builder: prediction tests and validation tests.
These are run for a number of previous load scenarios. For a given load, the 
prediction tests check the values in the ``temperatures.dat`` and ``states.dat``
files produced by the model run against those in the "gold standard" versions of 
these files. The validation tests check the values in the ``validation_data.pkl``
file produced by the model run against the ones in the "gold standard" version of
this file. 

kadi State Builder Tests
++++++++++++++++++++++++

The kadi state builder tests run the same prediction and validation comparisons
as the ACIS state builder tests, except using the kadi state builder which 
constructs the history prior to the start of the load using commands from the 
kadi database. 

Violations Tests
++++++++++++++++

Violations tests set up a thermal model run which is guaranteed to violate thermal
limits, often by setting those limits in such a way as to force a violation. It then 
checks that the violations which occur are the expected ones, including the start 
and stop times for the violations, the maximum or minimum temperatures for the 
violations, and (in the case of the focal plane model) the obsids. The "gold 
standard" answers for these tests are stored in a JSON file.

The Model Specification for Tests
+++++++++++++++++++++++++++++++++

A model specification file in JSON format is set aside for testing, and can be
different from the one currently in use for thermal models. It should only be
updated sparingly, usually if there are major changes to the structure of a 
model. For directions on how to update, see :ref:`update_model_spec`.

Running Tests
=============

Running the Entire ``acis_thermal_check`` Test Suite
++++++++++++++++++++++++++++++++++++++++++++++++++++

There are two equivalent ways to run the entire ``acis_thermal_check`` test 
suite. The first is to go to the root of the ``acis_thermal_check`` directory
and run ``py.test`` like so (this assumes you have the Ska environment 
activated):

.. code-block:: text

    [~]$ cd ~/Source/acis_thermal_check # or wherever you put it

    [~]$ py.test -s acis_thermal_check

The ``-s`` flag is optionally included here so that the output has maximum
verbosity.

Normally, the outputs of the thermal model runs used in the tests are stored 
in a temporary directory which is discarded after the tests have been carried 
out. If you want to dump these outputs to a different location for later 
examination, use the ``test_root`` argument on the command line:

.. code-block:: text

    [~]$ cd ~/Source/acis_thermal_check

    [~]$ py.test -s acis_thermal_check --test_root=/Users/jzuhone/test_outputs

You can also import the ``acis_thermal_check`` package from an interactive 
Python session and run the ``test()`` method on it to run all of the tests:

.. code-block:: pycon

    >>> import acis_thermal_check
    >>> acis_thermal_check.test()

Running the Test Suite for a Particular Model
+++++++++++++++++++++++++++++++++++++++++++++

If you only want to run the tests for a particular model, simply run the same
command, but specify the ``tests`` subdirectory appropriate to the model you
want to test:

.. code-block:: text

    [~]$ cd ~/Source/acis_thermal_check

    [~]$ ls acis_thermal_check/tests/
    __init__.py  acisfp  conftest.py  dea  fep1actel  psmc
    __pycache__  beppcb  data         dpa  fep1mong   regression_testing.py

    # Run all the 1PDEAAT tests only
    [~]$ py.test -s acis_thermal_check/tests/psmc

Running Specific Test Types
+++++++++++++++++++++++++++

If you only want to run certain types of tests for a particular model, you
can call ``py.test`` on the file containing those tests:

.. code-block:: text

    [~]$ cd ~/Source/acis_thermal_check

    [~]$ ls acis_thermal_check/tests/dea
    __init__.py  answers             test_dea_acis.py  test_dea_viols.py
    __pycache__  dea_test_spec.json  test_dea_kadi.py

    # Run the ACIS state builder tests only for 1DEAMZT
    [~]$ py.test -s acis_thermal_check/tests/dea/test_dea_acis.py

    # Run the kadi state builder tests only for 1DEAMZT
    [~]$ py.test -s acis_thermal_check/tests/dea/test_dea_kadi.py

    # Run the violation tests only for 1DEAMZT
    [~]$ py.test -s acis_thermal_check/tests/dea/test_dea_viols.py

What Happens if Some or All of the Tests Fail? 
++++++++++++++++++++++++++++++++++++++++++++++

Most warnings when running the tests are normal and benign. At this current time,
they typically look like this:

.. code-block:: text

    ../../miniconda3/envs/ska/lib/python3.8/site-packages/ipyparallel/client/view.py:8
    /Users/jzuhone/miniconda3/envs/ska/lib/python3.8/site-packages/ipyparallel/client/view.py:8: DeprecationWarning: the imp module is deprecated in favour of importlib; see the module's documentation for alternative uses
    import imp

    acis_thermal_check/tests/dpa/test_dpa_acis.py::test_prediction[MAR0617A]
    <frozen importlib._bootstrap>:219: RuntimeWarning: numpy.ufunc size changed, may indicate binary incompatibility. Expected 192 from C header, got 216 from PyObject

Any other warnings should be reported at the 
`acis_thermal_check issues page <https://github.com/acisops/acis_thermal_check>`_.

If you get errors or failures, you can investigate them by running with the 
``--test_root`` option (see above) and comparing the contents of the 
``temperatures.dat``, ``states.dat``, or ``validation_data.pkl`` files against
those in the "gold standard" answers. For example, if I wanted to investigate
failures in the 1DPAMZT model, I would first run with the ``--test_root`` option:

.. code-block:: text

    [~]$ cd ~/Source/acis_thermal_check

    [~]$ py.test -s acis_thermal_check/tests/dpa --test_root=./test_dpa_outputs

    # list the output directory
    [~]$ ls test_dpa_outputs
    acis  kadi  viols

    # list the contents of the acis state builder tests
    [~]$ ls test_dpa_outputs/acis
    APR0217B  AUG3017A  JUL3117B  MAR0817B  MAR1517B  SEP0417A
    AUG2517C  JUL2717A  MAR0617A  MAR1117A  MAR2017E  SEP0917C

    # list the contents of the acis state builder tests for 
    [~]$ ls test_dpa_outputs/acis/APR0217B
    1dpamzt.png               index.rst             states.dat
    1dpamzt_valid.png         pitch_valid.png       temperatures.dat
    1dpamzt_valid_hist.png    pitch_valid_hist.png  tscpos_valid.png
    acis_thermal_check.css    pow_sim.png           tscpos_valid_hist.png
    ccd_count_valid.png       roll.png              validation_data.pkl
    CR092_0107.backstop.hist  roll_valid.png        validation_quant.csv
    html4css1.css             roll_valid_hist.png
    index.html                run.dat

    # now check the "gold standard" answers
    [~]$ ls acis_thermal_check/tests/dpa/answers
    APR0217B  AUG3017A  JUL3018A_viol.json  MAR0617A  MAR1117A  MAR2017E  SEP0917C
    AUG2517C  JUL2717A  JUL3117B            MAR0817B  MAR1517B  SEP0417A
    
    # Check the answers for the APR0217B load for either the ACIS or kadi tests
    [~]$ ls acis_thermal_check/tests/dpa/answers/APR0217B
    states.dat  temperatures.dat  validation_data.pkl

You can use Python or a diffing tool to check the ``states.dat`` or 
``temperatures.dat`` files, and you can use python to check the 
``validation_data.pkl`` file.

If you want to check the violations tests, then look at the values in the (say)
``JUL3018A_viol.json`` file and compare them to the ``index.rst`` file generated
under the ``viols`` directory where you specified the ``--test_root``:

.. code-block:: text

    # Check for the violations JSON file for JUL3018A
    [~]$ ls acis_thermal_check/tests/dpa/answers
    APR0217B  AUG3017A  JUL3018A_viol.json  MAR0617A  MAR1117A  MAR2017E  SEP0917C
    AUG2517C  JUL2717A  JUL3117B            MAR0817B  MAR1517B  SEP0417A

    # Check for the index.rst file run by the violations test
    [~]$ ls test_dpa_outputs/viols
    JUL3018A
    
    [~]$ ls test_dpa_outputs/viols/JUL3018A
    1dpamzt.png               index.rst             states.dat
    1dpamzt_valid.png         pitch_valid.png       temperatures.dat
    1dpamzt_valid_hist.png    pitch_valid_hist.png  tscpos_valid.png
    acis_thermal_check.css    pow_sim.png           tscpos_valid_hist.png
    ccd_count_valid.png       roll.png              validation_data.pkl
    CR211_1004.backstop.hist  roll_valid.png        validation_quant.csv
    html4css1.css             roll_valid_hist.png
    index.html                run.dat

Updating the "Gold Standard" Answers
====================================

New "gold standard" answers for a given model may need to be generated for two
reasons. First, you may be making a new model and need to generate the initial 
set of answers. Second, if you are updating ACIS code and the regression tests 
failed to pass for one or more models, but the failures are understood and they 
are due to changes you made which need to become part of the software (such as 
a bugfix or a feature enhancement), then the "gold standard" answers need to be
updated. 

To generate new answers for all of the models, go to the root of the 
``acis_thermal_check`` directory that you are working in, and run ``py.test`` 
with the ``--answer_store`` argument like so:

.. code-block:: text

    [~]$ cd ~/Source/acis_thermal_check

    [~]$ py.test -s acis_thermal_check --answer_store

This will overwrite the old answers, but since they are also under git version 
control you will be able to check any differences before committing the new
answers. 

If you want to overwrite the answers for a single model, simply run the same
command, but specify the ``tests`` subdirectory appropriate to the model you
want to update:

.. code-block:: text

    [~]$ cd ~/Source/acis_thermal_check

    [~]$ ls acis_thermal_check/tests/
    __init__.py  acisfp  conftest.py  dea  fep1actel  psmc
    __pycache__  beppcb  data         dpa  fep1mong   regression_testing.py

    [~]$ py.test -s acis_thermal_check/tests/dpa --answer_store

.. _update_model_spec:

Updating the Model Specification File
=====================================

If you need to update the model specification file, simply replace the current
version of the file in its respective directory:

.. code-block:: text

    [~]$ cd ~/Source/acis_thermal_check

    [~]$ ls acis_thermal_check/tests/acisfp
    __init__.py  acisfp_test_spec.json  test_acisfp_acis.py  test_acisfp_viols.py
    __pycache__  answers                test_acisfp_kadi.py

where in this case ``acisfp_test_spec.json`` is the file you want to replace. 

.. _adding_new_test_loads:

Adding New Loads for the Prediction and Validation Tests
========================================================

The tests of the prediction and validation outputs for the thermal models
run on a series of past loads. These are stored in a ``test_loads`` dictionary
in the ``regression_testing.py`` file in the ``acis_thermal_check/tests`` 
directory:

.. code-block:: python

    # Loads for regression testing
    test_loads = {"normal": ["MAR0617A", "MAR2017E", "JUL3117B", "SEP0417A"],
                  "interrupt": ["MAR1517B", "JUL2717A", "AUG2517C", "AUG3017A",
                                "MAR0817B", "MAR1117A", "APR0217B", "SEP0917C"]}
    all_loads = test_loads["normal"]+test_loads["interrupt"]

    nlets = {"MAR0617A", "MAR0817B", "SEP0417A"}

Note that the dictionary is split up into two lists, one for ``"normal"`` loads
and another for ``"interrupt"`` loads, the latter including loads which begin
after an interrupt of any kind, including safing actions, radiation replans, ToOs,
etc. If you want to add new loads to the list to be tested, simply add them to
either the ``"normal"`` or ``"interrupt"`` list as appropriate. 

Note also the ``nlets`` set below this dictionary. The standard NLET file in 
use for creating histories may not always be the "correct" one for the test 
load in question, since at the time the model was originally run for the load
review the contents of the NLET file at that time may not have included events 
which happened later. If you want to recreate the situation at the time the
model was originally run as precisely as possible, specify the name of the load
in the ``nlets`` set and add the NLET file to the 
``acis_thermal_check/tests/data/nlets`` directory, with the naming convention
``"TEST_NLET_{load_name}.txt"``, where ``"load_name"`` is of the form``"MAR0617A"``.