Load Review regression testing

Setup
Preview

Scenarios:
lr trio
Sanity check
Backstop only
Replan cases
Health and Safety script

2.0 Updating regression tests

This page describes the process for conducting a regression test for ACIS load review scripts.

0.1 Setup

It is assumed that you have already logged in as acisdude and established the ACIS load review operational environment with a setloads command. Extend the environment with

% setloads
% source lr_test_source
% cd $REGRESS
You'll find yourself on the directory /pool14/duderg/regress.

0.2 Quick preview

To run a single regression case casename, you'll issue the commands:

% gmake chkout
(... a step to set up links to script(s) under test...)
%gmake links
% gmake CASE=casename casename
At the sftp prompt, get the correct backstop file copied into the test load review directory:
sftp> ^Z
% gmake CASE=casename backstop
% fg sftp> exit
The casename target will run a gmake diff step. That in turn will display either a ***PASSED*** or a ***FAILED*** on the log file $REGRESS/<casename>_regr.log.

We will discuss four scenarios, all of which look roughly like the one above:

  1. The lr trio. Runs lr in its entirety on three historical cases selected for variety of activities. If your terminal supports the "bell" character, you can reduce your typing load by invoking 'gmake lrq' rather than 'gmake lr'.
  2. Sanity check. The scenario to use after your changes to load review executables have been checked into production.
  3. Backstop only check. This scenario includes the same cases as the first, but runs backstop directly, which cuts down on your required interaction. It adds three cases which incorporate multiple deliberate backstop file errors, in order to ensure proper error messages are issued for checklist violations.
  4. Health and safety. Runs just the health and safety script on the three cases.
NOTE: One case, oct2811x, is not presently built into any scenario. It will become part of the lrq target. It exercises a few checklist items, and you can invoke it as a single case.

We'll go over the steps in some detail for the first scenario, noting only differences for the other three. If changes involve new scripts, or changes to script names, refer to the "SPECIAL CASE" within the lr trio discussion.

1.0 The lr trio

  1. CVS extraction.
    % gmake chkout
    This checks out all the files needed for the regression tests into pool14. This is a volatile disk area, so they won't necessarily still be there the next time you go to run a regression. You can view them all at $REGRESS/../src/regress.
  2. Set up links to new code. The regression tests look to the directory /data/acis/acisdude/reg_script to find all the scripts they will run. "gmake chkout" cleared everything out of this directory, via a dependency on the gmake target cleanlinks.

    For each script that you want to test, set up a soft link in reg_script to the new code. For example, if you had made changes just to acis-backstop.pl and saa_checker.pl, you'd do

    % pushd /data/acis/acisdude/reg_script
    % ln -s /data/acis/LoadReviews/acis-backstop-V12.20.pl acis-backstop.pl
    % ln -s /data/acis/LoadReviews/saa_checker-V45.5.pl saa_checker.pl
    % popd (back to $REGRESS)

    SPECIAL CASE IF YOU HAVE INTRODUCED A NEW SCRIPT, or changed the name of an old one: You will have to update the regression Makefile so that it will create the appropriate softlinks in the following "make links" step. See "Regression Makefile updates" near the tail end of this document.

  3. Tricky but invisible linking. The Makefile will set up all the other links for you.
    %gmake links
    For scripts you haven't specified, this target will set soft links to the production versions. For lr, it will replace either your link to the code under test, or its own link to the production version, with a copy of lr in which the hard-coded pathnames to the production script area are swapped for pathnames to reg_script. It will do something similar for acis-backstop.pl, creating a copy just like the one you wanted to run, except that it will now look at a historically correct version of the .tln file to get its information about the parameter blocks called by the SI modes.
  4. Run the lr cases.
    % gmake lr

    This will run lr three times, once for each of our historical cases. Each of these lr's will conduct its business on the directory /data/acis/LoadReviews/yyyy/RG_mmmddyy/oflsL, where the letter L is taken from the last character of the casename, and the mmmddyy from its first 7 characters.

    As each case runs, lr will ask you for your username and password to lucky. Fill these in as you ordinarily would. Rather than a "get" command to sftp, though, you'll follow the instructions which will appear on your screen a little bit above the SFTP to lucky prompt.

    sftp> ^Z
    % gmake CASE=casename backstop
    % fg
    [two carriage returns, to get the sftp prompt back]
    sftp> exit

    When lr completes each case, the Makefile target will execute
    gmake CASE=casename diff,
    comparing the outputs you got with the expected ones. It looks in detail at the history files and ACIS-LoadReview.txt. At this point, it only checks for the existence of the rest. So if you've updated one of the subsidiary models, keep an eye out for error messages on the screen.

    ALTERNATE STEP 4: Quick lr.
    You can bypass most of the typing involved in this scenario by typing

    % gmake lrq
    This will start a python program which will take control of the lr script's stdin and stdout, and do almost all of your keyboard inputs for you. You will be asked once for your username on lucky. All the cases will then run automatically, except that once for each case you will be asked to input your password on lucky.

    CAUTION: You will have a limited time to enter your password, after which sftp will time out, and you'll have to start everything over. A few seconds before the password is requested, three ASCII bell characters will be issued to the terminal. If your terminal can't sound a beep, you'll have to watch the screen carefully for prompts.

  5. Examine diff output

    The gmake diff step will display either a ***PASSED*** or a ***FAILED*** on the log file $REGRESS/<casename>_regr.log. on your screen. There will be one of these for each of the three historical cases. If any of the comparisons have failed, examine the file $REGRESS/<casename>_regr.log. It will contain the full output of all the diffs performed for that case, concatenated together, the expected lines denoted by a "<" and the actual output denoted by ">".

    These outputs are consolidated into two summary files:

    qwik_regr.log, which displays only the failing tests, and
    qwik_alldiffs, which displays all the differences.
    Header records in qwik_alldiffs indicate the source of each set of diffs, showing both the case and the two files under comparison. Caution: If the scenario ended prematurely, so not all cases were run, inspection of these two files won't indicate a problem.You can check for this condition with a 'ls' command on the $REGRESS directory. Verify that every expected <case>_alldiffs file is present, even if a successful test rendered it empty.

  6. Update the regression comparison files. To be described. Leave this to Royce for now. The test itself is complete without this step.

1.1 Sanity check

This is the scenario that you will run after installing previously tested new code in production. Its sole function is to guarantee that something hasn't been badly busted, so it's content with running a single load. Because proper linkages out of lr are one of its central concerns, it does run lr, even if supposedly only backstop has changed.

It's also truly simple. Since there is no separate test version of any script to point to, there's no links setup.

  1. If you're not still in a previously existing regression session, set up. Issue a setloads and source lr_test_source.
  2. % cd $REGRESS
    % gmake sanity
  3. Follow onscreen instructions as in scenario one, to get the backstop file in place.
  4. Inspect the .log file on the $REGRESS directory. There will be just one.

1.2 The backstop only regression

If the only change is to acis-backstop.pl, the regression task is a lot simpler than it is for lr.

  1. Set up:
    % setloads
    % source ~/lr_test_source
    % cd $REGRESS
  2. Prepare the scenario
    % gmake realclean
    % gmake chkout
    This will clean up any leftovers, and re-check everything out of CVS, and set up initial links to executables. A shortcut with the same effect is:
    % gmake start
  3. Point to code under test If your new backstop source code is on directory path /when/pigsfly,
    % pushd /data/acis/acisdude/reg_script
    % rm acis-backstop.pl
    % ln -s /when/pigsfly/acis-backstop.pl acis-backstop.pl
    % popd % gmake links
  4. Invoke the scenario
    % gmake backstops
  5. Inspect the output Each case will have its summary output file <casename>regr.log. If every pair of lines includes a ***PASS***, the run was a success. If any includes a ***FAIL***, look at <casename>alldiffs, which concatenates the results of diffs on all the compared files. (See the discussion at the end of the lr trio text for a recommended shortcut.)

This gmake target will avoid lr, and run just the backstop script on the three historical cases. None of that pesky sftp'ing and gmake backstopping when you don't really need it. Then it will run the error coverage cases: aug0210x and aug0210y.

1.3 Replan

% gmake replan

This runs five replan scenarios, and it's very similar to the lr trio (1.1). However, you have to provide a little more input at each step.

As each case begins, you'll see not only a "gmake CASE=" line, but two other pieces of information:
How to respond to Y/N queries (with a 1 or a 2 in each case)
The start time for the load
Note the query responses. You will enter the "gmake CASE=" line, as for the lr trio. Before you type "quit" to the sftp prompt, cut the text for the start time. Then type "quit", as for the lr trio.

a series of Y/N questions will appear. Answer with 1 or 2 as you were instructed. When queried for the start time, paste your saved text.

The remainder of this scenario will run in just the same way as the lr trio did.

1.4 H&S Only

Run this scenario when there's been a change to LRHS.

  1. Set up If you're not still in a previously existing regression session, issue a setloads and source lr_test_source.
  2. Clear old runs, and get fresh CVS checkout
    % cd $REGRESS
    % gmake start
    This will clean out any old regression test files, and check the latest ones out from CVS.
  3. Point to the code under test. Establish a soft link to your test code version of LRHS, say for instance /somewhere/LRHS_Test.py:
    % pushd /data/acis/acisdude/reg_script % ln -s /somewhere/LRHS_Test.py LRHS % popd % gmake links
  4. Run the scenario
    % gmake hands
  5. Inspect the output Each case will have its summary output file <casename>_regr.log. If every pair of lines includes a ***PASS***, the run was a success. If any includes a ***FAIL***, look at <casename>alldiffs, which concatenates the results of diffs on all the compared files, in this scenario just ACIS-LoadReview_HandS.txt. (See the discussion at the end of the lr trio text for a recommended shortcut.)

2.0 Regression Makefile updates

Various "gmake clean" targets clear out the $REGRESS directory, and copy back in a fresh copy of the Makefile from
/data/acis/LoadReviews/script/lr_regress_Makefile
If you need to alter the Makefile temporarily, save a copy of the current lr_regress_Makefile, place your desired version on LoadReviews/script, run your tests, and put the original back in place.

If you're making a permanent change, alter the copy on LoadReviews/script, run your test, and once satisfied that the new Makefile is working as intended, go over to $REGRESS/../src/regress, copy the new makefile to replace lr_regress_Makefile there, and check it into CVS:
% cvs commit lr_regress_Makefile
In particular, if you've added or changed the name of an executable, add or modify the corresponding softlink line under the target "links":
-@cd $(RSCRIP); ln -s $(SCRIP)/<scriptname> /<scriptname>
The code just cited assumes the new script, or a link to it, is in place on LoadReviews/script. If it resides elsewhere, insert that path in the place of $(SCRIP).

3.0 Known bugs and limitations

At this juncture, the regression examines outputs only for lr, acis-backstop and H&S, and only for non-interrupt cases.

The Makefile generates numerous (ignored) error messages. Be brave and ignore them. It also generates a lot of chatter about what it's doing. Feedback on what parts of this chatter to suppress is welcome.