~cvs/scrib/doc/ic_regression.html

# Regression G

Regression G is an in-house ScribApp instance for testing modules with replay files through the med_batch interp.

## Overview

1. Starting Regression G
2. Navigating the GUI
3. Performing the regression tests
4. Viewing the results
5. Creating a test case
6. Known bugs
7. Future Plans

## Starting Regression G

### Command shell

To run 'Regression G', you'll wan't to:

1. `> su icemcfd`
2. `> cd /home/josie/ic_regression`
3. `> ./go.tcl`

NB: `go.tcl` includes a few CVS commands which update the `./application` directory, so it is ideal to be working on a machine for which these CVS commands work, in order that the application updates itself properly.

NB: it may be that you have to start up a "med server" on the remote testing machines. Do this something like:

- `> rlogin gemini`
- `> su icemcfd`
- `> $ICEM_ACN/lib/batch_queue/icbqs_serverd`

  ### Application settings

  You are now in a ScribApp GUI, being asked for a "application settings."

  Either select a file you have saved before or, if you have not saved your own settings yet, load `jasons_regression_settings.bsm`. Please do not do save your settings to that file.

  back to overview

  ## Navigating the GUI

  There are three tabs in the UI:

  - "Regression settings"
  - "Select models by modules"
  - "Select models by functions called"

  ### Notebook tab: "Regression settings"

  Testing ICEM_ACN base: if not specified, it will inherit from your startup shell. It is 'base' because it will be modified internally foreach platform specified, necessarily.

  Base regression test models directory: this is likely '`/home/josie/ic_regression/db`' such that the dirs below it are organized by module.

  Regression output directory: into this dir go logfiles and XML/HTML reports generated by the "Regression G" program. A reasonable value is '`/home/josie/ic_regression/regression_out`'.

  Testing site: This is for mapping platform names to hostnames. Not sure if this works yet. Tell Jason if you try it and encounter any problems. NB: Do not distribute the "regression_hosts.tcl" file

  Platforms: Select platforms. This listbox gets a little confused, especially when a settings file loads data into it.

  Report problems to maintainer: this will send an email with the log to the stored maintainer. The stored maintainer is likely wrong, likely me.

  Perform comparisons: this will use Forest's mesh comparison stuff. It doesn't really work just yet. Turn it off.

  Use time profiling: this will compare the execution time to the benchmark time in the model preferences file.

  ### Notebook tab: "Select models by modules"

  Use this tab to choose models by module. Changes this the tree in this tab should be reflected elsewhere (e.g. in the "Select models by functions called" tree).

  ### Notebook tab: "Select models by functions called"

  Use this tab to choose models by ic_* commands organized either by (i) the "representative commands" in the model-prefs, or else (ii) all those harvested from the mode's specified replay file. Changes in this the tree in this tab should be reflected elsewhere.

  ### Left-column button: "Test all models"

  Test all models in specified database, irrespective of the selections in the trees.

  ### Left-column button: "Test selected models"

  Test the models currently "checked" in the trees. The trees share data, so settings in one tree might override previous settings in the other.

  ### Left-column button: "View *ML reults"

  You will be asked to select either an HTML or XML results file from a previous for display in a browser (mozilla or netscape on *nux, IE on Windows).

  back to overview

  ## Performing the regression tests

  Press either "Test all models" or "Test selected models". Messages will be printed, lots of them. After each model has been run on each platform, you will be presented with an information box with the simple results (number failed, number successful, number with weird error).

  The trees in the "Select models by modules" and "Select models by functions called" notebook trees will be updated with the result of reach test. A "check" will be placed beside successful tests, and the "no" (as in "No smoking" or "Ghostbusters") beside the failed tests. Those tests will be unselected for future runs although their icon will not change. You can reactivate them by clicking (some number of times) on the icon until it displays a check-mark.

  back to overview

  ## Viewing the results

  After the tests have completed, you will be presented with a dialog asking whether you'd like to view the the results in XML or HTML. (If you do not view the results right away, the file names will be be printed to the "Execution messages" window, and you can use the "View *ML reults" button later.)

  The XML format (writen by ~cvs/ic_xml) is that of HQ's TestBench. It's not necessarily valid XML, but we can still parse it. The HTML format is an XSLTransform of the XML. The XSL needs work for good looking HTML with CSS.

  back to overview

  ## Creating a test case

  The regression database directory structure looks something like this:

      - db dir
        |
        - $module1 (e.g., tetra, hexa, etc.)
        | |
          | - $test_model1 (e.g., "test_model_001")
          | | |
          | | |- regression_prefs.tcl (note importance of name)
          | | |- $input_file
          | | |- $standard_output_file
          | |
          | + $test_model2
          |
          + $module2
          |
          + $module3

  Each "db>$module>$test_model" directory should contain a file called `regression_prefs.tcl`. Everything else is fairly flexible.

  ### The regression_prefs.tcl preferences file

  It is essentially a TCL array, and gets source as such.

      #
      # This file configures the Regression G Test Model
      #
      array set regression_prefs {

          representative_cmds      "ic_run_tetra"

          maintainers     "jason"

          input_file      "rotational_period.tin"
          replay_file     "med.replay"
          output_type     "mesh"
          output_file     "test_output.uns"

          std_output_file "std_output.uns"
          std_exectime    90

          comparison_type "fuzzy"
          max_time        12000
      }

  ### Med replay file

      #
      # This script runs the test.
      # NB: the variables imported to it: $input_file, $output_file, $log_file
      #
      set log_file "tetra_cmd.log"
      #
      ic_load_tetin $input_file 0 1
      ic_run_tetra $input_file $output_file run_cutter 1 run_smoother 1 smoother_n_iter 5 smoother_aspect 0.4 n_processors 2 log $log_file
      ic_unload_mesh
      ic_unload_tetin

  Not the use of special variable names in the replay file, the setting of which is respects the preferencfces file. This should allow scripts and preferences files to be more flexible.

  In this case, after the script `$replay_file` completes, the `$output_file` would be compared to the `$output_type` with the `$comparison_type` file `$std_output_file` via the `$comparison_type` method. If the script fails, or the comparison fails, or the script takes longer to execute than `$std_exectime`, then the test returns FAILURE. If the "Report to maintainers" option is on, then an email with the logfile will be sent to each of the `#maintainers`.

  In light of the way Regression G harvests `ic_*` commands from the replay file in the test-model, you can control the profile of the model (and speed up application startup time) by setting the `representative_cmds`. In the case of this variable, all commands except `$representative_cmds` will be ignored in building the "Test selected models" tree.

  NB: only files which exist in a test-model directory prior to the test will remain after script execution: everything else should be removed. You can always check logs in your "Regression output directory" directory, where *ML results files are also saved.

  back to overview

  ## Known bugs

  - tDOM, the Tcl-Expart-XML parser used in `ic_xml` has not been installed everywhere yet, rather only on ,tt>linux and `win40`.
  - Note also the fixme elements in this document.

  back to overview

  ## Future Plans

  - Fix `form_tree` grid problem in `scrib_app_data_tree`. Right now, width and height are hardcoded. Let me know if you run into trouble with this.
  - Improve XSL formatting of XML files to HTML. I've had trouble getting the `dom xslt` to abide by my CSS.

  back to overview

  Author: Jason K. Aftosmis (jason at icemcfd dot com)
  Copyright (c) 1993-2004 ANSYS/ICEM CFD Engineering
  $Revision$ on $Date$