batch_queue/doc/readme.html

# batch_queue

The purpose of this pckage is to simply and transparently communicate with any type of a number of batch-queueing-systems simultanesouly managin multipleqeues of various types, if necessary.

## Overview

1. Setup
2. Supported Batch-Queueing-Systems
3. Creating the Library Documentation
4. Using the Interface
5. The IcBQS Implementation
6. Adding Implementations
7. Current usage
8. Testing the module
9. Distribution
10. History
11. Known bugs
12. TODO & Future Plans

## Setup

To setup this module/package, source `batch_queue/setup.tcl` in your Tcl interp. If successful, it will provide the tcl-package `batch_queue` and the version will be the CVS-revision string of `batch_queue/lib/generic.tcl`.

To start a server, use the shell script `icbqs_serverd`, or do something like `nohup tclsh icbqs_server.tcl &`.

back to overview

## Supported Batch-Queueing-Systems

- IcBQS (see the section on this module)
- PBS (see `batch_queue/lib/pbs.tcl`). For more info on our OpenPBS account, see batch_queue/doc/history.txt.
- Apple's XGrid (well, not really fully impl'd; see `batch_queue/lib/xgrid.tcl` and the files in `doc/xgrid/`.)
- Suns' GridEngine (well, not really impl'd; see `batch_queue/lib/gridengine.tcl`)
- Ansys (well, not really; just a placeholder since I thought MBU had some usable stuff; see `batch_queue/lib/ansys.tcl`)

back to overview

## Creating the Library Documentation

Documentation for all of the procs in the `batch_queue/lib` dir (created at `batch_queue/doc/library_documentation.html` by default) can be created with:

        cd /path/to/batch_queue
        make libdoc
        $your_browser ./doc/index.html

or:

        cd /path/to/batch_queue
        make readdoc

which will both call the `libdoc` target, and load this current file into `lynx` (on Unix).

Note that the `libdoc` target assumes that `guibase` module is installed at the same level as `batch_queue` module, and it need the proc `makedoc`. Also, that target uses the file `create_documentation.tcl` found in this `doc` dir.

back to overview

## Using the Interface

### batch_queue/lib/generic.tcl

The public methods of the `batch_queue` namespace are (in approximate order of normal call):

- `batch_queue::init_queue`
- `batch_queue::submit_job`
- `batch_queue::exec_queued_jobs`
- `batch_queue::queue_summary`
- `batch_queue::delete_queue`
- `batch_queue::fixme`

NB: Documentation for these and all other procs and be created with the "make libdoc" command in the batch_queue directory. See the section on creating library documentation

A very good place to do check out proc usage is in the `tests/test_all.tcl` file. Please go there!

Ideally, all calls to this module should go through a `batch_queue::*` proc, but I have thus far left it that one can access particular implementation, but this sacrifices the ability to track queues and jobs nicely.

back to overview

## The IcBQS Implementation

This is our own (fast, light, low-funcationality, self-contained) BQS, the interface to which resides in `batch_queue/lib/icbqs.tcl`, the pubic interface for IcBQS.

There are three files for the IcBQS implementation:

- `batch_queue/lib/icbqs_server.tcl` manages commands on the server-side
- `batch_queue/lib/icbqs_client.tcl` has procs for talking to a server
- `batch_queue/lib/icbqs.tcl` has the batch_queue_icbqs_* procs, IcBQS constituting the public-interface.

To start up a server, do something like:

          tclsh batch_queue/lib/icbqs_server.tcl -port_start 6790 -port_end 6792 -valid_commands {$commandone $commandtwo}

or:

         icbqs_serverd -port_start 6790 -port_end 6792 -valid_commands {$commandone $commandtwo}

thought the -valid_commands arg is not really supported.

Some useful commands are:

- `icbqs_server_how_busy_are_you`
- `icbqs_server_print_data`
- `icbqs_server_help`
- `icbqs_server_quit`

NB: Documentation for these and all other procs and be created with the "make libdoc" command in the batch_queue directory. See the section on creating library documentation

To set up a client to it, do something like:

          icbqs_client_setup -server_port 6790 -servers [list $server1 $server2]

Other than that, everyhing else you need to use IcBQS should (be able to be) done through batch_queue_icbsqs_* commands. For those commands, see the section on creating library documentation.

With the IcBQS module, it is possible that to ftp files from the client over to the server. This is accoomplished with the icbqs_client_ftp_* commands, which rely on tcllib/ftp/ftp.tcl.

back to overview

## Adding Implementations

Each BQS should get its own implementation file in `batch_queue/lib` for procs that implement the interface proc in the `generic.tcl` file.

back to overview

## Current Usage

Some current usages are found in:

- AI*Batch surf (in `scrib` module)
- Regression G (in `scrib` module)
- IcePak (coming soon)

back to overview

## Testing the Module

To use the test modules living in the `tests` directory, do:

           make test

If testing IcBQS, it will not set up the server daemons for you, rather you must do that yourself.

back to overview

## Distribution

We don't want to distribute this document in installations; we especially don't want to distribute `doc/history.txt`, and I think we should omit the `doc` directory altogether.

back to overview

## History

See batch_queue/doc/history.txt for related correspondence. In that file, you'll find our password to the OpenPBS site!

back to overview

## Known bugs

- There are certainly many...see the fixme elements in this document.
- Also, do:

      cd /path/to/batch_queue/lib
      grep -i fixme *.tcl

  for potential problem and blatant holes!

back to overview

## TODO & Future Plans

- Create procs to wrap remote shell script execution & file manipulation, then restrict the executable commands on the remote hosts so that only native icbqs_* procs can be executed. Just in case a socket gets hijacked, you know.
- Improve the ability to monitor resources on the servers, making sure the `ps` works on all plats. A `ps` Tcl extension or package seems like a good idea.
- Improve the error and signal handling on the servers.
- Improve the icbqs_serverd daemon. Note this page http://wiki.tcl.tk/daemon. It should be easier to the user to start up the daemons on the remote machines.
- Set up the FTP for cases where the remote host does not shared NFS mounting of the working directory, etc.
- Use TLS for secure OpenSSL sockets. See http://www.sensus.org/tcl/tls.html (which is out of date) and other options listed also at http://expect.nist.gov/tkbiff/doc-secure.html).
- Figure out how to support wild things like Globus, etc., I suppose. Note things like pyGlobus.

back to overview

Author: Jason K. Aftosmis (jason at icemcfd dot com)
Copyright (c) 1993-2004 ANSYS/ICEM CFD Engineering
$Revision$ on $Date$