=head1 NAME

CFX5::WorkDir - Keep a working directory up to date

=head2 VERSION

  $Id$

=head1 SYNOPSIS

  $dir=new CFX5::WorkDir('static_mixer_001.dir');
  $dir->set_basename('static_mixer_001');
  $dir->input_file('static_mixer.def' => def);
  $dir->mark_output_file('prt');
  $dir->mark_output_file('cfx5.mms' => 'static_mixer_001.mms');
  $dir->enter();
  # ... run the solver ...
  $dir->leave();
  $dir->output_file_done('1.trn' => 'static_mixer_001/1.trn');
  $dir->output_marked_files_done();
  $dir->junk_directory();

=head1 DESCRIPTION

This module is an abstraction of the sort of working directory used by
the solver, which is created, has files linked into it and/or created
in it, has one or more processes run in it, and then has various files
copied or moved out of it again.

=head1 DESIGN

=head2 MOTIVATION

This module was first conceived during the creation of the new-style
"restructured" solver script, L<CFX5::Job>.  The need was seen to have
an abstracted way to handle a working directory, in order to cut down
on code complexity and duplication, and to make it possible to pass
information about the state of a working directory in a simple manner
between functions.

It should not be limited to that one task: the intent of the module is
to provide a reasonably generic representation of a working directory
which can transparently handle different implementations and
architectures.  This original motivation will affect the functionality
which is implemented first, however.

=head2 EXTENSIONS

The module as presented here is far from complete.  Some extensions which have already been discussed, and which it would be nice to add, include:

=over

=item Delete file call

=item Stacked enter/leave

So one can do something like

  $dir->penter();
  $dir->penter();
  $dir->pleave();
  # still in there
  $dir->pleave();
  # now back out again

which would make writing modules which run in an existing WorkDir
object much easier.

=item history of a file

=item support for directories as well as files

=item "stacked state"

Currently, the "output_marked_files" calls can only really be used for
one set of files.  It would be nice to allow some way of having
different sets of these, so a solver run can set itself up to copy out
the files it wants to but still allow the refiner already to have set
this up.  Or something.  This one needs a little more thought ...

=back

=cut

package CFX5::WorkDir;
use strict;

use CFX5::Output qw(:all);
use CFX5::PathManip qw(split_file join_file fullpath relative_path
                       normalize_filename is_absolute change_cwd);

use CFX5::FileOps qw(create_directory create_empty_file
		     copy_file_for_read copy_file 
		     move_file move_file_to_backup make_writable );
use CFX5::FileStat;

use File::Copy;

# for the compiler; this is needed by File::Copy, but it's done
# dynamically.  Also we make sure we don't pick up the definitions of
# dirname and basename from File::Basename.
BEGIN {
  require File::Basename;
  *File::Copy::basename = \&File::Basename::basename;
  *File::Copy::dirname = \&File::Basename::dirname;
}


=head1 CALLING CONVENTIONS

=head2 FILE SPECIFICATION ARGUMENTS

In general, a file can be specified with either one or two strings,
usually as an optional argument.  If two strings are given, the first
is the "local" name inside the working directory, and the second is
the full name, which is taken relative to the starting directory if it
is a relative path.  

If only one string is given, the behaviour depends on whether the file
already has a "full name" associated with it, for example from a
previous two-string specification.  If one already exists this is
used; otherwise, the string is used as a suffix for the basename (see
C<set_basename>) to construct the full filename.  In either case, the
single string always specifies the "local name" of the file.

For example, the (somewhat artificial) call

  $dir->input_file('res' => 'static_mixer_001.res');

can be abbreviated as

  $dir->input_file('res');

If it is the first reference to the definition file, one cannot abbreviate

  $dir->input_file('def' => 'static_mixer.def');

but one can then happily (if strangely ...) use the call

  $dir->output_file_update('def');

to update it.

=head1 CONSTRUCTORS

=head2 new

  $dir=new CFX5::WorkDir 'static_mixer_001.dir';

Creates a new C<CFX5::WorkDir> object, and the directory which goes
with it.  If the directory already exists, or cannot be created,
signals an error and returns "undef".

=cut

sub new {
  my $this = shift;
  my $class = ref($this) || $this;
  my $self = {};
  bless $self, $class;
  $self->initialize(@_) or $self=undef;
  return $self;
}

sub initialize {
  my $self=shift;
  my $dir=shift;

  $dir=fullpath($dir);

  unless (create_directory($dir)) {
    _error("Unable to create $dir: $!");
    return 0;
  }

  $self->{dir}=$dir;
  $self->{'exists'}=1;

  $self->{files}={};
  
  1;
}

=head1 CONFIGURATION

=head2 set_basename

  $dir->set_basename('static_mixer_001');

In general, the files in this directory are assumed to persist in the
form I<basename>.I<filename>, where I<filename> is the full name of
the file as it exists in the directory; this function sets
I<basename>.  

=cut

sub set_basename {
  my $self=shift;
  $self->{basename} = $_[0];	# whether or not it's defined
}

=head1 INTERNAL FUNCTIONS

=head3 dir

  my $path=$self->dir();

Returns the full path of the working directory.

=cut

sub dir {
  my $self=shift;
  return $self->{dir};
}

=head3 file_list

  my @files=$self->file_list();

List is not ordered.

=cut

sub file_list {
  my $self=shift;
  return keys %{$self->{files}};
}

=head3 file_extname

  my $ename=$self->file_extname('def');

Returns the external name of a file (full pathname).

=cut

sub file_extname {
  my $self=shift;
  my $f=shift;
  
  # do we know about this file?
  exists $self->{files}->{$f} or return undef;
  
  # do we already have an external name for it?
  if (exists $self->{files}->{$f}->{extname}) {
    return $self->{files}->{$f}->{extname};
  }

  # no, do we have a basename?
  if (defined $self->{basename}) {
    return fullpath($self->{basename}.".$f");
  }

  # no luck today
  return undef;
}

=head1 FILE INTRODUCTION

The functions in this section are used to introduce existing files to
the directory, in various different ways.  No notice is taken of
whether the files are to be moved or copied out again; fortunately,
this information is not needed at this stage.  (Yay!)

The files which are introduced to the working directory are assumed to
be in one of the following classes:

=over

=item Simple input files

This is the simplest class, it simply means that this file is read by
whatever runs in this directory, and should be made available under
the requested name.  By marking a file as a simple input file, the
programmer is saying either:

=over

=item *

that the file will never be written to

=item *

that the file may be written to, but it does not matter whether the
original file is updated concurrently or not

(This is almost invariably because she will copy it back to the same name at
the end of the run).

=back

=item Input files which need copying

If a file is both read from and written to, but the source file should
not be changed, the file may be explicitly copied into the directory.
This should be avoided if possible for large files.

=item Files which are already in the working directory

Sometimes it is necessary to have the same file in the working
directory appear under different names at different times, for example
when the results file from the solver is to be used as a def file for
the interpolation.  The files then need to be renamed inside the
directory, without losing information as to their source.

=item Files which are to be created directly

For example, the definition file in a run with initial values is a
combination of the def file and the ini file.  This file does not have
an existance outside the temporary directory.

=back

If the file in the working directory already exists, these functions
will go ahead and overwrite it with the new one.  If you want to be
more careful you can check yourself before calling the function.

=head2 input_file

  $dir->input_file('par' => 'static_mixer.par') or die;

Simple input file.

=cut

sub input_file {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  my $details = exists $self->{files}->{$int} ? $self->{files}->{$int} : {};

  if (defined $ext) {
    # user-specified
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  } else {
    $ext = $self->file_extname($int);
  }

  unless (defined $ext) {
    _error("Cannot create working file $int: No external name known");
    return undef;
  }

  _debug("$int <= $ext");

  $self->{files}->{$int} = $details;

  if (test_e(join_file($self->{dir},$int))) {
    unlink join_file($self->{dir},$int);
  }
  unless (copy_file_for_read($ext => join_file($self->{dir},$int))) {
    _error("Cannot create working file $int (copying from $ext):\n  $!");
    return undef;
  }
  1;
}

=head2 input_file_copy

  $dir->input_file_copy('def' => 'static_mixer.def') or die;

Makes a copy of the input file.

=cut

sub input_file_copy {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  my $details = exists $self->{files}->{$int} ? $self->{files}->{$int} : {};

  if (defined $ext) {
    # user-specified
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  } else {
    $ext = $self->file_extname($int);
  }

  unless (defined $ext) {
    _error("Cannot create working file $int: No external name known");
    return undef;
  }

  _debug("$int <= $ext");

  $self->{files}->{$int} = $details;

  if (test_e(join_file($self->{dir},$int))) {
    unlink join_file($self->{dir},$int);
  }
  unless (copy_file($ext => join_file($self->{dir},$int))) {
    _error("Cannot create working file $int (linking from $ext):\n  $!");
    return undef;
  }

  # make sure we have write permission
  unless (make_writable(join_file($self->{dir},$int))) {
    _error("Error giving write permissions to $int: $!");
    return undef;
  }

  1;
}

=head2 rename_file

  $dir->rename_file('res' => 'def') or die;

Renames a file within the working directory.  If the destination file
exists, it is overwritten, along with the internal information about
it.

=cut

sub rename_file {
  my $self=shift;
  my $from=shift;
  my $to=shift;

  unless (exists $self->{files}->{$from}) {
    _error("Cannot rename working file $from: unknown file");
    return 0;
  }

  my $details = $self->{files}->{$from};

  $self->{files}->{$to} = $details;
  delete $self->{files}->{$from};

  _debug("$to <- $from");

  if (test_e(join_file($self->{dir},$to))) {
    unlink join_file($self->{dir},$to);
  }
  unless (move_file(join_file($self->{dir},$from) => join_file($self->{dir},$to))) {
    _error("Cannot create working file $to from $from: $!");
    return undef;
  }
  1;
}

=head2 creatable_file

  $newfile=$dir->creatable_file('hst');
  die unless defined $newfile;

This function, although it can take two arguments (in order to store
an external name for later use), does not use an external file;
instead, it is for use when creating a new file in the working
directory.  

It returns the full pathname for the file in the working directory,
which the caller may then use to access the file at will, and records
the existance of the file internally.

=cut

sub creatable_file {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  my $details = exists $self->{files}->{$int} ? $self->{files}->{$int} : {};

  _debug("$int <= (creatable)");

  if (defined $ext) {
    # user-specified, so record and ignore it
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  }

  $self->{files}->{$int} = $details;  

  return join_file($self->{dir},$int);
}

=head2 notice_file

  $dir->notice_file('hst');

This function is used to point out the fact that a file has been
created by an external program in the working directory, so the
WorkDir object can handle it as for other files.

=cut

sub notice_file {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  my $details = exists $self->{files}->{$int} ? $self->{files}->{$int} : {};

  _debug("$int <= (noticed)");

  if (defined $ext) {
    # user-specified, so record and ignore it
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  }

  $self->{files}->{$int} = $details;  

  return 1;
}

=head1 FILE CHECKING

Functions to check for various properties of various files.

=head2 have_file

  $dir->input_file('par') unless $dir->have_file('par');

Returns TRUE if the file exists in the working directory, FALSE else.
This does not check that the file has internal state in this object.

=cut

sub have_file {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  return(test_e(join_file($self->{dir},$int)) ? 1 : '');
}

=head2 is_non_empty

  $dir->input_file('par') unless $dir->is_non_empty('par');

Returns TRUE if the file exists in the working directory and has a
non-zero size, FALSE else.  This does not check that the file has
internal state in this object.

=cut

sub is_non_empty {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  my $f=join_file($self->{dir},$int);
  return (-s $f ? 1 : '');
}

=head2 know_file

  $dir->input_file('par') unless $dir->know_file('par');

Returns TRUE if the file has internal state in this object, FALSE
else.  This does not check the existance of the file; see
C<have_file>.

=cut

sub know_file {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  return 0 unless exists $self->{files}->{$int};

  # we allow people to specify an 'ext' argument, and we parse it as
  # normal, although I can't see any reason to want to do this! ;-)
  my $details = exists $self->{files}->{$int} ? $self->{files}->{$int} : {};

  if (defined $ext) {
    # user-specified, so record and ignore it
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  }

  $self->{files}->{$int} = $details;  

  # the test is at the top; if we get here, we do know it.
  1;
}

=head2 file_intpath

  fix_out_file($dir->file_intpath('out'));

Returns the full pathname of the file inside the working directory.

=cut

sub file_intpath {
  my $self=shift;
  my $int=shift;

  return join_file($self->{dir},$int);
}

=head1 FILE REMOVAL

=head2 remove_file

  remove_file('def');

Removes the named file from the working directory, and any internal
state information associated with it.  This does not do any outputting
of the file; you should use C<output_file_done> beforehand if you want
to ensure that it has been extracted first.

If the file has no state in this object, it is simply deleted.

=cut

sub remove_file {
  my $self=shift;
  my $int=shift;

  # forget about it first.  Then even if we can't get rid of it, we
  # won't get confused later.
  delete $self->{files}->{$int};

  _debug("$int <= (removed)");

  # now, what do we do with it?
  my $intf=join_file($self->{dir},$int);

  if (test_d $intf) {
    _error("Cannot remove directory $intf");
    return undef;
  }
  
  unless (unlink $intf) {
    _error("Error deleting $intf: $!");
    return undef;
  }

  # OK, we did it.
  1;
}

=head1 FILE EXTRACTION

Having run the program, it is (usually!) necessary to get the output
back from it.  These functions allow you to do that manually; there
are also ways of setting things up to do this automatically, see
below.

Again, we can usefully divide the operations into descriptive classes.
Note that the actual operation which is used to retrieve the file
depends a lot on how (and if) it was introduced.

=over

=item Files we are done with and want to keep

The name sums it all up, really.  This may or may not cause the file
to disappear from the working directory.

=item Files we want to make sure are up to date

During a long run, for example, it may be useful to take snapshots at
various points so the user can do some things while the process is
running.  This will not necessarily freeze the user's copy of a file,
for example on UNIX if it's symlinked in anyway it won't be touched,
but it will make sure that there is an up-to-date copy in the user's
directory.

=back

These functions take similar forms of arguments to the input functions
above, in either the one- or two-argument forms.

=head2 output_file_done

  $dir->output_file_done('res') or warn;

Copies, moves, or whatever, the file.  The physical file may or may
not be removed, but its information will be removed from the object.

=cut

sub output_file_done {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  unless (exists $self->{files}->{$int}) {
    _error("Unable to retrieve $int from working directory: unknown file");
    return undef;
  }

  my $details = $self->{files}->{$int};

  if (defined $ext) {
    # user-specified
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  } else {
    $ext = $self->file_extname($int);
  }

  unless (defined $ext) {
    _error("Unable to retrieve $int from working directory: No external name known");
    return undef;
  }

  _debug("$int => $ext");

  # now, what do we do with it?
  my $intf=join_file($self->{dir},$int);
  my $handled=0;
  if (test_l $intf) {
    # it's a symlink; make sure it's a link to the right file, first!
    # readlink will die if this OS can't handle symlinks, but then one
    # assumes that test_l will fail too ...
    my $dest=readlink($intf);
    # assume that the link was created by us; then it was created with
    # a relative path (since that's what copy_file_for_read does,
    # probably correctly) between the two, so look at that.
    my $rel=relative_path($intf,$ext);
    if ($dest eq $rel) {
      # and the link points to where we wanted to put the file, so
      # it's already OK.  Don't bother removing the link.
      $handled=1;
    }
  }
  if (!$handled and (test_l $intf or test_f $intf)) {
    # it's something we can copy around, or move if it's a file
    if (test_l $intf) {
      # we copy symlinks, to make sure we dereference them properly.
      # We should know now that the place we're copying to is not the
      # same as the place we're copying from, but otherwise we're in
      # trouble!
      if (copy_file($intf, $ext)) {
	$handled=1;
      } else {
	_error("Unable to retrieve $int from working directory:\nCannot copy to $ext: $!");
	return undef;
      }
    } else {
      # files we move, since we're done. Use several tries in case of timing issues..
      my $tries=0;
      while ($tries < 3 and !$handled) {
        if (!$self->have_file($intf.'.lck') and move_file($intf, $ext)) {
  	  $handled=1;
        } else {
        # wait a bit for things to sort themselves out
          sleep(3);
          $tries++;
        }
      }
      if (!$handled) {
        # try to copy it then...
        if (copy_file($intf, $ext)) {
          _warn("The file $int was copied to $ext as it could not be moved");
  	  $handled=1;
        } else {
      	  _error("Unable to retrieve $int from working directory:\nCannot move to $ext: $!");
	  return undef;
        }
      }
    }
  }
  if (!$handled and ! test_f $intf) {
    _error("Unable to retrieve $int from working directory: file not found");
    $handled = 1;
  }
  # did we manage to do something?
  if (!$handled) {
    _error("Unable to retrieve $int from working directory: Unknown type");
    return undef;
  }

  # OK, we did it.  Puff pant.
  delete $self->{files}->{$int};
  $ext;
}

=head2 output_file_update

  $dir->output_file_update('out') or warn;

Makes sure the user copy of a file is up to date as of the now.

=cut

sub output_file_update {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  unless (exists $self->{files}->{$int}) {
    _error("Unable to retrieve $int from working directory: unknown file");
    return undef;
  }

  my $details = $self->{files}->{$int};

  if (defined $ext) {
    # user-specified
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  } else {
    $ext = $self->file_extname($int);
  }

  unless (defined $ext) {
    _error("Unable to retrieve $int from working directory: No external name known");
    return undef;
  }

  _debug("$int => $ext");

  # now, what do we do with it?
  my $intf=join_file($self->{dir},$int);
  my $handled=0;
  if (test_l $intf) {
    # it's a symlink; make sure it's a link to the right file, first!
    # readlink will die if this OS can't handle symlinks, but then one
    # assumes that test_l will fail too ...
    my $dest=readlink($intf);
    my $abs=$dest;
    unless (is_absolute($abs)) {
      $abs = normalize_filename(join_file($self->{dir},$abs));
    }
    _debug("$intf is symlink to $dest, absolutely $abs [expect $ext]");
    # even if we created the link, it may be relative rather than absolute
    if ($dest eq $ext or $abs eq $ext) {
      # and the link points to where we wanted to put the file, so
      # it's already OK.
      $handled=1;
    }
  }
  if (!$handled and (test_l $intf or test_f $intf)) {
    # it's something we can copy around, but if it's a symlink, it's a
    # symlink to the wrong file.  If we can link we have two choices:
    # we can just copy it, or we can make a copy and then link the
    # existing file to the new copy.  For now we take the former,
    # since it's a lot easier!
    if (copy_file($intf, $ext)) {
      $handled=1;
    } else {
      _error("Unable to retrieve $int from working directory:\nCannot copy to $ext: $!");
      return undef;
    }
  }
  # did we manage to do something?
  if (!$handled) {
    _error("Unable to retrieve $int from working directory: Unknown type");
    return undef;
  }

  # OK, we did it.  Puff pant.
  $self->{files}->{$int} = $details;
  $ext;
}

=head1 AUTOMATIC FILE EXTRACTION

Often, the only time it's easy to know which files will need
extracting is at the beginning of a run, at the same time as the file
is created.  Rather than forcing each program to keep track of this,
we allow this information to be stored in the object.  This has two
parts: marking and extracting.

=cut '

=head2 MARKING FILES FOR AUTOMATIC EXTRACTION

These functions are basically counterparts of the C<output_file_*>
functions, but rather than doing it immediately, they wait until told.

=head3 mark_output_file

  $dir->mark_output_file('res');

Marks the file for extraction (using C<output_file_done>) when
C<output_marked_files_done> is called.

=cut

sub mark_output_file {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  unless (exists $self->{files}->{$int}) {
    _error("Unable to mark $int for automatic retrieval: unknown file");
    return undef;
  }

  my $details = $self->{files}->{$int};

  if (defined $ext) {
    # user-specified, so record and ignore it
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  }

  $details->{auto_done}=1;
  $details->{auto_update}=0;
  $details->{auto_ignore}=0;
  $details->{auto_leave}=0;

  $self->{files}->{$int} = $details;

  1;
}

=head3 mark_output_file_update

  $dir->mark_output_file_update('out');

Marks the file for updating whenever C<output_marked_files_update> is
called, I<and for extraction/removal> when C<output_marked_files_done>
is called.

=cut

sub mark_output_file_update {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  unless (exists $self->{files}->{$int}) {
    _error("Unable to mark $int for automatic retrieval: unknown file");
    return undef;
  }

  my $details = $self->{files}->{$int};

  if (defined $ext) {
    # user-specified, so record and ignore it
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  }

  $details->{auto_done}=1;
  $details->{auto_update}=1;
  $details->{auto_ignore}=0;
  $details->{auto_leave}=0;

  $self->{files}->{$int} = $details;

  1;
}

=head3 mark_output_file_update_only

  $dir->mark_output_file_update_only('trn');

Marks the file for updating whenever C<output_marked_files_update> or
C<output_marked_files_done> are called.  This will never automatically
remove the data about the file, but may result in unnecessary copies
of the file being left lying around after C<output_marked_files_done> is
called.

=cut

sub mark_output_file_update_only {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  unless (exists $self->{files}->{$int}) {
    _error("Unable to mark $int for automatic retrieval: unknown file");
    return undef;
  }

  my $details = $self->{files}->{$int};

  if (defined $ext) {
    # user-specified, so record and ignore it
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  }

  $details->{auto_done}=0;
  $details->{auto_update}=1;
  $details->{auto_ignore}=0;
  $details->{auto_leave}=0;

  $self->{files}->{$int} = $details;

  1;
}

=head3 mark_output_file_ignore

  $dir->mark_output_file_ignore('res');

Marks the file as expected but ignorable.  This will undo any of the
other C<mark_output_file*> calls, if any have been made, and will also
register the file as normal.

=cut

sub mark_output_file_ignore {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  unless (exists $self->{files}->{$int}) {
    _error("Unable to mark $int for automatic retrieval: unknown file");
    return undef;
  }

  my $details = $self->{files}->{$int};

  if (defined $ext) {
    # user-specified, so record and ignore it
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  }

  $details->{auto_done}=0;
  $details->{auto_update}=0;
  $details->{auto_ignore}=1;
  $details->{auto_leave}=0;

  $self->{files}->{$int} = $details;

  1;
}

=head3 mark_output_file_leave

  $dir->mark_output_file_leave('mon');

Marks the file as expected but to leave alone.  This will undo any of
the other C<mark_output_file*> calls, if any have been made, and will
also register the file as normal.  See C<list_remaining_files>.

=cut

sub mark_output_file_leave {
  my $self=shift;
  my $int=shift;
  my $ext=shift;

  unless (exists $self->{files}->{$int}) {
    _error("Unable to mark $int for automatic retrieval: unknown file");
    return undef;
  }

  my $details = $self->{files}->{$int};

  if (defined $ext) {
    # user-specified, so record and ignore it
    $ext=fullpath($ext);
    $details->{extname}=$ext;
  }

  $details->{auto_done}=0;
  $details->{auto_update}=0;
  $details->{auto_ignore}=0;
  $details->{auto_leave}=1;

  $self->{files}->{$int} = $details;

  1;
}

=head2 CHECKING THE OPERATION OF AUTOMATIC EXTRACTION

To make sure you know what is going on, some functions are provided to
query the likely results of running an automatic extraction at the
current time.  This is important for two reasons:

=over

=item Unexpected Files

Sometimes, unexpected files can be created which you may want to keep.
Unless you know they are there, you can do nothing about them!

=item Missing Files

If a file is missing, the C<output_marked_files_*> functions will give
errors about being unable to extract a given file.  If you want to
handle these errors yourself, you will need to remove the offending
files from the list manually.

=back

I<Note>: I am still not entirely convinced that this is the best way
to do things, but it seems to keep most closely to the Principle of
Least Surprise.

=head3 list_unknown_files

  warn "About to remove", $job->list_unknown_files();

Returns a list of the names of all files in the working directory
which are not known about by the working directory object.  Files
which were created using the named object, or which have been marked
for update in some way, will not be listed here.

Note that "files" includes directories here!

=cut

sub list_unknown_files {
  my $self=shift;

  local *DIR;
  unless (opendir DIR, $self->{dir}) {
    _error("Error reading working directory $self->{dir}: $!");
    return ();
  }
  my @rv;
  my $f;
  while (defined ($f = readdir DIR)) {
    next if ($f eq '.' or $f eq '..');
    push @rv, $f
      unless exists $self->{files}{$f};
  }
  closedir DIR;
  @rv;
}

=head3 list_remaining_files

  @files=$workdir->list_remaining_files();

This is very similar to the C<list_unknown_files> call above, but it
also includes those files which are marked to be left by the
C<mark_output_file_leave> call.

=cut

sub list_remaining_files {
  my $self=shift;

  my @rv=$self->list_unknown_files();

  for (keys %{$self->{files}}) {
    if ($self->{files}->{$_}->{auto_leave}) {
      push @rv, $_;
    }
  }

  @rv;
}

=head3 list_missing_files

  for ($job->list_missing_files()) {
    $job->mark_output_file_ignore($_);
  }

Returns a list of the files which are missing in the working
directory, ie which would cause errors at the next call to one of the
C<output_marked_files_*> calls.  This does not include those files which
are marked to be ignored.

=cut

sub list_missing_files {
  my $self=shift;
  my @rv;

  for (keys %{$self->{files}}) {
    if ($self->{files}->{$_}->{auto_done}
	|| $self->{files}->{$_}->{auto_update}) {
      push @rv, $_
	unless test_e(join_file($self->{dir},$_));
    }
  }
  @rv;
}

=head2 INVOKING AUTOMATIC EXTRACTION

=head3 output_marked_files_done

  $dir->output_marked_files_done();

Outputs and forgets most files which have been marked for output with
the functions in L<MARKING FILES FOR AUTOMATIC EXTRACTION>.  This will
use the C<output_file_done> function even for files marked for
updating (except with C<mark_output_file_update_only>).

=cut

sub output_marked_files_done {
  my $self=shift;
  my $ok=1;
  for (keys %{$self->{files}}) {
    if ($self->{files}->{$_}->{auto_done}) {
      $self->output_file_done($_) or $ok=0;
    }
  }
  $ok;
}

=head3 output_marked_files_update

  $dir->output_marked_files_update();

Updates those files which have been marked for update.

=cut

sub output_marked_files_update {
  my $self=shift;
  my $ok=1;
  for (keys %{$self->{files}}) {
    if ($self->{files}->{$_}->{auto_update}) {
      $self->output_file_update($_) or $ok=0;
    }
  }
  $ok;
}

=head1 ENTERING AND LEAVING

Functions in this section are used to C<chdir> to and from the working
directory, in a similar way to C<pushd> and C<popd> in the shell.

=head2 enter

  $dir->enter();

Enters the working directory, recording the current directory.  If
C<enter> has already been called -- and no C<leave> after it -- this
will do nothing.

=cut

sub enter {
  my $self=shift;
  $self->{entered}={} unless exists $self->{entered};
  return 1 if ($self->{entered}->{in});
  $self->{entered}->{origin} = fullpath('.');
  if (change_cwd($self->{dir})) {
    $self->{entered}->{in} = 1;
    return 1;
  } else {
    _error("Unable to change to working directory: $!");
    return 0;
  }  
}

=head2 leave

  $dir->leave();

Leaves the working directory, returning to the directory C<enter> was
called from.  If not in the working directory, gives an error.

=cut

sub leave {
  my $self=shift;
  $self->{entered}={} unless exists $self->{entered};
  unless ($self->{entered}->{in}) {
    _error("Attempt to leave working directory when not in it");
    return 0;
  }
  unless (exists $self->{entered}->{origin}) {
    internal_error("Marked as entered dir, but no origin");
  }
  if (change_cwd($self->{entered}->{origin})) {
    $self->{entered}->{in} = 0;
    return 1;
  } else {
    _error("Unable to change back to original directory: $!");
    return 0;
  }  
}

=head1 REMOVING THE DIRECTORY

=head2 junk_directory

  $dir->output_marked_files_done();
  $dir->junk_directory() or warn "Error deleting working directory";

This call is the standard way to remove the physical directory from
the disc, and should be called at the end of a run.  It is named
C<junk_directory> to make it clear that it I<does not> check whether
any files need updating or anything, it simply deletes the directory
and all the files in it.

If this function is called again, it will give an error.

=cut

sub junk_directory {
  my $self=shift;

  unless ($self->{'exists'}) {
    internal_error("Cannot remove directory twice");
    return '';
  }

  # make sure we're not still running in that directory
  $self->{entered}={} unless exists $self->{entered};
  if ($self->{entered}->{in}) {
    $self->leave();
  }
  
  # assume, for now, that there are no subdirectories in this
  # directory.  Then removing all the files is fairly easy, with a
  # loop like the following!  In order to cope with NFS and NT locking
  # issues, we try multiple times to remove any files before
  # aborting, with sleeps in between.

  my $max_tries = 20;
  for (my $tries=1; $tries<=$max_tries; $tries++) {

    # Open the working directory, read list of filenames and close
    my @names = ();
    local *DIR;
    unless (opendir DIR, $self->{dir}) {
      _error("Unable to read $self->{dir}: $!");
      $self->{'exists'} = 0;
      return '';
    }

    while (defined (my $name = readdir DIR)) {
      next if ($name eq '.' or $name eq '..');
      push @names, $name;
    }
    closedir DIR;

    # Attempt to remove files. Exit normally if all are removed.
    my @msglines = ();
    my $filesleft = 0;
    for my $name (@names) {
      my $f = join_file($self->{dir},$name);
      if (test_d $f) {
        push @msglines, "$name: Is a directory";
        $filesleft++;
      } else {
        unless (unlink $f) {
          push @msglines, "Unable to delete: $name: $!";
          $filesleft++;
        }
      }
    }

    last unless ($filesleft > 0);

    # Files remain after 3 attempts: warn
    #              after max attempts: error
    if ($tries == 3 || $tries >= $max_tries) {
      unshift @msglines,
           "After $tries deletion attempts, files remain in the working directory:",
           $self->{dir};

      if ($tries >= $max_tries) {
        _error(join "\n\n", @msglines);
      } else {
        push @msglines, 'Deletion attempts will continue.';
        _warn(join "\n\n", @msglines);
      }
    }
    
    sleep (3);
  }

  # and now, regardless of success/failure above (one never knows
  # one's own luck ...), try to remove the directory.

  $max_tries = 3;
  for (my $tries=1; $tries<=$max_tries; $tries++) {
    last if (rmdir $self->{dir});
    if ($tries >= $max_tries) {
      _error("After $tries attempts: Unable to remove working directory: $!");
    }
    sleep(3);
  }

  $self->{'exists'}=0;

  return (error_pending() ? '' : 1);
}

=head1 DEBUGGING

=head2 CHECKPOINTING

=head3 enable_checkpoints

  $dir->enable_checkpoints();

Without this call, the C<checkpoint> call will do nothing.  This call
can be inserted into a script for debugging purposes.

=cut

sub enable_checkpoints {
  my $self=shift;
  $self->{debugging} = {} unless exists $self->{debugging};
  $self->{debugging}{checkpoints}++;
  1;
}

=head3 checkpoint

  $dir->checkpoint();
  $dir->checkpoint('before adaption');

A checkpoint writes a few lines describing the contents of the working
directory to a file "I<dir>.checkpoint".  This is for use when
debugging operations which use the working directory calls, and as
such is a NOP by default; to enable this functionality, use the
C<enable_checkpoints> call on the object.

If the file exists, it is appended to.

The optional argument gives a title for the checkpoint.

=cut

sub checkpoint {
  my $self=shift;
  my $title=shift || caller;

  $self->{debugging} = {} unless exists $self->{debugging};
  return unless $self->{debugging}{checkpoints};

  local *CP;
  unless (open CP, ">>".$self->{dir}.".checkpoint") {
    internal_error("Error opening checkpoint file for append: $!");
  }
  print CP "=" x 75, "\n";
  my $pad=(75 - length($title))/2;
  $pad=0 if $pad < 0;
  print CP " " x $pad, $title, "\n";
  print CP "-" x 75, "\n";
  local *DIR;
  my %proccd;
  if (opendir DIR, $self->{dir}) {
    # got a directory
    while (defined (my $f = readdir(DIR))) {
      my @stat=stat $self->{dir} . "/$f";
      my %details = (
		     size => $stat[7],
		     mode => $stat[2],
		     atime => $stat[8],
		     mtime => $stat[9],
		     ctime => $stat[10],
		    );
      print CP ($f,"\t",
		(map { "($_ $self->{files}{$f}{$_}) " } sort keys %{$self->{files}{$f}}),
		(map { "($_ $details{$_}) " } sort keys %details),
		"\n");
      $proccd{$f}++;
    }
    closedir DIR;
  } elsif (! test_d $self->{dir}) {
    print CP "Directory does not exist\n";
  } else {
    print CP "Error reading directory: $!\n";
  }

  # now write details for the remaining files
  for my $f (sort keys %{$self->{files}}) {
    next if $proccd{$f}++;
    print CP ($f,"\t",
	      (map { "($_ $self->{files}{$f}{$_}) " } sort keys %{$self->{files}{$f}}),
	      "\n");
  }

  close CP or internal_error("Error writing checkpoint file: $!");
  1;
}

sub finalise_dir {
  my $self = shift ; 
  return move ($self->{dir}, $self->{basename}) ;
}

1;
