The fundamental difference between Distributed ANSYS and shared-memory ANSYS is that
N number of ANSYS processes will be running at the same
time (where N is the total number of CPU processor cores
used) for one model. These N processes may be running on a
single machine (in the same working directory) or on multiple machines. These processes
are not aware of each other's existence unless they are communicating (sending
messages). Distributed ANSYS, along with the MPI software, provide the means by which the
processes communicate with each other in the right location and at the appropriate time.
The following topics give a summary of behavioral differences between Distributed ANSYS and shared-memory ANSYS in specific areas of the ANSYS program:
Upon startup and during a parallel solution, Distributed ANSYS appends
n to the current jobname (where
n stands for the process rank). The
master process rank is 0 and the slave processes are numbered from 1 to
N - 1. In the rare case that the
user-supplied jobname ends in a numeric value (0...9) or an underscore, an underscore is
automatically appended to the jobname prior to appending the process
rank. This is done to avoid any potential conflicts with the new file
names.
Therefore, upon startup and during a parallel solution, each process
will create and use files named
Jobnamen.EXT.
These files contain the local data that is specific to each Distributed ANSYS
process. Some common examples include the .LOG and
.ERR files as well as most files created during
solution such as .ESAV, .FULL,
.RST, and .MODE. See
Program-Generated Files in the Basic Analysis Guide
for more information on files that ANSYS typically creates.
Other actions (/PREP7, /POST1,
etc.) which are performed only by the master process will work on the
global Jobname.EXT files
by default. These files (e.g., Jobname.DB and
Jobname.RST) contain the global data for the
entire model. For example, only the master process will save and resume
the Jobname.DB file.
After a parallel solution successfully completes, Distributed ANSYS
automatically merges some of the (local)
Jobnamen.EXT
files into a single (global) file named
Jobname.EXT.
These include files such as Jobname.RST,
Jobname.ESAV,
Jobname.EMAT, and so on (see the
DMPOPTION command for a complete list of these
files). This action is performed when the FINISH
command is executed upon leaving the solution processor. These files
contain the same information about the final computed solution as files
generated for the same model computed with shared-memory ANSYS.
Therefore, all downstream operations (such as postprocessing) can be
performed using shared-memory ANSYS (or in the same manner as
shared-memory ANSYS) by using these global files.
If any of these global
Jobname.EXT
files are not needed for downstream operations, you can reduce the
overall solution time by suppressing the file combination for individual
file types (see the DMPOPTION command for more
information). If it is later determined that a global
Jobname.EXT file
is needed for a subsequent operation or analysis, the local files can be
combined by using the COMBINE command.
Distributed ANSYS will not delete most files written by the slave processes when the analysis is completed. If you choose, you can delete these files when your analysis is complete (including any restarts that you may wish to perform). If you do not wish to have the files necessary for a restart saved, you can issue RESCONTROL,NORESTART.
File copy, delete, and rename operations can be performed across all
processes by using the DistKey option on the
/COPY, /DELETE, and
/RENAME commands. This provides a convenient way
to manage local files created by a distributed parallel solution. For
example,
/DELETE,Fname,Ext,,ON
automatically appends the process rank number to the specified file name
and deletes Fnamen.Ext
from all processes. See the /COPY,
/DELETE, and /RENAME command
descriptions for more information. In addition, the
/ASSIGN command can be used to control the name
and location of specific local and global files created by
Distributed ANSYS.
You can launch Distributed ANSYS in either interactive or batch mode for the
master process. However, the slave processes are always in batch mode.
The slave processes cannot read the
start.ans or
stop.ans files. The master process
sends all /CONFIG,LABEL
commands to the slave processes as needed.
On Windows systems, there is no ANSYS output console window when running the Distributed ANSYS GUI (interactive mode). All standard output from the master process will be written to a file named file0.out. (Note that the jobname is not used.)
When a Distributed ANSYS job is executed, the output for the master process is
written in the same fashion as shared-memory ANSYS. By default, the
output is written to the screen; or, if you specified an output file via
the launcher or the -o command line option, the output
for the master process is written to that file.
The slave processes automatically write this output to
Jobnamen.OUT.
Typically, these slave process output files have little value because
all of the relevant job information is written to the screen or the
master process output file. One exception is when the
DDOPTION command is used to specify an
alternative domain decomposition approach such as frequency or cyclic
harmonic index domain decomposition. In this case, the solution
information for the harmonic frequencies or cyclic harmonic indices
solved by the slave processes are only written to the output files
(Jobnamen.OUT)
for those processes.
The same principle also applies to the error file
Jobnamen.ERR.
When a warning or error occurs on one of the slave processes during the
Distributed ANSYS solution, the process writes that warning or error message to
its error file and then communicates the warning or error message to the
master process. Typically, this allows the master process to write the
warning or error message to its error file and output file and, in the
case of an error message, allows all of the Distributed ANSYS processes to exit
the program simultaneously.
In some cases, an error message may fail to be
fully communicated to the master process. If this happens, you can view
each Jobnamen.ERR and/or
Jobnamen.OUT
file in an attempt to learn why the job failed. The error files and
output files written by all the processes will be incomplete but may
still provide some useful information as to why the job failed.
In some rare cases, the job may hang. When this
happens, you can use the cleanup script (or .bat
file on Windows) to kill the processes. The cleanup script is
automatically written into the initial working directory of the master
process and is named
cleanup-ansys-[machineName]-[processID].sh
(or .bat).
In pre- and postprocessing, APDL works the same in Distributed ANSYS as in shared-memory ANSYS. However, in the solution processor (/SOLU), Distributed ANSYS does not support certain *GET items. In general, Distributed ANSYS supports global solution *GET results such as total displacements and reaction forces. It does not support element level results specified by ESEL, ESOL, and ETABLE labels. Unsupported items will return a *GET value of zero.
Multiple commands entered in an input file can be condensed into a single line if the commands are separated by the $ character (see Condensed Data Input in the Command Reference). Distributed ANSYS cannot properly handle condensed data input. Each command must be placed on its own line in the input file for a Distributed ANSYS run.
Upon starting a solution, Distributed ANSYS automatically decomposes the
problem into N CPU domains so that each
process (or each CPU core) works on only a portion of the simulation.
This domain decomposition is done only by the master process. You can
control which domain decomposition method is used by setting the
Decomp argument on the
DDOPTION command.
By default, the domain decomposition is mesh-based
(Decomp = MESH), which means each CPU
domain is a group or subset of elements within the whole model. For
certain harmonic analyses, the domain decomposition can be based on the
frequency domain (Decomp = FREQ), in which
case each CPU domain computes the harmonic solution for the entire model
at a different frequency point. For certain cyclic symmetry analyses,
the domain decomposition can be based on the harmonic indices
(Decomp = CYCHI), in which case each CPU
domain computes the cyclic solution for a different harmonic
index.
The NProcPerSol
argument on the DDOPTION command gives you the
flexibility to combine the FREQ or DYCHI decomposition methods with the
mesh-based domain decomposition method. Consider a harmonic analysis
with 50 frequency points requested (NSUBST,50) run on
a workstation using Distributed ANSYS with 16 cores (-dis
-np 16). Using mesh decomposition
(DDOPTION,MESH) essentially solves one frequency
at a time with 16 groups of elements (1x16). Using
DDOPTION,FREQ,1 solves 16 frequencies at a time
with 1 group of elements; that is, the entire FEA model (16x1). Using
the NProcPerSol field allows you to consider
alternative combinations in between these 2 scenarios. You could try
DDOPTION,FREQ,2 to solve 8 frequencies at a time
with 2 groups of elements per solution (8x2), or
DDOPTION,FREQ,4 to solve 4 frequencies with 4
groups of elements (4x4), and so on. Note that the total core count
specified at startup cannot be altered, and the program works to
maintain the NProcPerSol value as input,
which means the number of frequency or cyclic harmonic index solutions
solved at a time may need to be adjusted to fit within the other defined
parameters.
There are pros and cons for each domain decomposition approach. For
example, with the default MESH method, the total amount of memory
required to solve the simulation in N
processes is typically not much larger than the amount of memory
required to solve the simulation on one process. However, when using the
FREQ or CYCHI domain decomposition method, the amount of memory required
to solve the simulation on N processes is
typically N times greater than the amount of
memory to solve the simulation on one process. As another example, with
the MESH domain decomposition method, the application requires a
significant amount of data communication via MPI, thus requiring a fast
interconnect for optimal performance. With the FREQ and CYCHI methods,
very little data communication is necessary between the processes and,
therefore, good performance can still be achieved using slower
interconnect hardware.
All FEA data (elements, nodes, materials, sections, real constants, boundary conditions, etc.) required to compute the solution for each CPU domain is communicated to the slave processes by the master process. Throughout the solution, each process works only on its piece of the entire model. When the solution phase ends (for example, FINISH is issued in the solution processor), the master process in Distributed ANSYS works on the entire model again (that is, it behaves like shared-memory ANSYS).
In Distributed ANSYS, the OUTPR command prints NSOL and RSOL in the same manner as in shared-memory ANSYS. However, for other items such as ESOL, Distributed ANSYS prints only the element solution for the group of elements belonging to the CPU domain of the master process. Therefore, OUTPR, ESOL has incomplete information and is not recommended. Also, the order of elements is different from that of shared-memory ANSYS due to domain decomposition. A direct one-to-one element comparison with shared-memory ANSYS will be different if using OUTPR.
Both shared-memory ANSYS and Distributed ANSYS can handle a large number of coupling and constraint equations (CE/CP) and contact elements. However, specifying too many of these items can force Distributed ANSYS to communicate more data among each process, resulting in longer elapsed time to complete a distributed parallel job. You should reduce the number of CE/CP if possible and make potential contact pairs in a smaller region to achieve non-deteriorated performance. In addition, for assembly contact pairs or small sliding contact pairs, you can use the command CNCHECK,TRIM to remove contact and target elements that are initially in far-field (open and not near contact). This trimming option will help to achieve better performance in Distributed ANSYS runs.
Shared-memory ANSYS can postprocess the last set of results using the Jobname.DB file (if the solution results were saved), as well as using the Jobname.RST file. Distributed ANSYS, however, can only postprocess using the Jobname.RST file and cannot use the Jobname.DB file as solution results are not entirely written to the database. You will need to issue a SET command before postprocessing.
By default, Distributed ANSYS will automatically combine the local results
files (for example,
Jobnamen.RST) into a
single global results file (Jobname.RST). This step
can be expensive depending on the number of load steps and the amount of
results stored for each solution. It requires each local results file to
be read by each slave process, communicated to the master process, and
then combined together and written by the master process. As a means to
reduce the amount of communication and I/O performed by this operation,
the DMPOPTION command can be used to skip the step of
combining the local results file into a single global results file. Then
the RESCOMBINE command macro can be used in
/POST1 to individually read each local results
file until the entire set of results is placed into the database for
postprocessing. If needed, a subsequent RESWRITE
command can then be issued to write a global results file for the
distributed solution.
Note that if the step of combining the results file is skipped, it may impact downstream analyses that rely on a single global results file for the entire model. If it is later determined that a global results file (e.g., Jobname.RST) is needed for a subsequent operation, you can use the COMBINE command to combine the local results files into a single, global results file.
Distributed ANSYS supports multiframe restarts for nonlinear static, full transient, and mode-superposition transient analyses. The procedures and command controls are the same as described in the Basic Analysis Guide. However, restarts in Distributed ANSYS have additional limitations based on the procedure used, as described in the following sections:
See also Additional Consideration for the Restart for more information on restarting a distributed-memory parallel solution.
This procedure requires that you use the same number of cores for the restart as in the original run. It does not require any additional commands beyond those used in a typical multiframe restart procedure.
The total number of cores used when restarting Distributed ANSYS must not be altered following the first load step and first substep.
Example 1: If you use the following command line for the first load step:
ansys182 -dis -np 8 -i input -o output1
then, you must also use 8 cores (-dis -np 8) for the
multiframe restart, and the files from the original analysis that are
required for the restart must be located in the current working
directory. Additionally, this means you cannot perform a restart using
shared-memory parallel processing (shared-memory ANSYS) if Distributed ANSYS was
used prior to the restart point.
When running across machines, the job launch procedure (or script) used when restarting Distributed ANSYS must not be altered following the first load step and first substep. In other words, you must use the same number of machines, the same number of cores for each of the machines, and the same host (master) and slave relationships among these machines in the restart job that follows.
Example 2: If you use the following
command line for the first load step where the host machine (which
always appears first in the list of machines) is mach1,
and the slave machines are mach2 and
mach3:
ansys182 –dis –machines mach1:4:mach2:1:mach3:2 –i input –o output1
then for the multiframe restart, you must use a command line such as this:
ansys182 –dis –machines mach7:4:mach6:1:mach5:2 –i restartjob –o output2
This command line uses the same number of machines (3), the same
number of cores for each machine in the list (4:1:2), and the same
host/slave relationship (4 cores on host, 1 core on first slave, and 2
cores on second slave) as the original run. Any alterations in the
-machines field, other than the actual machine
names, will result in restart failure. Finally, the files from the
original analysis that are required for the restart must be located in
the current working directory on each of the machines.
The files needed for a restart must be available on the machine(s) used for the restarted analysis. Each machine has its own restart files that are written from the previous run. The restart process needs to use these files to perform the correct restart actions.
For Example 1 above, if the two analyses (-dis –np
8) are performed in the same working directory, no action
is required; the restart files will already be available. However, if
the restart is performed in a new directory, all of the restart files
listed in Table 4.4: Required Files for Multiframe Restart - Procedure 1 must be copied (or moved) into
the new directory before performing the multiframe restart.
For Example 2 above, the restart files listed in the “Host
Machine” column in Table 4.4: Required Files for Multiframe Restart - Procedure 1 must be copied
(or moved) from mach1 to mach7, and
all of the files in the “Slave Machines” column must be
copied (or moved) from mach2 to mach6
and from mach3 to mach5 before
performing the multiframe restart.
The .Xnnn file
extension mentioned here refers to the
.Rnnn
and .Mnnn files
discussed in Multiframe File Restart Requirements in the Basic Analysis Guide.
The
Jobnamen.RST
files are optional. The restart can be performed successfully
without them.
In this procedure, the total number of cores used when restarting Distributed ANSYS can be altered following the first load step and first substep. In addition, you can perform the restart using either a distributed-memory parallel solution (Distributed ANSYS) or a shared-memory parallel solution (shared-memory ANSYS). Some additional steps beyond the typical multiframe restart procedure are required to ensure that necessary files are available.
Note: This procedure cannot be used when performing a restart for the following analysis types: mode-superposition transient analysis; an analysis that includes mesh nonlinear adaptivity.
In this procedure, the
Jobname.Rnnn
file must be available. This file is not generated by default in
Distributed ANSYS, even when restart controls are activated via the
RESCONTROL command. You must either use the
command DMPOPTION,RNN,YES in the prior (base)
analysis, or you must manually combine the
Jobnamen.Rnnn
files into the
Jobname.Rnnn file
using the COMBINE command.
For example, if you use the following command line for the first load step:
ansys182 –dis –np 8 –i input –o output1
then for the multiframe restart, you can use more or less than 8 cores
(-np , where
NN does not equal 8).
The files from the original analysis that are required for the restart must be located in the current working directory. If running across machines, the restart files are only required to be in the current working directory on the host machine.
Table 4.5: Required Files for Multiframe Restart - Procedure 2
| Host Machine | Slave Machines |
|---|---|
| Jobname.LDHI | - - |
| Jobname.RDB | - - |
Jobname.Rnnn | - - |
| Jobname.RST [1] | - - |
The advantage of using Procedure 1 (same number of cores) is faster
performance by avoiding the potentially costly steps of combining the
Jobnamen.Rnnn
files and later splitting the
Jobnamen.Rnnn
(and possibly the Jobname.RST/RTH/RMG) files during the
restarted analysis. The disadvantage is that there are more files that need to
be managed during the restart process.
The advantage of Procedure 2 (different number of cores) is less files to manage during the restart process, but at the cost of additional time to combine the local restart files and then, later on, to split this file data during the restarted analysis. Depending on the size of the simulation, this overhead may be insignificant.
In all restarts, the Jobname.RST results file (or
Jobname.RTH or Jobname.RMG) on the
host machine is recreated after each solution by merging the
Jobnamen.RST files
again.
If you do not require a restart, issue RESCONTROL,NORESTART
in the run to remove or to avoid writing the necessary restart files on the host
and slave machines. If you use this command, the slave processes will not have
files such as .ESAV, .OSAV,
.RST, or
.X000, in the working
directory at the end of the run. In addition, the host process will not have
files such as .ESAV, .OSAV,
.X000,
.RDB, or .LDHI at the end of the
run. The program will remove all of the above scratch files at the end of the
solution phase (FINISH or /EXIT). This
option is useful for file cleanup and control.