When running a simulation, the solution time is typically dominated by three main parts: the time spent to create the element matrices and form the global matrices or global systems of equations, the time to solve the linear system of equations, and the time spent calculating derived quantities (such as stress and strain) and other requested results for each element.
The distributed-memory parallelism offered via Distributed ANSYS allows the entire solution phase to run in parallel, including the stiffness matrix generation, linear equation solving, and results calculations. As a result, a simulation using distributed-memory parallel processing usually achieves much faster solution times than a similar run performed using shared-memory parallel processing, particularly at higher core counts.
Distributed ANSYS can run a solution over multiple cores on a single machine or on multiple machines (that is, a cluster). It automatically decomposes the model into smaller domains, transfers the domains to each core, solves each domain simultaneously, and creates a complete solution to the model. The memory and disk space required to complete the solution can also be distributed over multiple machines. By utilizing all of the resources of a cluster (computing power, RAM, memory and I/O bandwidth), distributed-memory parallel processing can be used to solve very large problems much more efficiently compared to the same simulation run on a single machine.
Distributed ANSYS works by launching multiple ANSYS processes on either a single machine or on multiple machines (as specified by one of the following command line options: -np, -machines, or -mpifile). The machine that the distributed run is launched from is referred to as the master or host machine (or in some cases, primary compute node), and the other machines are referred to as the slave machines (or compute nodes). The first process launched on the master machine is referred to as the master or host process; all other processes are referred to as the slave processes.
Each Distributed ANSYS process is essentially a running process of shared-memory ANSYS. These processes are launched through the specified MPI software layer. The MPI software allows each Distributed ANSYS process to communicate, or exchange data, with the other processes involved in the distributed simulation.
Distributed ANSYS does not currently support all of the analysis types, elements, solution options, etc. that are available with shared-memory ANSYS (see Supported Features). In some cases, Distributed ANSYS stops the analysis to avoid performing an unsupported action. If this occurs, you must launch shared-memory ANSYS to perform the simulation. In other cases, Distributed ANSYS will automatically disable the distributed-memory parallel processing capability and perform the operation using shared-memory parallelism. This disabling of the distributed-memory parallel processing can happen at various levels in the program.
The master process handles the inputting of commands as well as all of the pre- and postprocessing actions. Only certain commands (for example, the SOLVE command and supporting commands such as /SOLU, FINISH, /EOF, /EXIT, and so on) are communicated to the slave processes for execution. Therefore, outside of the SOLUTION processor (/SOLU), Distributed ANSYS behaves very similar to shared-memory ANSYS. The master process works on the entire model during these pre- and postprocessing steps and may use shared-memory parallelism to improve performance of these operations. During this time, the slave processes wait to receive new commands from the master process.
Once the SOLVE command is issued, it is communicated to the slave processes and all Distributed ANSYS processes become active. At this time, the program makes a decision as to which mode to use when computing the solution. In some cases, the solution will proceed using only a distributed-memory parallel (DMP) mode. In other cases, similar to pre- and postprocessing, the solution will proceed using only a shared-memory parallel (SMP) mode. In a few cases, a mixed mode may be implemented which tries to use as much distributed-memory parallelism as possible for maximum performance. These three modes are described further below.
Pure DMP mode The simulation is fully supported by Distributed ANSYS, and distributed-memory parallelism is used throughout the solution. This mode typically provides optimal performance in Distributed ANSYS.
Mixed mode The simulation involves a particular set of computations that is not supported by Distributed ANSYS. Examples include certain equation solvers and remeshing due to mesh nonlinear adaptivity. In these cases, distributed-memory parallelism is used throughout the solution, except for the unsupported set of computations. When that step is reached, the slave processes in Distributed ANSYS simply wait while the master process uses shared-memory parallelism to perform the computations. After the computations are finished, the slave processes continue to compute again until the entire solution is completed.
Pure SMP mode The simulation involves an analysis type or feature that is not supported by Distributed ANSYS. In this case, distributed-memory parallelism is disabled at the onset of the solution, and shared-memory parallelism is used instead. The slave processes in Distributed ANSYS are not involved at all in the solution but simply wait while the master process uses shared-memory parallelism to compute the entire solution.
When using shared-memory parallelism inside of Distributed ANSYS (in mixed mode or SMP mode, including all pre- and postprocessing operations), the master process will not use more cores on the master machine than the total cores you specify to be used for the Distributed ANSYS solution. This is done to avoid exceeding the requested CPU resources or the requested number of licenses.
The following table shows which steps, including specific equation solvers, can be run in parallel using shared-memory ANSYS and Distributed ANSYS.
Table 4.1: Parallel Capability in Shared-Memory and Distributed ANSYS
| Solvers/Feature | Shared-Memory ANSYS | Distributed ANSYS |
|---|---|---|
| Sparse | Y | Y |
| PCG | Y | Y |
| ICCG | Y | Y [1] |
| JCG | Y | Y [1] [2] |
| QMR | Y | Y [1] |
| Block Lanczos eigensolver | Y | Y [3] |
| PCG Lanczos eigensolver | Y | Y |
| Supernode eigensolver | Y | Y [1] |
| Subspace eigensolver | Y | Y |
| Unsymmetric eigensolver | Y | Y |
| Damped eigensolver | Y | Y |
| QRDAMP eigensolver | Y | Y |
| Element formulation, results calculation | Y | Y |
| Graphics and other pre- and postprocessing | Y | Y [1] |
For static analyses and transient analyses using the full method (TRNOPT,FULL), the JCG equation solver runs in pure DMP mode only when the matrix is symmetric. Otherwise, it runs in SMP mode.
Block Lanczos can be run in both DMP and SMP modes for modal analysis. It can be run only in SMP mode for buckling analysis.
The maximum number of cores allowed in a Distributed ANSYS analysis is currently set at 8192. Therefore, you can run Distributed ANSYS using anywhere from 2 to 8192 cores (assuming the appropriate HPC licenses are available) for each individual job. Performance results vary widely for every model when using any form of parallel processing. For every model, there is a point where using more cores does not significantly reduce the overall solution time. Therefore, it is expected that most models run in Distributed ANSYS can not efficiently make use of hundreds or thousands of cores.
Files generated by Distributed ANSYS are named
Jobnamen.ext, where
n is the process number. (See Differences in General Behavior
for more information.) The master process is always numbered 0, and the slave processes are
1, 2, etc. When the solution is complete and you issue the FINISH command
in the SOLUTION processor, Distributed ANSYS combines all
Jobnamen.RST files into a single
Jobname.RST file, located on the master machine. Other files, such
as .MODE, .ESAV, .EMAT, etc.,
may be combined as well upon finishing a distributed solution. (See Differences in Postprocessing for more information.)
The remaining sections explain how to configure your environment to run Distributed ANSYS, how to run a Distributed ANSYS analysis, and what features and analysis types are supported in Distributed ANSYS. You should read these sections carefully and fully understand the process before attempting to run a distributed analysis. The proper configuration of your environment and the installation and configuration of the appropriate MPI software are critical to successfully running a distributed analysis.