4.6. Troubleshooting

This section describes problems which you may encounter while using Distributed ANSYS, as well as methods for overcoming these problems. Some of these problems are specific to a particular system, as noted.

4.6.1. Setup and Launch Issues

To aid in troubleshooting, you may need to view the actual MPI run command line. On Linux the command is mpirun, and you can view the command line by setting the ANS_SEE_RUN_COMMAND environment variable to 1. On Windows the command is mpiexec, and you can view the command line by setting the ANS_SEE_RUN and ANS_CMD_NODIAG environment variables to TRUE.

Job fails to launch

The first thing to check when a Distributed ANSYS job fails to launch is that the MPI software you wish to use is installed and properly configured (see Configuring Distributed ANSYS).

Next, if running across multiple machines, ensure that the working directory path is identical on all machines (or that you are using a shared network directory) and that you have permission to write files into the working directory used by each machine.

Finally, make sure that you are running the distributed solution on a homogeneous cluster. The OS level and processors must be identical on all nodes in the cluster. If they are not, you may encounter problems. For example, when running Distributed ANSYS across machines using Intel MPI, if the involved cluster nodes have different processor models, the program may hang (i.e., fail to launch). In this situation, no data is written to any files and no error message is output.

When using Distributed ANSYS with Intel MPI on a Windows cluster, the program may hang when launching. One workaround is to switch to IBM MPI. Another workaround is to run on a Windows HPC server cluster which uses Microsoft MPI.

No permission to system

This error can occur if you do not have login access to a remote system where Distributed ANSYS is supposed to run. If you use Linux, you can experience this problem if you have not set permissions on the .rhosts file properly. Before starting Distributed ANSYS, be sure you have access to all the remote systems you have specified (i.e., you should be able to rsh to those systems) and that the control node is included in its own .rhosts file. If you run on Linux, be sure you have run the following command:

chmod 600 .rhosts

MPI: could not run executable

If you encounter this message, verify that you have the correct version of MPI and that it is installed correctly, and verify that you have a .rhosts file on each machine. If not, create a .rhosts file on all machines where you will run Distributed ANSYS, make sure the permissions on the file are 600, and include an entry for each hostname where you will run Distributed ANSYS.

Error executing ANSYS. Refer to System-related Error Messages in the ANSYS online help. If this was a Distributed ANSYS job, verify that your MPI software is installed correctly, check your environment settings, or check for an invalid command line option.

You may encounter the above message when setting up IBM MPI or running Distributed ANSYS using IBM MPI on a Windows platform. This may occur if you did not correctly run the set password bat file. Verify that you completed this item according to the IBM MPI installation instructions.

You may also see this error if Ansys Inc\v182\ansys\bin\<platform> (where <platform> is intel or winx64) is not in your PATH.

If you need more detailed debugging information, use the following:

  1. Open a Command Prompt window and set the following:

    SET ANS_SEE_RUN=TRUE
    SET ANS_CMD_NODIAG=TRUE
  2. Run the following command line: ansys182 -b -dis -i myinput.inp -o myoutput.out.

Distributed ANSYS fails to launch when running from a fully-qualified pathname.

Distributed ANSYS will fail if the ANSYS 18.2 installation path contains a space followed by a dash if %ANSYS182_DIR%\bin\<platform> (where <platform> is intel or winx64) is not in the system PATH. Add %ANSYS182_DIR%\bin\<platform> to the system PATH and invoke ansys182 (without the fully qualified pathname). For example, if your installation path is:

C:\Program Files\Ansys -Inc\v182\bin\<platform>

The following command to launch Distributed ANSYS will fail:

"C:\Program Files\Ansys -Inc\v182\bin\<platform>\ansys182.exe” -g 

However, if you add C:\Program Files\Ansys -Inc\v182\bin\<platform> to the system PATH, you can successfully launch Distributed ANSYS by using the following command:

ansys182 -g

The required licmsgs.dat file, which contains licensing-related messages, was not found or could not be opened. The following path was determined using environment variable ANSYS182_DIR. This is a fatal error - - exiting.

Check the ANSYS182_DIR environment variable to make sure it is set properly. Note that for Windows HPC clusters, the ANSYS182_DIR environment variable should be set to \\HEADNODE\Ansys Inc\v182\ansys, and the ANSYSLIC_DIR environment variable should be set to \\HEADNODE\Ansys Inc\Shared Files\Licensing on all nodes.

WARNING: Unable to read mpd.hosts or list of hosts isn't provided. MPI job will be run on the current machine only. (Intel MPI)

When using Intel MPI, you must have an mpd.hosts file in your working directory when going across boxes that contain a line for each box. Otherwise, you will encounter the following error.

WARNING: Unable to read mpd.hosts or list of hosts isn't provided. MPI job will be run on the current machine only.
mpiexec: unable to start all procs; may have invalid machine names
remaining specified hosts:
xx.x.xx.xx (hostname)

4.6.2. Stability Issues

This section describes potential stability issues that you may encounter while running Distributed ANSYS.

Intel MPI May Crash on AMD Processors

When using Distributed ANSYS with Intel MPI on systems having AMD processors, the program may become unstable and crash. The workaround is to use IBM MPI instead of Intel MPI.

Recovering from a Computer, Network, or Program Crash

When a Distributed ANSYS job crashes unexpectedly (e.g., seg vi, floating point exception, out-of-disk space error), an error message may fail to be fully communicated to the master process and written into the output file. If this happens, you can view all of the output and/or error files written by each of the slave processes (e.g., Jobnamen.OUT and/or Jobnamen.ERR) in an attempt to learn why the job failed. In some rare cases, the job may hang. When this happens, you must manually kill the processes; the error files and output files written by all the processes will be incomplete but may still provide some useful information as to why the job failed.

Be sure to kill any lingering processes (Linux: type kill -9 from command level; Windows: use Task Manager) on all processors and start the job again.

Job Fails with SIGTERM Signal (Linux Only)

Occasionally, when running on Linux, a simulation may fail with a message like the following:

MPI Application rank 2 killed before MPI_Finalize() with signal 15

forrtl: error (78): process killed (SIGTERM)

This typically occurs when computing the solution and means that the system has killed the ANSYS process. The two most common occurrences are (1) ANSYS is using too much of the hardware resources and the system has killed the ANSYS process or (2) a user has manually killed the ANSYS job (i.e., kill -9 system command). Users should check the size of job they are running in relation to the amount of physical memory on the machine. Most often, decreasing the model size or finding a machine with more RAM will result in a successful run.

4.6.3. Solution and Performance Issues

This section describes solution and performance issues that you may encounter while running Distributed ANSYS.

Poor Speedup or No Speedup

As more cores are utilized, the runtimes are generally expected to decrease. The biggest relative gains are typically achieved when using two cores compared to using a single core. When significant speedups are not seen as additional cores are used, the reasons may involve both hardware and software issues. These include, but are not limited to, the following situations.

Hardware

Oversubscribing hardware In a multiuser environment, this could mean that more physical cores are being used by multiple simulations than are available on the machine. It could also mean that hyperthreading is activated. Hyperthreading typically involves enabling extra virtual cores, which can sometimes allow software programs to more effectively use the full processing power of the CPU. However, for compute-intensive programs such as ANSYS, using these virtual cores rarely provides a significant reduction in runtime. Therefore, it is recommended you do not use hyperthreading; if hyperthreading is enabled, it is recommended you do not exceed the number of physical cores.

Lack of memory bandwidth On some systems, using most or all of the available cores can result in a lack of memory bandwidth. This lack of memory bandwidth can impact the overall scalability.

Slow interconnect speed When running Distributed ANSYS across multiple machines, the speed of the interconnect (GigE, Infiniband, etc.) can have a significant impact on the performance. Slower interconnects cause each Distributed ANSYS process to spend extra time waiting for data to be transferred from one machine to another. This becomes especially important as more machines are involved in the simulation. See Interconnect Configuration at the beginning of this chapter for the recommended interconnect speed.

Software

Simulation includes non-supported features The shared and distributed-memory parallelisms work to speed up certain compute-intensive operations in /PREP7, /SOLU and /POST1. However, not all operations are parallelized. If a particular operation that is not parallelized dominates the simulation time, then using additional cores will not help achieve a faster runtime.

Simulation has too few DOF (degrees of freedom) Some analyses (such as transient analyses) may require long compute times, not because the number of DOF is large, but because a large number of calculations are performed (i.e., a very large number of time steps). Generally, if the number of DOF is relatively small, parallel processing will not significantly decrease the solution time. Consequently, for small models with many time steps, parallel performance may be poor because the model size is too small to fully utilize a large number of cores.

I/O cost dominates solution time For some simulations, the amount of memory required to obtain a solution is greater than the physical memory (i.e., RAM) available on the machine. In these cases, either virtual memory (i.e., hard disk space) is used by the operating system to hold the data that would otherwise be stored in memory, or the equation solver writes extra files to the disk to store data. In both cases, the extra I/O done using the hard drive can significantly impact performance, making the I/O performance the main bottleneck to achieving optimal performance. In these cases, using additional cores will typically not result in a significant reduction in overall time to solution.

Large contact pairs For simulations involving contact pairs with a large number of elements relative to the total number of elements in the entire model, the performance of Distributed ANSYS is often negatively impacted. These large contact pairs require Distributed ANSYS to do extra communication and often cause a load imbalance between each of the cores (i.e., one core might have two times more computations to perform than another core). In some cases, using CNCHECK,TRIM can help trim any unnecessary contact/target elements from the larger contact pairs. In other cases, however, manual interaction will be required to reduce the number of elements involved in the larger contact pairs.

Different Results Relative to a Single Core

Distributed-memory parallel processing initially decomposes the model into domains. Typically, the number of domains matches the number of cores. Operational randomness and numerical round-off inherent to parallelism can cause slightly different results between runs on the same machine(s) using the same number of cores or different numbers of cores. This difference is often negligible. However, in some cases the difference is appreciable. This sort of behavior is most commonly seen on nonlinear static or transient analyses which are numerically unstable. The more numerically unstable the model is, the more likely the convergence pattern or final results will differ as the number of cores used in the simulation is changed.


Release 18.2 - © ANSYS, Inc. All rights reserved.