Before running an analysis with Distributed ANSYS, you must configure your system properly. Your system must meet specified criteria and you must have supported MPI software correctly installed.
The following topics are available for configuring and starting Distributed ANSYS:
Whether you are running on a single machine or multiple machines, the following condition is true:
Distributed ANSYS allows you to use two cores without using any HPC licenses. Additional licenses will be needed to run a distributed solution with more than two cores. Several HPC license options are available. For more information, see HPC Licensing in the Parallel Processing Guide.
If you are running on a single machine, there are no additional requirements for running a distributed solution.
If you are running across multiple machines (for example, a cluster), your system must meet these additional requirements to run a distributed solution.
Homogeneous network: All machines in the cluster must be the same type, OS level, chip set, and interconnects.
You must be able to remotely log in to all machines, and all machines in the cluster must have identical directory structures (including the ANSYS 18.2 installation, MPI installation, and working directories). Do not change or rename directories after you've launched ANSYS. For more information, see Directory Structure Across Machines in the Parallel Processing Guide.
All machines in the cluster must have ANSYS 18.2 installed, or must have an NFS mount to the ANSYS 18.2 installation. If not installed on a shared file system, ANSYS 18.2 must be installed in the same directory path on all systems.
All machines must have the same version of MPI software installed and running. The table below shows the MPI software and version level supported for each platform.
The MPI software supported by Distributed ANSYS depends on the platform (see the table below).
The files needed to run Distributed ANSYS using Intel MPI or IBM Platform MPI are included on the installation media and are installed automatically when you install ANSYS 18.2. Therefore, when running on a single machine (for example, a laptop, a workstation, or a single compute node of a cluster) on Windows or Linux, or when running on a Linux cluster, no additional software is needed. However, when running on multiple Windows machines (a cluster), you must install the MPI software separately (see Installing the Software later in this section).
Table 4.2: Platforms and MPI Software
| Platform | MPI Software | More Information | ||||
|---|---|---|---|---|---|---|
| Linux, Intel Xeon EM64T and AMD Opteron 64-bit |
|
| ||||
|
Windows 64-bit Windows 7 x64 / Windows 8.1 x64 / Windows 10 x64 |
|
| ||||
| Windows HPC Server 2012 x64 | Microsoft HPC Pack (MS MPI v7.1) | http://www.microsoft.com/hpc/ |
ANSYS LS-DYNA If you are running ANSYS LS-DYNA, you can use LS-DYNA's parallel processing (MPP or SMP) capabilities. Use the launcher or the command line method as described in Activating Distributed ANSYS in the Parallel Processing Guide to run LS-DYNA MPP. For Windows and Linux systems, see the following table for LS-DYNA MPP MPI support. For more information on using ANSYS LS-DYNA in general, and its parallel processing capabilities specifically, see the ANSYS LS-DYNA User's Guide.
Table 4.3: LS-DYNA MPP MPI Support on Windows and Linux
| MPI version for DYNA MPP | 64-bit Windows | 64-bit Linux |
|---|---|---|
| Intel MPI | X | X |
| MS MPI | X | n/a |
To run Distributed ANSYS on a cluster, you must install ANSYS 18.2 on all machines in the cluster, or have an NFS mount to the ANSYS 18.2 installation. Install ANSYS 18.2 following the instructions in the ANSYS, Inc. Installation Guide for your platform. Be sure to complete the installation, including all required post-installation procedures. On Windows systems, you must use the Universal Naming Convention (UNC) for all file and path names for Distributed ANSYS to work correctly.
You can install Intel MPI or IBM MPI from the installation launcher by choosing . For installation instructions, see the following sections in the ANSYS, Inc. Windows Installation Guide:
| Intel-MPI 5.1.3.180 Installation Instructions |
| IBM MPI 9.1.4.2 Installation Instructions |
You must complete certain post-installation steps before running Distributed ANSYS on a Microsoft HPC Server 2012 system. The post-installation instructions provided below assume that Microsoft HPC Server 2012 and Microsoft HPC Pack (which includes MS MPI) are already installed on your system. The post-installation instructions can be found in the following README files:
Program Files\ANSYS Inc\V182\commonfiles\MPI\WindowsHPC\README.mht
or
Program Files\ANSYS Inc\V182\commonfiles\MPI\WindowsHPC\README.docx
Microsoft HPC Pack examples are also located in Program Files\ANSYS Inc\V182\commonfiles\MPI\WindowsHPC. Jobs are submitted to the Microsoft HPC Job Manager either from the command line or the Job Manager GUI.
To submit a job via the GUI, go to . Then click on .
After you've ensured that your cluster meets the prerequisites and you have ANSYS 18.2 and the correct version of MPI installed, you need to configure your distributed environment using the following procedure.
Obtain the machine name for each machine on the cluster.
From the Start menu, click Computer, then click on System properties. The full computer name is listed. Note the name of each machine (not including the domain).
From the All apps screen, open the Control Panel. Click System and Security> System. The full computer name is listed. Note the name of each machine (not including the domain).
From the Start menu, pick . The full computer name is listed under PC Name. Note the name of each machine (not including the domain).
Type hostname on each machine in the
cluster. Note the name of each machine.
Linux only: First determine if the cluster uses the secure shell (ssh) or remote shell (rsh) protocol.
For ssh: Use the ssh-keygen command to generate a pair of authentication keys. Do not enter a passphrase. Then append the new public key to the list of authorized keys on each compute node in the cluster that you wish to use.
For rsh: Create a
.rhosts file in the home directory. Add the
name of each compute node you wish to use on a separate line in the
.rhosts file. Change the permissions of the
.rhost file by issuing: chmod 600
.rhosts. Copy this .rhosts file to
the home directory on each compute node in the cluster you wish to
use.
Verify communication between compute nodes on the cluster via ssh or rsh. You should not be prompted for a password. If you are, correct this before continuing. For more information on using ssh/rsh without passwords, search online for "Passwordless SSH" or "Passwordless RSH", or see the man pages for ssh or rsh.
If you want the list of machines to be populated in the Mechanical APDL Product Launcher, you need to configure the hosts182.ans file. You can use the ANS_ADMIN utility to configure this file. You can manually modify the file later, but we strongly recommend that you use ANS_ADMIN to create this file initially to ensure that you establish the correct format.
/ansys_inc/v182/ansys/bin/ans_admin182
The ANS_ADMIN interface is similar for Windows and Linux:
Choose to configure the hosts182.ans file.
Under Select file to configure, choose the hosts182.ans file to be configured and choose . Click OK.
Enter the system name (from Step 1) in the Machine hostname field and click . On the next dialog box, enter the system type in the Machine type drop-down, and the number of cores in the Max number of jobs/processors field. Click . Repeat this step for each machine in the cluster.
When you are finished adding machines, click then .
The hosts182.ans file should be located in your current working directory, your home directory, or the apdl directory. All three locations are equally valid; the location should be chosen based on user preference.
Windows only: Verify that all required environment variables are properly set. If you followed the post-installation instructions described above for Microsoft HPC Pack (Windows HPC Server 2012), these variables should be set automatically.
On the head node, where ANSYS 18.2 is installed, check these variables:
ANSYS182_DIR=C:\Program Files\ANSYS Inc\v182\ansys
ANSYSLIC_DIR=C:\Program Files\ANSYS Inc\Shared Files\Licensing
where C:\Program Files\ANSYS Inc is the location of the product install and C:\Program Files\ANSYS Inc\Shared Files\Licensing is the location of the licensing install. If your installation locations are different than these, specify those paths instead.
On Windows systems, you must use the Universal Naming Convention (UNC) for all ANSYS, Inc. environment variables on the compute nodes for Distributed ANSYS to work correctly.
On the compute nodes, check these variables:
ANSYS182_DIR=\\head_node_machine_name\ANSYS Inc\v182\ansys
ANSYSLIC_DIR=\\head_node_machine_name\ANSYS Inc\Shared Files\Licensing
For distributed LS-DYNA:
On the head node and the compute nodes, set LSTC_LICENSE to ANSYS. This tells the LS-DYNA executable to use ANSYS, Inc. licensing.
Since the LS-DYNA run will use ANSYS, Inc. licensing for LS-DYNA, you do not need to set LSTC_LICENSE_SERVER.
Windows only: Share out the ANSYS Inc directory on the head node with full permissions so that the compute nodes can access it.
The tasks explained in this section are optional. They are not required to get Distributed ANSYS to run correctly, but they may be useful for achieving the most usability and efficiency, depending on your system configuration.
On Linux systems, you can also set the following environment variables:
ANSYS_NETWORK_START - This is the time, in seconds, to wait before timing out on the start-up of the client (default is 15 seconds).
ANSYS_NETWORK_COMM - This is the time to wait, in seconds, before timing out while communicating with the client machine (default is 5 seconds).
ANS_SEE_RUN_COMMAND - Set this environment variable to 1 to display the actual command issued by ANSYS.
On Windows systems, you can set the following environment variables to display the actual command issued by ANSYS:
ANS_SEE_RUN = TRUE
ANS_CMD_NODIAG = TRUE
The mpitest program performs a simple communication test to verify that the MPI software is set up correctly. The mpitest program should start without errors. If it does not, check your paths and permissions; correct any errors and rerun.
When running the mpitest program, you must use an even number of nodes.
On Linux:
For Intel MPI (default), issue the following command:
mpitest182 -machines machine1:2For IBM MPI, issue the following command:
mpitest182 -mpi ibmmpi -machines machine1:2You can use any of the same command line arguments (such as
-machines) with the mpitest program as you can
with Distributed ANSYS.
On Windows:
Issue the following command to run a local test on Windows using Intel MPI:
ansys182 -np 2 -mpitest
Use the following procedure to run a distributed test on Windows using Intel MPI:
Create a file named machines in your local/home directory. Open the machines file in an editor.
Add your master and slave machines in your cluster. For example, in this
cluster of two machines, the master machine is
gowindows1. List the machine name separately for each
core on that machine. For example, if gowindows1 has four
cores and gowindows2 has two, the
machines file would look like this:
| gowindows1 |
| gowindows1 |
| gowindows1 |
| gowindows1 |
| gowindows2 |
| gowindows2 |
From a command prompt, navigate to your working directory. Run the following:
ansys182 -mpifile machines -mpitest
Low-end hardware, such as slow interconnects, will reduce the speed improvements you see in a distributed parallel simulation. For optimal performance, we typically recommend that you use an interconnect with a high communication bandwidth (1000 megabytes/second or higher) and a low communication latency (5 microseconds or lower). This is due to the significant amount of data that must be transferred between processes during a distributed parallel simulation.
Distributed ANSYS supports the following interconnects. Not all interconnects are available on all platforms; see http://www.ansys.com/Support/Platform+Support for a current list of supported interconnects. Other interconnects may work but have not been tested.
InfiniBand (recommended)
GigE
On Windows x64 systems, use the Network Wizard in the Compute Cluster Administrator to configure your interconnects. See the Compute Cluster Pack documentation for specific details on setting up the interconnects. You may need to ensure that Windows Firewall is disabled for Distributed ANSYS to work correctly.