1. Setting up Distributed ANSYS

This section describes the prerequisites, including software requirements, for running Distributed ANSYS and the steps necessary to set up the environment for Distributed ANSYS.

The following topics are available:

1.1. Prerequisites for Running Distributed ANSYS

Whether you are running on a single machine or multiple machines, the following condition is true:

  • Distributed ANSYS allows you to use two cores without using any HPC licenses. Additional licenses will be needed to run a distributed solution with more than two cores. Several HPC license options are available. For more information, see HPC Licensing in the Parallel Processing Guide.

If you are running on a single machine, there are no additional requirements for running a distributed solution.

If you are running across multiple machines (for example, a cluster), your system must meet these additional requirements to run a distributed solution.

  • Homogeneous network: All machines in the cluster must be the same type, OS level, chip set, and interconnects.

  • You must be able to remotely log in to all machines, and all machines in the cluster must have identical directory structures (including the ANSYS 18.2 installation, MPI installation, and working directories). Do not change or rename directories after you've launched ANSYS. For more information, see Directory Structure Across Machines in the Parallel Processing Guide.

  • All machines in the cluster must have ANSYS 18.2 installed, or must have an NFS mount to the ANSYS 18.2 installation. If not installed on a shared file system, ANSYS 18.2 must be installed in the same directory path on all systems.

  • All machines must have the same version of MPI software installed and running. The table below shows the MPI software and version level supported for each platform.

1.1.1. MPI Software

The MPI software supported by Distributed ANSYS depends on the platform (see the table below).

The files needed to run Distributed ANSYS using Intel MPI or IBM Platform MPI are included on the installation media and are installed automatically when you install ANSYS 18.2. Therefore, when running on a single machine (for example, a laptop, a workstation, or a single compute node of a cluster) on Windows or Linux, or when running on a Linux cluster, no additional software is needed. However, when running on multiple Windows machines (a cluster), you must install the MPI software separately (see Installing the Software later in this section).

Table 2:  Platforms and MPI Software

PlatformMPI SoftwareMore Information
Linux, Intel Xeon EM64T and AMD Opteron 64-bit
Intel MPI 5.1.3
IBM MPI 9.1.4.2
Intel MPI: http://software.intel.com/en-us/articles/intel-mpi-library-documentation/
IBM MPI: http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/mpi/index.html

Windows 64-bit Windows 7 x64 / Windows 8.1 x64 / Windows 10 x64

Intel MPI 5.1.3
IBM MPI 9.1.4.2
Intel MPI: http://software.intel.com/en-us/articles/intel-mpi-library-documentation/
IBM MPI: http://www-03.ibm.com/systems/technicalcomputing/platformcomputing/products/mpi/index.html
Windows HPC Server 2012 x64Microsoft HPC Pack (MS MPI v7.1) http://www.microsoft.com/hpc/

ANSYS LS-DYNA If you are running ANSYS LS-DYNA, you can use LS-DYNA's parallel processing (MPP or SMP) capabilities. Use the launcher or the command line method as described in Activating Distributed ANSYS in the Parallel Processing Guide to run LS-DYNA MPP. For Windows and Linux systems, see the following table for LS-DYNA MPP MPI support. For more information on using ANSYS LS-DYNA in general, and its parallel processing capabilities specifically, see the ANSYS LS-DYNA User's Guide.

Table 3:  LS-DYNA MPP MPI Support on Windows and Linux

MPI version for DYNA MPP 64-bit Windows64-bit Linux
Intel MPIXX
MS MPIXn/a

1.1.2. Installing the Software

To run Distributed ANSYS on a cluster, you must install ANSYS 18.2 on all machines in the cluster, or have an NFS mount to the ANSYS 18.2 installation. Install ANSYS 18.2 following the instructions in the ANSYS, Inc. Installation Guide for your platform. Be sure to complete the installation, including all required post-installation procedures. On Windows systems, you must use the Universal Naming Convention (UNC) for all file and path names for Distributed ANSYS to work correctly.

Installing MPI software on Windows

You can install Intel MPI or IBM MPI from the installation launcher by choosing Install MPI for ANSYS, Inc. Parallel Processing. For installation instructions, see the following sections in the ANSYS, Inc. Windows Installation Guide:

Intel-MPI 5.1.3.180 Installation Instructions
IBM MPI 9.1.4.2 Installation Instructions
Microsoft HPC Pack (Windows HPC Server 2012)

You must complete certain post-installation steps before running Distributed ANSYS on a Microsoft HPC Server 2012 system. The post-installation instructions provided below assume that Microsoft HPC Server 2012 and Microsoft HPC Pack (which includes MS MPI) are already installed on your system. The post-installation instructions can be found in the following README files:

Program Files\ANSYS Inc\V182\commonfiles\MPI\WindowsHPC\README.mht

or

Program Files\ANSYS Inc\V182\commonfiles\MPI\WindowsHPC\README.docx

Microsoft HPC Pack examples are also located in Program Files\ANSYS Inc\V182\commonfiles\MPI\WindowsHPC. Jobs are submitted to the Microsoft HPC Job Manager either from the command line or the Job Manager GUI.

To submit a job via the GUI, go to Start> All Programs> Microsoft HPC Pack> HPC Job Manager. Then click on Create New Job from Description File.

1.2. Setting Up the Cluster Environment for Distributed ANSYS

After you've ensured that your cluster meets the prerequisites and you have ANSYS 18.2 and the correct version of MPI installed, you need to configure your distributed environment using the following procedure.

  1. Obtain the machine name for each machine on the cluster.

    Windows 7:

    From the Start menu, click Computer, then click on System properties. The full computer name is listed. Note the name of each machine (not including the domain).

    Windows 8.1:

    From the All apps screen, open the Control Panel. Click System and Security> System. The full computer name is listed. Note the name of each machine (not including the domain).

    Windows 10:

    From the Start menu, pick Settings >System >About. The full computer name is listed under PC Name. Note the name of each machine (not including the domain).

    Linux:

    Type hostname on each machine in the cluster. Note the name of each machine.

  2. Linux only: First determine if the cluster uses the secure shell (ssh) or remote shell (rsh) protocol.

    • For ssh: Use the ssh-keygen command to generate a pair of authentication keys. Do not enter a passphrase. Then append the new public key to the list of authorized keys on each compute node in the cluster that you wish to use.

    • For rsh: Create a .rhosts file in the home directory. Add the name of each compute node you wish to use on a separate line in the .rhosts file. Change the permissions of the .rhost file by issuing: chmod 600 .rhosts. Copy this .rhosts file to the home directory on each compute node in the cluster you wish to use.

    Verify communication between compute nodes on the cluster via ssh or rsh. You should not be prompted for a password. If you are, correct this before continuing. For more information on using ssh/rsh without passwords, search online for "Passwordless SSH" or "Passwordless RSH", or see the man pages for ssh or rsh.

  3. If you want the list of machines to be populated in the Mechanical APDL Product Launcher, you need to configure the hosts182.ans file. You can use the ANS_ADMIN utility to configure this file. You can manually modify the file later, but we strongly recommend that you use ANS_ADMIN to create this file initially to ensure that you establish the correct format.

    Windows 7:

    Start >All Programs >ANSYS 18.2 >Utilities >ANS_ADMIN 18.2

    Windows 8.1:

    All apps >ANSYS 18.2 >ANS_ADMIN 18.2

    Windows 10:

    Start >All apps >ANSYS 18.2 >ANS_ADMIN 18.2

    Linux:

    /ansys_inc/v182/ansys/bin/ans_admin182

    The ANS_ADMIN interface is similar for Windows and Linux:

    • Choose ANSYS/Workbench Configure Cluster to configure the hosts182.ans file.

    • Under Select file to configure, choose the hosts182.ans file to be configured and choose Configure for Distributed ANSYS. Click OK.

    • Enter the system name (from Step 1) in the Machine hostname field and click Add. On the next dialog box, enter the system type in the Machine type drop-down, and the number of cores in the Max number of jobs/processors field. Click Add. Repeat this step for each machine in the cluster.

    • When you are finished adding machines, click Close then File > Exit.

    The hosts182.ans file should be located in your current working directory, your home directory, or the apdl directory. All three locations are equally valid; the location should be chosen based on user preference.

  4. Windows only: Verify that all required environment variables are properly set. If you followed the post-installation instructions described above for Microsoft HPC Pack (Windows HPC Server 2012), these variables should be set automatically.

    On the head node, where ANSYS 18.2 is installed, check these variables:

    ANSYS182_DIR=C:\Program Files\ANSYS Inc\v182\ansys

    ANSYSLIC_DIR=C:\Program Files\ANSYS Inc\Shared Files\Licensing

    where C:\Program Files\ANSYS Inc is the location of the product install and C:\Program Files\ANSYS Inc\Shared Files\Licensing is the location of the licensing install. If your installation locations are different than these, specify those paths instead.

    On Windows systems, you must use the Universal Naming Convention (UNC) for all ANSYS, Inc. environment variables on the compute nodes for Distributed ANSYS to work correctly.

    On the compute nodes, check these variables:

    ANSYS182_DIR=\\head_node_machine_name\ANSYS Inc\v182\ansys

    ANSYSLIC_DIR=\\head_node_machine_name\ANSYS Inc\Shared Files\Licensing

    For distributed LS-DYNA:

    On the head node and the compute nodes, set LSTC_LICENSE to ANSYS. This tells the LS-DYNA executable to use ANSYS, Inc. licensing.

    Since the LS-DYNA run will use ANSYS, Inc. licensing for LS-DYNA, you do not need to set LSTC_LICENSE_SERVER.

  5. Windows only: Share out the ANSYS Inc directory on the head node with full permissions so that the compute nodes can access it.

1.2.1. Optional Setup Tasks

The tasks explained in this section are optional. They are not required to get Distributed ANSYS to run correctly, but they may be useful for achieving the most usability and efficiency, depending on your system configuration.

On Linux systems, you can also set the following environment variables:

  • ANSYS_NETWORK_START - This is the time, in seconds, to wait before timing out on the start-up of the client (default is 15 seconds).

  • ANSYS_NETWORK_COMM - This is the time to wait, in seconds, before timing out while communicating with the client machine (default is 5 seconds).

  • ANS_SEE_RUN_COMMAND - Set this environment variable to 1 to display the actual command issued by ANSYS.

On Windows systems, you can set the following environment variables to display the actual command issued by ANSYS:

  • ANS_SEE_RUN = TRUE

  • ANS_CMD_NODIAG = TRUE

1.2.2. Using the mpitest Program

The mpitest program performs a simple communication test to verify that the MPI software is set up correctly. The mpitest program should start without errors. If it does not, check your paths and permissions; correct any errors and rerun.

When running the mpitest program, you must use an even number of nodes.

On Linux:

For Intel MPI (default), issue the following command:

mpitest182 -machines machine1:2

For IBM MPI, issue the following command:

mpitest182 -mpi ibmmpi -machines machine1:2

You can use any of the same command line arguments (such as -machines) with the mpitest program as you can with Distributed ANSYS.

On Windows:

Issue the following command to run a local test on Windows using Intel MPI:

ansys182 -np 2 -mpitest

Use the following procedure to run a distributed test on Windows using Intel MPI:

  1. Create a file named machines in your local/home directory. Open the machines file in an editor.

  2. Add your master and slave machines in your cluster. For example, in this cluster of two machines, the master machine is gowindows1. List the machine name separately for each core on that machine. For example, if gowindows1 has four cores and gowindows2 has two, the machines file would look like this:

    gowindows1
    gowindows1
    gowindows1
    gowindows1
    gowindows2
    gowindows2
  3. From a command prompt, navigate to your working directory. Run the following:

    ansys182 -mpifile machines -mpitest

1.2.3. Interconnect Configuration

Low-end hardware, such as slow interconnects, will reduce the speed improvements you see in a distributed parallel simulation. For optimal performance, we typically recommend that you use an interconnect with a high communication bandwidth (1000 megabytes/second or higher) and a low communication latency (5 microseconds or lower). This is due to the significant amount of data that must be transferred between processes during a distributed parallel simulation.

Distributed ANSYS supports the following interconnects. Not all interconnects are available on all platforms; see http://www.ansys.com/Support/Platform+Support for a current list of supported interconnects. Other interconnects may work but have not been tested.

  • InfiniBand (recommended)

  • GigE

On Windows x64 systems, use the Network Wizard in the Compute Cluster Administrator to configure your interconnects. See the Compute Cluster Pack documentation for specific details on setting up the interconnects. You may need to ensure that Windows Firewall is disabled for Distributed ANSYS to work correctly.


Release 18.2 - © ANSYS, Inc. All rights reserved.