4.2. Activating Distributed ANSYS

After you've completed the configuration steps, you can use several methods to start Distributed ANSYS: We recommend that you use the Mechanical APDL Product Launcher to ensure the correct settings. All methods are explained here.

Notes on Running Distributed ANSYS:

You can use an NFS mount to the ANSYS 18.2 installation; however, we do not recommend NFS-mounting the working directories. Doing so can result in significant declines in performance.

Only the master process reads the config.ans file. Distributed ANSYS ignores the /CONFIG,NOELDB and /CONFIG,FSPLIT commands.

The program limits the number of processes used to be less than or equal to the number of physical cores on the machine. This is done to avoid running the program on virtual cores (for example, by means of hyperthreading), which typically results in poor per-core performance. For optimal performance, consider closing down all other applications before launching Mechanical APDL.

4.2.1. Starting Distributed ANSYS via the Launcher

Use the following procedure to start Distributed ANSYS via the launcher.

This procedure is also used for running ANSYS LS-DYNA MPP. See LS-DYNA Parallel Processing Capabilities in the ANSYS LS-DYNA User's Guide for a detailed explanation on using ANSYS LS-DYNA's parallel processing capabilities.

  1. Open the Mechanical APDL Product Launcher:

    Windows:

    Start >Programs >ANSYS 18.2 >Mechanical APDL Product Launcher 18.2

    Linux:
    launcher182
  2. Select the correct environment and license.

  3. Go to the High Performance Computing Setup tab. Select Use Distributed Computing (MPP).

    Specify the MPI type to be used for this distributed run. MPI types include:

    • Intel MPI

    • IBM MPI

    • MS MPI (Windows only)

    See the MPI table in the beginning of this chapter for the specific MPI version for each platform. If you choose MS MPI, you cannot specify multiple hosts or an MPI file.

    Choose whether you want to run on a local machine, specify multiple hosts, or specify an existing MPI file (such as a host.list or an IBM MPI appfile):

    • If local machine, specify the number of cores you want to use on that machine.

    • If multiple hosts, select the machines you want to use from the list of available hosts. The list of available hosts is populated from the hosts182.ans file. Click on the machines you want to use and click Add to move them to the Selected Hosts list to use them for this run. If you click Add more than once for a machine, the number of cores to be used on that machine will increment each time, up to the maximum allowed in the hosts182.ans file. (Note that the Select Multiple Hosts option is not available when running the LS-DYNA MPP version on a Windows system.)

      You can also add or remove a host, but be aware that adding or removing a host from here will modify only this run; the hosts182.ans file will not be updated with any new information from this dialog box.

    • If specifying an MPI file, type in the full path to the file, or browse to the file. If typing in the path, you must use the absolute path.

    Additional Options for Linux Systems On Linux systems, you can choose to use the remote shell (rsh) protocol instead of the secure shell (ssh) protocol; ssh is the default.

    Also in the launcher, you can select the Use launcher-specified working directory on all nodes option on the High Performance Computing Setup tab. This option uses the working directory as specified on the File Management tab as the directory structure on the master and all nodes. If you select this option, all machines will require the identical directory structure matching the working directory specified on the launcher.

  4. Click Run to launch ANSYS.

4.2.2. Starting Distributed ANSYS via Command Line

You can also start Distributed ANSYS via the command line using the following procedures.

These procedures are also used for running ANSYS LS-DYNA MPP. To run the LS-DYNA MPP version, substitute lsdyna182 for ansys182 in the examples below. Only the command line options specific to Distributed ANSYS apply to LS-DYNA MPP (i.e., -dis, -np, -machines, -mpifile). See LS-DYNA Parallel Processing Capabilities in the ANSYS LS-DYNA User's Guide for a detailed explanation on using ANSYS LS-DYNA's parallel processing capabilities.

Running on a Local Host If you are running Distributed ANSYS locally (that is, running across multiple cores on a single machine), you need to specify the number of cores you want to use:

ansys182 -dis -np n

If you are using Intel MPI, you do not need to specify the MPI software via the command line option. To specify IBM MPI, use the -mpi ibmmpi command line option as shown below:

ansys182 -dis -mpi ibmmpi -np n 

For example, if you run a job in batch mode on a local host using four cores with an input file named input1 and an output file named output1, the launch commands for Linux and Windows would be as shown below.

On Linux:

ansys182 -dis -np 4 -b < input1 > output1  (for default Intel MPI)

or

ansys182 -dis -mpi ibmmpi -np 4 -b < input1 > output1  (for IBM MPI)

On Windows:

ansys182 -dis -np 4 -b -i input1 -o output1  (for default Intel MPI)

or

ansys182 -dis -mpi ibmmpi -np 4 -b -i input1 -o output1  (for IBM MPI)

Running on Multiple Hosts If you are running Distributed ANSYS across multiple hosts, you need to specify the number of cores you want to use on each machine:

ansys182 -dis -machines machine1:np:machine2:np:machine3:np

To specify IBM MPI (instead of the default Intel MPI), use the -mpi command line option as shown below:

ansys182 -dis -mpi ibmmpi -machines machine1:np:machine2:np:machine3:np

On Linux, you may also need to specify the shell protocol used by the MPI software. Distributed ANSYS uses the secure shell protocol by default, but in some cluster environments it may be necessary to force the use of the remote shell protocol. This can be done via the -usersh command line option.

Consider the following examples which assume a batch mode run using two machines (using four cores on one machine and using two cores on the other machine), with an input file named input1 and an output file named output1. The launch commands for Linux and Windows would be as shown below.

On Linux:

ansys182 -dis -b -machines machine1:4:machine2:2 < input1 > output1  (for default Intel MPI)

or

ansys182 -dis -mpi ibmmpi -b -machines machine1:4:machine2:2 < input1 > output1 (for IBM MPI)

or

ansys182 -dis -b -machines machine1:4:machine2:2 -usersh < input1 > output1  (for remote shell protocol between compute nodes)

On Windows:

ansys182 -dis -b -machines machine1:4:machine2:2 -i input1 -o output1  (for default Intel MPI)

or

ansys182 -dis -mpi ibmmpi -b -machines machine1:4:machine2:2 -i input1 -o output1  (for IBM MPI)

The first machine specified with -machines in a Distributed ANSYS run must be the host machine and must contain any files necessary for the initiation of your job (i.e., input file, config.ans file, database file, etc.).

If both the -np and -machines options are used on a single command line, the -np will be ignored.

Note that the -machines option is not available when running the LS-DYNA MPP version on a Windows system.

Specifying a Preferred Parallel Feature License If you have more than one HPC license feature, you can use the -ppf command line option to specify which HPC license to use for the parallel run. See HPC Licensing for more information.

4.2.3. Starting Distributed ANSYS via the HPC Job Manager

If you are running on Windows x64 systems using Microsoft HPC Pack (MS MPI), you need to use the HPC Job Manager to start Distributed ANSYS. For more information, refer to the following README files:

Program Files\ANSYS Inc\V182\commonfiles\MPI\WindowsHPC\README.mht

or

Program Files\ANSYS Inc\V182\commonfiles\MPI\WindowsHPC\README.docx

4.2.4. Starting Distributed ANSYS in the Mechanical Application (via ANSYS Workbench)

If you are running ANSYS Workbench, you can start a Distributed ANSYS job in the Mechanical application. Go to Tools > Solve Process Settings; select the remote solve process you want to use and click the Advanced button. To enable Distributed ANSYS, ensure that Distribute Solution (if possible) is selected (this is the default). If necessary, enter any additional command line arguments to be submitted; the options are described in Starting Distributed ANSYS via Command Line. If Distribute Solution (if possible) is selected, you do not need to specify the -dis flag on the command line.

If you are running a remote solution on multiple machines, use the -machines option to specify the machines and cores on which to run the job. If you are running a remote solution on one machine with multiple cores, specify the number of cores in the Max Number of Utilized Cores field; no command line arguments are needed.

For more information on running a distributed solution in ANSYS Workbench, see Using Solve Process Settings.

4.2.5. Using MPI appfiles

You can specify an existing MPI file (such as an Intel MPI appfile) on the command line, rather than typing out multiple hosts or a complicated command line:

ansys182 -dis -mpifile appfile_name

For an IBM MPI appfile, include the -mpi command line option:

ansys182 -dis -mpi ibmmpi -mpifile appfile_name

The format of the appfile is system-dependent.

If the file is not in the current working directory, you will need to include the full path to the file. The file must reside on the local machine.

You cannot use the -mpifile option in conjunction with the -np (local host) or -machines (multiple hosts) options.

If the Specify Multiple Hosts launcher option or the -machines command line option was used, ANSYS generates a default appfile named host.list. You can rename this file, move it, or modify it for future runs if necessary. See the documentation for your vendor/MPI type for details on working with the appfile.

Using the Intel MPI appfile Intel MPI uses an appfile to define the machines in the array (or the local host). A typical Intel MPI appfile might look like this:

-host mach1  -env env_vars env_var_settings -np 2 /ansys_inc/v182/ansys/bin/ansysdis182 -dis
-host mach2  -env env_vars env_var_settings -np 2 /ansys_inc/v182/ansys/bin/ansysdis182 -dis

Using the IBM MPI appfile IBM MPI uses an appfile to define the machines in the array (or the local host). A typical IBM MPI appfile might look like this:

-h mach1 -np 2 /ansys_inc/v182/ansys/bin/ansysdis182 -dis -mpi ibmmpi
-h mach2 -np 2 /ansys_inc/v182/ansys/bin/ansysdis182 -dis -mpi ibmmpi

See the IBM MPI user documentation for details on working with the IBM MPI appfile.

Using Other appfiles A typical appfile for other machines might look like this:

mach1
   mach1
   mach2
   mach2

4.2.6. Directory Structure Across Machines

Distributed ANSYS writes files to the master and slave machines (or compute nodes) as the analysis progresses. The working directory for each machine can be on a local disk drive or on a network shared drive. For optimal performance, local disk storage is recommended instead of network storage. (For more information see the Performance Guide.) You must set up the same working directory path structure on the master and slave machines.

Keep the following points in mind as you set up your cluster environment:

  • If you don't have identical working directory structures set up on the master and all slave machines, Distributed ANSYS will fail to launch.

  • Distributed ANSYS will always use the current working directory on the master machine and will expect identical directory structures to exist on all slave nodes. If you are using the launcher, the working directory specified on the File Management tab is the directory that Distributed ANSYS will expect.


Release 18.2 - © ANSYS, Inc. All rights reserved.