1. ANSYS CFX Linux Parallel Setup

The following are required in order for you to execute ANSYS CFX parallel:

1.1. Setting Up Remote Access on Linux

Each system that will be used as a slave node must be configured to allow access via remote shell commands from the master node. This can be done globally for all users or on a per-user basis.

Often, networks where ssh, rsh or rlogin are used frequently will already be configured to allow remote access to all users. If this is the case, nothing more must be done.

The CFX5RSH environment variable is used to select the remote shell protocol. By default, CFX5RSH=ssh is set. If you want to use rsh, then set CFX5RSH=rsh either in the environment or in the cfx5rc file (see Resources Set in cfx5rc Files in the CFX Introduction).

In either case, ssh or rsh, the remote machine must not prompt for a password when you run remote commands.

1.1.1. Testing Remote Access

You can test remote access using ssh using the command:

ssh unixhost echo working

You can test remote access using rsh for a Linux slave node using the command:

rsh unixhost echo working

1.1.2. Set Up of ssh

If you use ssh for remote access, consult your system documentation on how to set up ssh between machines so that it does not require the user to enter a password.

1.1.3. Global Set Up of rsh

This method, which is used to allow remote access for all users, depends on whether NIS is used to maintain netgroups, as well as common password and group databases. If this is not the case, then you should log in to each slave node as root and create a file called /etc/hosts.equiv containing a line:

<master>

where <master> is the hostname of the master node. See your system documentation for more information about the use of rsh and the syntax of the hosts.equiv file.

If NIS is used to control remote access, then a netgroup must be created for users of ANSYS CFX by the root user on the NIS master server, and a line such as the one below added to /etc/hosts.equiv on each slave node by the root user:

+@<netgroup>

where <netgroup> is the name of a netgroup to which users of ANSYS CFX belong. A detailed description of how to configure NIS is beyond the scope of this manual. See your system documentation for more information about NIS.

1.1.4. Individual User Set Up for rsh

Individual users can create a file called .rhosts in their home directory on each slave containing a line:

<master> <user>

where <master> is the hostname of the master and <user> is their username on the master. This file should be made readable only by the user, for example, by running:

chmod 600 ~/.rhosts 

1.2. hostinfo.ccl File

In order to use the Distributed Parallel mode of ANSYS CFX, the file hostinfo.ccl must exist in the <CFXROOT>/config/ directory of the ANSYS CFX installation on the master node and be made readable by all users of the software. This file is a database containing information about the available hosts and where ANSYS CFX has been installed on each of them. The file is used by the ANSYS CFX-Solver when constructing a parallel run.

This file is written using the CFX Command Language. It defines a set of HOST DEFINITION objects, one for each available node. For example:

SIMULATION CONTROL:
  EXECUTION CONTROL:
    PARALLEL HOST LIBRARY:  
      HOST DEFINITION: kangaroo   
        Installation Root = /ansys_inc/v%v/CFX  
      END  
      HOST DEFINITION:wallaby   
        Installation Root = /usr/local/cfx   
        Host Architecture String = linux-amd64
        Number of Processors = 16
        Relative Speed = 1.7  
      END  
      HOST DEFINITION: mypc   
        Remote Host Name = my_pc  
      END 
    END
  END
END

Note:  If you manually create your hostinfo.ccl file, you must ensure that the SIMULATION CONTROL...END wrapper is present.


None of the values for each host are mandatory, and the following empty host definition is perfectly valid:

HOST DEFINITION: parakeet
END

Host names are limited to the set of valid CCL object names. In particular, they must not contain full stops (.) or underscores (_) and must start with a letter.

If a hostinfo.ccl file does not already exist when ANSYS CFX is installed, one will be created containing the installation host. You can add hosts to the hostinfo.ccl file using the cfx5parhosts utility, or by modifying the file using a text editor. Individual users may also create their own versions of this file in:

~/.hostinfo.ccl

which will be used in preference if it exists.

For most installations, it will be necessary to supply only the Installation Root parameter, which should point to the <CFXROOT> directory in which ANSYS CFX is installed. On mixed architectures, you may find it useful to supply the Number of Processors and/or Relative Speed parameters. A little time may be saved at startup by giving the Host Architecture String parameter explicitly, for example, using the cfx5parhosts utility.


Tip:  If an individual user wants to use a host that is not present in either the hostinfo.ccl in the installation config directory or the user's own version of this file, then the user can add this host to the list for a particular run by using the CFX-Solver Manager. However, this would have to be done each time a run is started.


The available parameters for the hostinfo.ccl file are as follows:

Installation Root

This is set to the <CFXROOT> installation directory on this host. If it is set to the special string none, this indicates that there is no ANSYS CFX installation on the remote host, which can sometimes be useful if only the solver binary is available.

Host Architecture String

ANSYS CFX will use this value to select the most appropriate solver executable for this node. These strings can be obtained by giving the command <CFXROOT>/bin/cfx5info -os on the node in question. When these short os values (for example, linux-amd64) are given in this position, the generic solver executable will always be used for this host. The string can also be set to the actual architecture of the remote host (for example, intel_xeon64.sse2_linux2.3.4), which is determined by giving the command <CFXROOT>/bin/cfx5info -arch. If these longer strings are used then CFX could use this information to select between solvers optimized for specific architectures. However, since there are currently no solvers optimized for specific architectures, this extra information is currently unused.

Number of Processors

As implied, this is the number of processors on the machine. It is used for display purposes only and can be safely omitted.

Relative Speed

The Relative Speed is a real number that is used by the ANSYS CFX-Solver when partition sizes are calculated. The ratio of relative speeds of each host is used to determine the size of each partition. As an example, consider a parallel run involving two machines, one with a relative speed of 2.0 and the other with a relative speed of 1.0. The faster machine would work on a partition size twice as large as the slower machine in this case.

The numbers themselves are arbitrary; as a guide you may want to set 1.0 for a 1 GHz processor, 0.75 for a 750 MHz processor, and so on. If a relative speed entry is not entered, a default of 1.0 is assumed for that host.

You can obtain relative speed values using the cfx5parhosts utility.

Remote Host Name

To include hosts in the parallel run with names that contain, for example, underscore characters, you can add the “Remote Host Name” parameter to the HOST DEFINITION with the correct network name for the host and use a readable alias as the name of the object.

Solver Executable

A solver executable may be explicitly supplied, if necessary. This is usually only required when using Installation Root = none, and is recommended for advanced users only. The following substitutions are made on the string:

%rroot directory of the installation
%pparallel suffix for the executable
%vversion of ANSYS CFX being run
%ooperating system string
%aarchitecture subdirectory specification; for example, linux/double

If it is not supplied, this parameter has the default value %r/bin/%a/solver%p.exe

1.2.1. Adding Hosts for Parallel Processing with the cfx5parhosts Utility

You can add new hosts to ANSYS CFX's database for parallel runs using the cfx5parhosts utility:

CFXROOT\bin\cfx5parhosts argument list

where argument list corresponds to one or more of the arguments listed below:

ArgumentDescription
-add host[,host,...]

Add information about the named hosts to the file. This assumes that ANSYS CFX is installed in the same directory on each listed host as on the host on which you are running.

host may be specified as [user@]hostname[:cfx-5 root] if the user name or the ANSYS CFX installation root directory differs from the local host.

-benchmark

Runs a standard benchmark case, and fills in the Relative Speed for the local host. The benchmark case will usually take less than 5 minutes to run.

-file file

Use the specified file as the hostinfo.ccl file.

-merge fileMerge host information from the named file.
-no-update After modifying the file, write back the information available without attempting to fill in any missing pieces.
-strict Used with -update. Normally, hosts which exist on the network but cannot be connected to with rsh or ssh are included in the resulting file with a comment. This switch will exclude these hosts.
-system Use the system host file. This is the default.
-update Updates the specified host information file. If any hosts do not have an architecture specified in the existing hostinfo.ccl file (for example, because it was added via the -add switch), it will connect to the host and query it to fill in the Host Architecture String parameter. This is the default behavior. Note that if the Installation Root parameter is incorrect for the host, it will use the standard system commands to guess a generic architecture string. This can happen if you use -add to include a host with a different installation directory than the local one.
-user Use the per-user host file.

1.3. Using Intel MPI (Message Passing Interface Library)

Most Linux systems support two parallel run modes: Intel MPI and IBM MPI. Intel MPI is the preferred parallel run mode on most of the supported ANSYS CFX platforms.

Intel MPI supports several network fabrics including:

  • Shared memory, typically used for intra-host communication.

  • DAPL-capable network fabrics, such as Infiniband, iWarp, Dolphin and XPMEM (through DAPL).

  • TCP/IP-capable network fabrics, such as Ethernet and InfiniBand (through IPoIB).

  • TMI-capable network fabrics including Qlogic and Myrianet (through Tag Matching Interface).

  • OFA-capable network fabrics including InfiniBand (through OFED verbs).

Intel MPI is automatically installed with ANSYS CFX in the<CFXROOT>/../commonfiles/ directory. You do not need a license from Intel to run the ANSYS CFX-Solver using Intel MPI.

Intel MPI run modes are selected in the ANSYS CFX-Solver Manager or on the command line (using the -start-method command line option) using the “Intel MPI Local Parallel” or “Intel MPI Distributed Parallel” start method options.

1.3.1. Environment Variables

In most cases, the default settings are sufficient and it may not be necessary to set any environment variables.

ANSYS CFX uses the environment variable CFX5_IMPI_DIR to select which Intel MPI installation is used. The default setting points to the version that is automatically installed by ANSYS CFX in the<CFXROOT>/../commonfiles/MPI/Intel directory. You can use cfx5info -v to find the current settings for CFX5_IMPI_DIR and other environment variables. If you want to use a different Intel MPI version, you can install that version in an alternative location and set CFX5_IMPI_DIR to that location.

There are several environment variables that can be used to control Intel MPI that may provide additional flexibility that is not directly accessible through the ANSYS CFX start-up scripts. For example, environment variables are available for network fabric selection (discussed briefly below), process pinning control, and collective operation control.

Any of these environment variables can be set by modifying the start command in the etc/start-methods.ccl file and using the Intel MPI options -env and/or -genv (an example of this usage is given in the section on network fabric selection). More detail on all options and environment variables is available in the Intel MPI documentation.

Some useful environment variables include those described below. If these environment variables are set the start-up scripts automatically set them for your ANSYS CFX runs.

I_MPI_MPD_REMSH

Specifies which method is used by Intel MPI to spawn jobs on remote machines. The Intel MPI default is ssh. However, by default the ANSYS CFX software sets this to be the same as the setting used for CFX5RSH, and rsh if CFX5RSH is not set. This variable is applicable only to Linux.

I_MPI_FABRICS and I_MPI_FABRICS_LIST

Specifies the networks fabrics to be used and the order in which Intel MPI searches for them.

1.3.2. Network Fabrics Selection

Intel MPI start methods use the default network fabrics selection built into Intel MPI. The default order in which Intel MPI checks for fabrics is DAPL, OFA, TCP, TMI. The environment variable I_MPI_FABRICS_LIST controls the search order used by Intel MPI.

If you want to force selection of a specific network fabric, then you can either set the I_MPI_FABRICS environment variable or modify the start command in the etc/start-methods.ccl file and add the necessary command line flags (for example: -env IMPI_FABRICS shm:dapl to select the combination of shared memory and DAPL fabrics) .

The following table shows how Intel MPI searches for drivers for the various network fabrics:

Interconnect1st attempt2nd attempt
shmN/AN/A
DAPLEnvironment variable I_MPI_DAT_LIBRARYlibdat.so or libdat.so.1 for DAPL 1.2 providers and libdat2.so or libdat2.so.1 for DAPL 1.2 providers
TCPN/AN/A
TMIEnvironment variable I_MPI_TMI_LIBRARY libtmi.so
OFAEnvironment variable I_MPI_OFA_LIBRARYlibibverbs.so

For example, Intel MPI expects that the DAT shared library for the DAPL network fabric is located either in the default library search path or in a specific location if a full path is specified. If the shared library cannot be found, Intel MPI will not take advantage of the DAPL network fabric. To fix this, you will have to install the shared libraries into the correct location or set the environment variable I_MPI_DAT_LIBRARY to point to the correct DAT shared library.

There are other environment variables not documented here which provide further control over network fabrics selection. These are described in the Intel MPI documentation.

1.4. Using IBM MPI (Message Passing Interface Library)

IBM MPI is available as an alternative parallel run mode on all the supported ANSYS CFX platforms.

IBM MPI supports several communication modes including shared memory and networked TCP/IP (Ethernet), luDAPL, VAPI, OpenFabrics (IB), GM-2, MX (Myrinet), Elan4 (Quadrics), QLogic InfiniPath, and EtherFabric (Level 5 Networks).

IBM MPI is automatically installed with ANSYS CFX in the <CFXROOT>/tools directory. You do not need a license from IBM to run the ANSYS CFX-Solver using IBM MPI.

IBM MPI run modes are selected in the ANSYS CFX-Solver Manager or on the command line (using the -start-method command line option) using the “IBM MPI Local Parallel” or “IBM MPI Distributed Parallel” start method options.

For IBM MPI, the –cpu_bind option is used by default and binds processes to cores using a cyclic order. This leads to better performance but, on multi-user clusters having scheduling systems that do not enforce exclusive use of cluster nodes, may cause problems because multiple jobs may be bound to the same CPU on the same node. In this case, the –cpu_bind argument to mpirun should be removed from the IBM MPI start methods in the <CFXROOT>/etc/start_methods.ccl file.

1.4.1. Environment Variables

In most cases, the default settings are sufficient and it may not be necessary to set any environment variables.

ANSYS CFX uses the environment variable CFX5_IBMMPI_DIR to select which IBM MPI installation is used. The default setting points to the version that is automatically installed with ANSYS CFX in the <CFXROOT>/../commonfiles/MPI/IBM directory. You can use cfx5info -v to find the current settings for CFX5_IBMMPI_DIR and other environment variables. If you want to use a different IBM MPI version, you can install that version in an alternative location and set CFX5_IBMMPI_DIR to that location. For distributed parallel runs, IBM MPI must be present at the same location (same absolute path) on all hosts. Typically, this is achieved by using a central ANSYS CFX installation, accessible by all hosts. In order to run a CFX job in distributed parallel using IBM MPI on hosts with different CFX installation locations, you must first install IBM MPI under the same location (same absolute path) on each host, then set the environment variable CFX5_IBMMPI_DIR to the location under which IBM MPI was installed.

There are several environment variables that can be used to control IBM MPI (documented in the IBM MPI documentation) that may provide additional flexibility not directly accessible through the ANSYS CFX startup scripts. If these environment variables are set the startup scripts automatically set them for your ANSYS CFX runs. Some useful environment variables include:

MPI_REMSH

Enables you to change which method is used by IBM MPI to spawn jobs on remote machines. The IBM MPI default is ssh. However, by default the ANSYS CFX software sets this to be the same as the setting used for CFX5RSH, and rsh if CFX5RSH is not set. This variable is applicable only to Linux and HP-UX.

MPI_IC_ORDER

Enables you to change the default order in which IBM MPI searches for interconnects.

1.4.2. Interconnect Selection

IBM MPI start methods use the default communication mode selection built into IBM MPI. The default order in which IBM MPI checks for interconnects is OpenFabric, VAPI, UDAPL, ITAPI, PSM, Myrinet MX, Myrinet GM, Elan, TCP/IP.

If you want to force selection of a specific device, then you can either modify the start command in the etc/start-methods.ccl file and add the necessary command line flags (for example: -GM to force selection of the Myrinet interconnect) or use the appropriate IBM MPI environment variables.

Some useful command line options include forcing other interconnect types (-TCP, -HMP, -ELAN, -ITAPI, -VAPI, -UDAPL, and so on), using prun on systems with quadrics support (-prun), XC support (-srun) and setting the subnet you want to use (-subnet, sometimes useful for selecting particular interconnects as well if your system is configured that way).

For some Linux-based cluster environments not supported by IBM MPI, it is possible that the Myrinet, Infiniband or Quadrics hardware drivers are not located where IBM MPI expects. If IBM MPI cannot detect the drivers for fast interconnects, it will generally default back to the lowest speed connection that it can detect (usually TCP/IP socket communication).

The following table gives information on the default search path IBM MPI uses to find drivers for the various communication interfaces:

Interconnect1st attempt2nd attempt3rd attempt
Infiniband (IB)Environment variable MPI_ICLIB_ITAPIlibitapi.so/usr/voltaire/lib/libitapi.so
Myrinet (GM)Environment variable MPI_ICLIB_GMlibgm.so OR libgm32.so/opt/gm/lib/libgm.so OR /opt/gm/lib/libgm32.so
ELANEnvironment variable MPI_ICLIB_ELANlibelan.sonone
UDAPLEnvironment variable MPI_ICLIB_UDAPL libdat.so none
VAPIEnvironment variable MPI_ICLIB_VAPIEnvironment variable MPI_ICLIB_VAPIDIRlibmtl_common.so, libmpga.so, libmosal.so, libvapi.so
InfinipathEnvironment variable MPI_ICLIB_PSMlibpsm_infinipath.so.1/usr/lib64/ libpsm_infinipath.so.1

For example, IBM MPI expects that GM stack shared libraries for Myrinet interconnects are located either in the default library search path or in the /opt/gm/lib64 (64 bit x86) directories. If the shared libraries are not in either location, IBM MPI will not take advantage of the high-speed interconnects. To fix this, you will have to copy the shared libraries into the correct location or set the environment variable MPI_ICLIB_GM to point to the correct GM stack shared library.

1.5. Using Cray MPI (Message Passing Interface Library)

The ANSYS CFX-Solver supports Cray’s native implementation of MPI on the Cray Linux Environment (CLE) using the start method ‘Cray MPI Distributed Parallel’.

Cray MPI is closely integrated with the batch queuing system and interconnects used on Cray systems.

Cray MPI is the recommended method of running on Cray systems, as opposed to using Cluster Compatibility Mode (CCM) with a different MPI.

When starting a solver run, the Cray MPI start method may be specified on the command line using the option: -start-method “Cray MPI Distributed Parallel”, having reserved the required processing elements via the built-in batch queuing system.

1.5.1. Cray MPI Options

The aprun command, which can be modified, is used by the start method to launch the job. Other options available from Cray are described in the documentation for aprun.

Note that numa node memory containment is enforced, using the -ss option. The default method used by Cray MPI to allocate processes to CPU cores is appropriate when all cores are used. However, if it is intended that each host is only partly utilized (for example, because of the memory requirements of the run), this can result in asymmetric utilization. In such cases, the -S option can be used to limit the number of processes per numa node. For example, -S 4 would limit the number of processes per numa node to 4.


Release 18.2 - © ANSYS, Inc. All rights reserved.