The following are required in order for you to execute ANSYS CFX parallel:
The ANSYS CFX-Solver must be installed on both master and slave nodes.
To run distributed parallel (where slave processes run on a different host to the master process), remote or secure shell access must be available from the master nodes (systems on which parallel runs will be started) to slave nodes (systems on which the ANSYS CFX-Solver will actually run). See Setting Up Remote Access on Linux, below.
The hostinfo.ccl file should be set up, as described in hostinfo.ccl File.
Each system that will be used as a slave node must be configured to allow access via remote shell commands from the master node. This can be done globally for all users or on a per-user basis.
Often, networks where ssh, rsh or rlogin are used frequently will already be configured to allow remote access to all users. If this is the case, nothing more must be done.
The CFX5RSH environment variable is used to select the
remote shell protocol. By default, CFX5RSH=ssh is set. If
you want to use rsh, then set CFX5RSH=rsh either in the
environment or in the cfx5rc file (see Resources Set in cfx5rc Files in the CFX Introduction).
In either case, ssh or rsh, the remote machine must not prompt for a password when you run remote commands.
You can test remote access using ssh using the command:
ssh unixhost echo working
You can test remote access using rsh for a Linux slave node using the command:
rsh unixhost echo working
If you use ssh for remote access, consult your system documentation on how to set up ssh between machines so that it does not require the user to enter a password.
This method, which is used to allow remote access for all users, depends on whether NIS is used to maintain netgroups, as well as common password and group databases. If this is not the case, then you should log in to each slave node as root and create a file called /etc/hosts.equiv containing a line:
<master>where <master> is the hostname of the master
node. See your system documentation for more information about the use of rsh
and the syntax of the hosts.equiv file.
If NIS is used to control remote access, then a netgroup must be created for users of ANSYS CFX by the root user on the NIS master server, and a line such as the one below added to /etc/hosts.equiv on each slave node by the root user:
+@<netgroup>where <netgroup> is the name of a netgroup to
which users of ANSYS CFX belong. A detailed description of how to configure NIS
is beyond the scope of this manual. See your system documentation for more
information about NIS.
Individual users can create a file called .rhosts in their home directory on each slave containing a line:
<master> <user>
where <master> is the hostname of the master and
<user> is their username on the master. This
file should be made readable only by the user, for example, by running:
chmod 600 ~/.rhosts
In order to use the Distributed Parallel mode of ANSYS CFX, the file hostinfo.ccl must exist in the <CFXROOT>/config/ directory of the ANSYS CFX installation on the master node and be made readable by all users of the software. This file is a database containing information about the available hosts and where ANSYS CFX has been installed on each of them. The file is used by the ANSYS CFX-Solver when constructing a parallel run.
This file is written using the CFX Command Language. It defines a set of HOST DEFINITION objects, one for each available node. For example:
SIMULATION CONTROL:
EXECUTION CONTROL:
PARALLEL HOST LIBRARY:
HOST DEFINITION: kangaroo
Installation Root = /ansys_inc/v%v/CFX
END
HOST DEFINITION:wallaby
Installation Root = /usr/local/cfx
Host Architecture String = linux-amd64
Number of Processors = 16
Relative Speed = 1.7
END
HOST DEFINITION: mypc
Remote Host Name = my_pc
END
END
END
END
Note: If you manually create your hostinfo.ccl file, you must
ensure that the SIMULATION CONTROL...END wrapper is
present.
None of the values for each host are mandatory, and the following empty host definition is perfectly valid:
HOST DEFINITION: parakeet END
Host names are limited to the set of valid CCL object names. In particular, they must not contain full stops (.) or underscores (_) and must start with a letter.
If a hostinfo.ccl file does not already exist when ANSYS CFX is installed, one will be created containing the installation host. You can add hosts to the hostinfo.ccl file using the cfx5parhosts utility, or by modifying the file using a text editor. Individual users may also create their own versions of this file in:
~/.hostinfo.ccl
which will be used in preference if it exists.
For most installations, it will be necessary to supply only the Installation Root parameter, which should point to the <CFXROOT> directory in which ANSYS CFX is installed. On mixed architectures, you may find it useful to supply the Number of Processors and/or Relative Speed parameters. A little time may be saved at startup by giving the Host Architecture String parameter explicitly, for example, using the cfx5parhosts utility.
Tip: If an individual user wants to use a host that is not present in either the hostinfo.ccl in the installation config directory or the user's own version of this file, then the user can add this host to the list for a particular run by using the CFX-Solver Manager. However, this would have to be done each time a run is started.
The available parameters for the hostinfo.ccl file are as follows:
This is set to the <CFXROOT> installation directory on this host.
If it is set to the special string none, this
indicates that there is no ANSYS CFX installation on the remote host,
which can sometimes be useful if only the solver binary is available.
ANSYS CFX will use this value to select the most appropriate solver
executable for this node. These strings can be obtained by giving the
command <CFXROOT>/bin/cfx5info -os on the node
in question. When these short os values (for
example, linux-amd64) are given in this position, the generic solver
executable will always be used for this host. The string can also be set
to the actual architecture of the remote host (for example,
intel_xeon64.sse2_linux2.3.4), which is determined by giving the command
<CFXROOT>/bin/cfx5info -arch. If these
longer strings are used then CFX could use this information to select
between solvers optimized for specific architectures. However, since
there are currently no solvers optimized for specific architectures,
this extra information is currently unused.
As implied, this is the number of processors on the machine. It is used for display purposes only and can be safely omitted.
The Relative Speed is a real number that is used by the ANSYS CFX-Solver when partition sizes are calculated. The ratio of relative speeds of each host is used to determine the size of each partition. As an example, consider a parallel run involving two machines, one with a relative speed of 2.0 and the other with a relative speed of 1.0. The faster machine would work on a partition size twice as large as the slower machine in this case.
The numbers themselves are arbitrary; as a guide you may want to set 1.0 for a 1 GHz processor, 0.75 for a 750 MHz processor, and so on. If a relative speed entry is not entered, a default of 1.0 is assumed for that host.
You can obtain relative speed values using the cfx5parhosts utility.
To include hosts in the parallel run with names that contain, for example, underscore characters, you can add the “Remote Host Name” parameter to the HOST DEFINITION with the correct network name for the host and use a readable alias as the name of the object.
A solver executable may be explicitly supplied, if necessary. This is usually only required when using Installation Root = none, and is recommended for advanced users only. The following substitutions are made on the string:
| %r | root directory of the installation |
| %p | parallel suffix for the executable |
| %v | version of ANSYS CFX being run |
| %o | operating system string |
| %a | architecture subdirectory specification; for example, linux/double |
If it is not supplied, this parameter has the default value %r/bin/%a/solver%p.exe
You can add new hosts to ANSYS CFX's database for parallel runs using the cfx5parhosts utility:
CFXROOT\bin\cfx5parhosts argument listwhere argument list corresponds to one or more of
the arguments listed below:
| Argument | Description |
|---|---|
-add
host[,host,...] |
Add information about the named hosts to the file. This assumes that ANSYS CFX is installed in the same directory on each listed host as on the host on which you are running.
|
| -benchmark |
Runs a standard benchmark case, and fills in the Relative Speed for the local host. The benchmark case will usually take less than 5 minutes to run. |
-file file |
Use the specified file as the hostinfo.ccl file. |
-merge file | Merge host information from the named file. |
| -no-update | After modifying the file, write back the information available without attempting to fill in any missing pieces. |
| -strict | Used with -update. Normally, hosts which
exist on the network but cannot be connected to with rsh or ssh
are included in the resulting file with a comment. This switch
will exclude these hosts. |
| -system | Use the system host file. This is the default. |
| -update | Updates the specified host information file. If any hosts do
not have an architecture specified in the existing
hostinfo.ccl file (for example, because
it was added via the -add switch), it will
connect to the host and query it to fill in the Host
Architecture String parameter. This is the default behavior.
Note that if the Installation Root parameter is incorrect for
the host, it will use the standard system commands to guess a
generic architecture string. This can happen if you use
-add to include a host with a different
installation directory than the local one. |
| -user | Use the per-user host file. |
Most Linux systems support two parallel run modes: Intel MPI and IBM MPI. Intel MPI is the preferred parallel run mode on most of the supported ANSYS CFX platforms.
Intel MPI supports several network fabrics including:
Shared memory, typically used for intra-host communication.
DAPL-capable network fabrics, such as Infiniband, iWarp, Dolphin and XPMEM (through DAPL).
TCP/IP-capable network fabrics, such as Ethernet and InfiniBand (through IPoIB).
TMI-capable network fabrics including Qlogic and Myrianet (through Tag Matching Interface).
OFA-capable network fabrics including InfiniBand (through OFED verbs).
Intel MPI is automatically installed with ANSYS CFX in the<CFXROOT>/../commonfiles/ directory. You do not need a license from Intel to run the ANSYS CFX-Solver using Intel MPI.
Intel MPI run modes are selected in the ANSYS CFX-Solver Manager or on the command line (using the -start-method command line option) using the “Intel MPI Local Parallel” or “Intel MPI Distributed Parallel” start method options.
In most cases, the default settings are sufficient and it may not be necessary to set any environment variables.
ANSYS CFX uses the environment variable CFX5_IMPI_DIR to select which Intel MPI installation is used. The default setting points to the version that is automatically installed by ANSYS CFX in the<CFXROOT>/../commonfiles/MPI/Intel directory. You can use cfx5info -v to find the current settings for CFX5_IMPI_DIR and other environment variables. If you want to use a different Intel MPI version, you can install that version in an alternative location and set CFX5_IMPI_DIR to that location.
There are several environment variables that can be used to control Intel MPI that may provide additional flexibility that is not directly accessible through the ANSYS CFX start-up scripts. For example, environment variables are available for network fabric selection (discussed briefly below), process pinning control, and collective operation control.
Any of these environment variables can be set by modifying the start command in the etc/start-methods.ccl file and using the Intel MPI options -env and/or -genv (an example of this usage is given in the section on network fabric selection). More detail on all options and environment variables is available in the Intel MPI documentation.
Some useful environment variables include those described below. If these environment variables are set the start-up scripts automatically set them for your ANSYS CFX runs.
Specifies which method is used by Intel MPI to spawn jobs on remote machines. The Intel MPI default is ssh. However, by default the ANSYS CFX software sets this to be the same as the setting used for CFX5RSH, and rsh if CFX5RSH is not set. This variable is applicable only to Linux.
Specifies the networks fabrics to be used and the order in which Intel MPI searches for them.
Intel MPI start methods use the default network fabrics selection built into Intel MPI. The default order in which Intel MPI checks for fabrics is DAPL, OFA, TCP, TMI. The environment variable I_MPI_FABRICS_LIST controls the search order used by Intel MPI.
If you want to force selection of a specific network fabric, then you can either set the I_MPI_FABRICS environment variable or modify the start command in the etc/start-methods.ccl file and add the necessary command line flags (for example: -env IMPI_FABRICS shm:dapl to select the combination of shared memory and DAPL fabrics) .
The following table shows how Intel MPI searches for drivers for the various network fabrics:
| Interconnect | 1st attempt | 2nd attempt |
|---|---|---|
| shm | N/A | N/A |
| DAPL | Environment variable I_MPI_DAT_LIBRARY | libdat.so or libdat.so.1 for DAPL 1.2 providers and libdat2.so or libdat2.so.1 for DAPL 1.2 providers |
| TCP | N/A | N/A |
| TMI | Environment variable I_MPI_TMI_LIBRARY | libtmi.so |
| OFA | Environment variable I_MPI_OFA_LIBRARY | libibverbs.so |
For example, Intel MPI expects that the DAT shared library for the DAPL network fabric is located either in the default library search path or in a specific location if a full path is specified. If the shared library cannot be found, Intel MPI will not take advantage of the DAPL network fabric. To fix this, you will have to install the shared libraries into the correct location or set the environment variable I_MPI_DAT_LIBRARY to point to the correct DAT shared library.
There are other environment variables not documented here which provide further control over network fabrics selection. These are described in the Intel MPI documentation.
IBM MPI is available as an alternative parallel run mode on all the supported ANSYS CFX platforms.
IBM MPI supports several communication modes including shared memory and networked TCP/IP (Ethernet), luDAPL, VAPI, OpenFabrics (IB), GM-2, MX (Myrinet), Elan4 (Quadrics), QLogic InfiniPath, and EtherFabric (Level 5 Networks).
IBM MPI is automatically installed with ANSYS CFX in the <CFXROOT>/tools directory. You do not need a license from IBM to run the ANSYS CFX-Solver using IBM MPI.
IBM MPI run modes are selected in the ANSYS CFX-Solver Manager or on the command line
(using the -start-method command line option) using the
“IBM MPI Local Parallel” or “IBM MPI Distributed
Parallel” start method options.
For IBM MPI, the –cpu_bind option is used by
default and binds processes to cores using a cyclic order. This leads to better
performance but, on multi-user clusters having scheduling systems that do not
enforce exclusive use of cluster nodes, may cause problems because multiple jobs may
be bound to the same CPU on the same node. In this case, the
–cpu_bind argument to mpirun
should be removed from the IBM MPI start methods in the
<CFXROOT>/etc/start_methods.ccl file.
In most cases, the default settings are sufficient and it may not be necessary to set any environment variables.
ANSYS CFX uses the environment variable CFX5_IBMMPI_DIR to select which IBM MPI installation is used. The default setting points to the version that is automatically installed with ANSYS CFX in the <CFXROOT>/../commonfiles/MPI/IBM directory. You can use cfx5info -v to find the current settings for CFX5_IBMMPI_DIR and other environment variables. If you want to use a different IBM MPI version, you can install that version in an alternative location and set CFX5_IBMMPI_DIR to that location. For distributed parallel runs, IBM MPI must be present at the same location (same absolute path) on all hosts. Typically, this is achieved by using a central ANSYS CFX installation, accessible by all hosts. In order to run a CFX job in distributed parallel using IBM MPI on hosts with different CFX installation locations, you must first install IBM MPI under the same location (same absolute path) on each host, then set the environment variable CFX5_IBMMPI_DIR to the location under which IBM MPI was installed.
There are several environment variables that can be used to control IBM MPI (documented in the IBM MPI documentation) that may provide additional flexibility not directly accessible through the ANSYS CFX startup scripts. If these environment variables are set the startup scripts automatically set them for your ANSYS CFX runs. Some useful environment variables include:
Enables you to change which method is used by IBM MPI to spawn jobs on remote machines. The IBM MPI default is ssh. However, by default the ANSYS CFX software sets this to be the same as the setting used for CFX5RSH, and rsh if CFX5RSH is not set. This variable is applicable only to Linux and HP-UX.
Enables you to change the default order in which IBM MPI searches for interconnects.
IBM MPI start methods use the default communication mode selection built into IBM MPI. The default order in which IBM MPI checks for interconnects is OpenFabric, VAPI, UDAPL, ITAPI, PSM, Myrinet MX, Myrinet GM, Elan, TCP/IP.
If you want to force selection of a specific device, then you can either modify the start command in the etc/start-methods.ccl file and add the necessary command line flags (for example: -GM to force selection of the Myrinet interconnect) or use the appropriate IBM MPI environment variables.
Some useful command line options include forcing other interconnect types (-TCP, -HMP, -ELAN, -ITAPI, -VAPI, -UDAPL, and so on), using prun on systems with quadrics support (-prun), XC support (-srun) and setting the subnet you want to use (-subnet, sometimes useful for selecting particular interconnects as well if your system is configured that way).
For some Linux-based cluster environments not supported by IBM MPI, it is possible that the Myrinet, Infiniband or Quadrics hardware drivers are not located where IBM MPI expects. If IBM MPI cannot detect the drivers for fast interconnects, it will generally default back to the lowest speed connection that it can detect (usually TCP/IP socket communication).
The following table gives information on the default search path IBM MPI uses to find drivers for the various communication interfaces:
| Interconnect | 1st attempt | 2nd attempt | 3rd attempt |
|---|---|---|---|
| Infiniband (IB) | Environment variable MPI_ICLIB_ITAPI | libitapi.so | /usr/voltaire/lib/libitapi.so |
| Myrinet (GM) | Environment variable MPI_ICLIB_GM | libgm.so OR libgm32.so | /opt/gm/lib/libgm.so OR /opt/gm/lib/libgm32.so |
| ELAN | Environment variable MPI_ICLIB_ELAN | libelan.so | none |
| UDAPL | Environment variable MPI_ICLIB_UDAPL | libdat.so | none |
| VAPI | Environment variable MPI_ICLIB_VAPI | Environment variable MPI_ICLIB_VAPIDIR | libmtl_common.so, libmpga.so, libmosal.so, libvapi.so |
| Infinipath | Environment variable MPI_ICLIB_PSM | libpsm_infinipath.so.1 | /usr/lib64/ libpsm_infinipath.so.1 |
For example, IBM MPI expects that GM stack shared libraries for Myrinet interconnects are located either in the default library search path or in the /opt/gm/lib64 (64 bit x86) directories. If the shared libraries are not in either location, IBM MPI will not take advantage of the high-speed interconnects. To fix this, you will have to copy the shared libraries into the correct location or set the environment variable MPI_ICLIB_GM to point to the correct GM stack shared library.
The ANSYS CFX-Solver supports Cray’s native implementation of MPI on the Cray Linux Environment (CLE) using the start method ‘Cray MPI Distributed Parallel’.
Cray MPI is closely integrated with the batch queuing system and interconnects used on Cray systems.
Cray MPI is the recommended method of running on Cray systems, as opposed to using Cluster Compatibility Mode (CCM) with a different MPI.
When starting a solver run, the Cray MPI start method may be specified on the
command line using the option: -start-method “Cray MPI
Distributed Parallel”, having reserved the required
processing elements via the built-in batch queuing
system.
The aprun command, which can be modified, is used by
the start method to launch the job. Other options available from Cray are
described in the documentation for aprun.
Note that numa node memory containment is enforced, using the
-ss option. The default method used by Cray MPI to
allocate processes to CPU cores is appropriate when all cores are used. However,
if it is intended that each host is only partly utilized (for example, because
of the memory requirements of the run), this can result in asymmetric
utilization. In such cases, the -S option can be used to
limit the number of processes per numa node. For example, -S
4 would limit the number of processes per numa node to 4.