ANSYS CFX-Solver must be installed on all hosts that are to be used for a parallel run.
Local parallel runs use Intel MPI (default) or IBM MPI for communication.
To run distributed parallel (where slave processes run on hosts other than the host with the master process), the desired MPI service must be installed, configured, and started on all involved hosts. In addition, you must make host information available to the master host. These tasks are discussed in Setting Up IBM MPI for Windows or Setting Up Intel MPI for Windows.
When the ANSYS CFX distributed parallel starts, it needs to know where to find the CFX-Solver executable on every host. You must make host information available to the master host by one of these mechanisms:
(Preferred mechanism) The hostinfo.ccl file on the master host is configured with any required information about slave hosts.
The hostinfo.ccl file registers information such as
the Installation Root and Host
Architecture String parameters for hosts used for
distributed parallel runs. The hostinfo.ccl file is
created automatically when installing ANSYS CFX. By default, it contains
details of the host on which it is installed.
For hosts that have the same architecture as the machine that runs the master process, registration in the hostinfo.ccl file is optional; if a particular host is not specified then all of its properties are assumed to be the same as those of the machine that runs the master process.
For machines that have a machine architecture different from the machine that runs the master process, the hostinfo.ccl file must include information about the properties that differ from those of the machine that runs the master process.
See hostinfo.ccl file for more information.
If your machine previously had ANSYS CFX installed, then ensure that remote
host interrogation is disabled either by deleting the environment variable
CFX_SOLVE_DISABLE_REMOTE_CHECKS or (equivalently)
by setting that environment variable to a value of 1.
This environment variable can be specified in your Windows environment or
your user cfx5rc.txt file or site
cfx5rc.txt file. See Resources Set in cfx5rc Files in the CFX Introduction for details.
Have ANSYS CFX determine the required information for itself.
If ANSYS CFX is to determine the required information for itself, then remote- or secure-shell access must be available from the master host (the system on which parallel runs will be started) to slave hosts (systems on which the ANSYS CFX-Solver will actually run). You must be able to execute an rsh or ssh command on the remote host without typing in your password. Unless you already have your Windows systems set up to allow this type of access, this method of setting up distributed parallel is not recommended.
You will also have to set the environment variable
CFX_SOLVE_DISABLE_REMOTE_CHECKS to a value of
0; this enables remote host interrogation. This
environment variable can be specified in your Windows environment or your
user cfx5rc.txt file or site
cfx5rc.txt file. See Resources Set in cfx5rc Files in the CFX Introduction for details.
In order to use the Distributed Parallel modes effectively, the file hostinfo.ccl must exist in the <CFXROOT>/config/ directory of the ANSYS CFX installation on the master host and be made readable by all users of the software. This file is a database containing information about the available hosts and where ANSYS CFX has been installed on each of them. The file is used by the ANSYS CFX-Solver when constructing a parallel run.
The hostinfo.ccl file is written using the CFX Command Language. It defines a set of HOST DEFINITION objects, one for each available host. For example:
SIMULATION CONTROL:
EXECUTION CONTROL:
PARALLEL HOST LIBRARY:
HOST DEFINITION: hostname1
Installation Root = C:\Program Files\ANSYS Inc\v%v\CFX
Host Architecture String = winnt-amd64
END # HOST DEFINITION hostname1
END # PARALLEL HOST LIBRARY
END # EXECUTION CONTROL
END # SIMULATION CONTROLNote: If you manually create your hostinfo.ccl file, you
must ensure that the SIMULATION CONTROL...END wrapper
is present.
None of the values for each host are mandatory. For example, the following empty host definition is perfectly valid:
HOST DEFINITION: hostname1 END
Host names are limited to the set of valid CCL object names. In particular, they must not contain full stops (.) or underscores (_) and must start with a letter.
If a hostinfo.ccl file does not already exist when ANSYS CFX is installed, one will be created containing the installation host. You can add hosts to the hostinfo.ccl file using the cfx5parhosts utility, or by modifying the file using a text editor.
For most installations, it will be necessary only to supply the Installation Root parameter, which should point to the <CFXROOT> directory in which ANSYS CFX is installed. On mixed networks, you may find it useful to supply the Number of Processors and/or Relative Speed parameters. A little time may be saved at startup by giving the Host Architecture String parameter explicitly, for example, using the cfx5parhosts utility.
The available parameters are as follows:
This is set to the <CFXROOT> installation directory on this host. If it is set to the special string none, this indicates that there is no ANSYS CFX installation on the remote host, which can sometimes be useful if only the solver binary is available.
This should be set to the actual architecture <arch> of the
remote host. ANSYS CFX will use this value to select the most
appropriate solver executable for this host. These strings can be
obtained by giving the command <CFXROOT>/bin/cfx5info
-os on the host in question. If the shorter <os>
values are given in this position, the generic solver executable
will always be used for this host.
As implied, this is the number of processors on the machine. It is used for display purposes only and can be safely omitted.
The Relative Speed is a real number that is used by the ANSYS CFX-Solver when partition sizes are calculated. The ratio of relative speeds of each host is used to determine the size of each partition. As an example, consider a parallel run involving two machines, one with a relative speed of 2.0 and the other with a relative speed of 1.0. The faster machine would work on a partition size twice as large as the slower machine in this case.
The numbers themselves are arbitrary; as a guide you may want to set 1.0 for a 1 GHz processor, 0.75 for a 750 MHz processor, and so on. If a relative speed entry is not entered, a default of 1.0 is assumed for that host.
You can obtain relative speed values using the cfx5parhosts utility.
To include hosts in the parallel run with names that contain, for example, underscore characters, you can add the “Remote Host Name” parameter to the HOST DEFINITION with the correct network name for the host and use a readable alias as the name of the object.
A solver executable may be explicitly supplied, if necessary. This is usually only required when using Installation Root = none, and is recommended for advanced users only. The following substitutions are made on the string:
| %r | root directory of the installation |
| %p | parallel suffix for the executable |
| %v | version of ANSYS CFX being run |
| %o | operating system string |
| %a | architecture subdirectory specification, for example, linux/double |
If it is not supplied, this parameter has the default value %r/bin/%a/solver%p.exe
You can add new hosts to ANSYS CFX's database for parallel runs using the cfx5parhosts utility; this is done by running:
CFXROOT\bin\cfx5parhostsargument list
where argument list corresponds to one or more
of the arguments listed below.
| Argument | Description |
|---|---|
-add
host[,host,...] |
Add information about the named host(s) to the file. This assumes that ANSYS CFX is installed in the same directory on each listed host as on the host on which you are running.
To add a set of separately-installed Windows hosts to
the hostinfo.ccl file, where the
installation may be in a different place on each host,
the recommended method is to gather the
hostinfo.ccl files created on
each host by the installation process, and merge them
together using the |
| -benchmark | Runs a standard benchmark case, and fills in the Relative Speed for the local host. The benchmark case will usually take less than 5 minutes to run. |
-file file | Use the specified file as the hostinfo.ccl file. |
-merge file | Merge host information from the named file. |
| -no-update | After modifying the file, write back the information available without attempting to fill in any missing pieces. |
| -strict | Used with -update. Normally, hosts that
exist on the network but cannot be connected to with rsh or
ssh are included in the resulting file with a comment. This
switch will exclude these hosts. |
| -system | Use the system host file. This is the default. |
| -update | Updates the specified host information file. If any hosts
do not have an architecture specified in the existing
hostinfo.ccl file (for example,
because it was added via the -add switch),
it will connect to the host and query it to fill in the Host
Architecture String parameter. This is the default behavior.
Note that if the Installation Root parameter is incorrect
for the host, it will use the standard system commands to
guess a generic architecture string. This can happen if you
use -add to include a host with a different
installation directory than the local one. |
| -user | Use the per-user host file. |
To install, configure, and start the Intel MPI service on a given host, install Intel MPI using the installation launcher. After installing Intel MPI in this way, the software is configured and started. For more information, see Product Installation with Client Licensing.
To enable Intel MPI parallel processing through a Windows firewall:
Click and select .
On the Control Panel dialog box, double-click .
Click Windows Firewall.
On the Windows Firewall dialog box, click the Exceptions tab and then click .
Browse to the following programs:
For Intel MPI:
<CFXROOT>\bin\<CFX5OS>\solver-mpi.exe
<CFXROOT>\bin\<CFX5OS>\double\solver-mpi.exe
<CFXROOT>\..\commonfiles\MPI\Intel\<version number>\winx64\bin\smpd.exe
<CFXROOT>\..\commonfiles\MPI\Intel\<version number>\winx64\bin\mpiexec.exe
After each selection, click to add that program to the list of programs that are permitted to pass through the Windows firewall.
When you are finished, click to close the Windows Firewall dialog box, then close the and the .
To install, configure, and start the IBM MPI service on a given host, install IBM MPI using the installation launcher. After installing IBM MPI in this way, the software is configured and started. For more information, see Product Installation with Client Licensing.
IBM MPI service requires that all hosts are on the same domain and that the user has an account for that domain.
IBM MPI service requires that all machines in the cluster have identical MPI installation directory paths.
IBM MPI uses the –cpu_bind option
by default and binds processes to cores using a cyclic order. This leads
to better performance but, on multi-user clusters having scheduling
systems that do not enforce exclusive use of cluster nodes, this may
cause problems because multiple jobs may be bound to the same CPU on the
same node. In this case, the –cpu_bind
argument to mpirun should be removed from the IBM
MPI start methods in the start_methods.ccl
file in the <CFXROOT>/etc directory.
To enable IBM MPI parallel processing through a Windows firewall:
Click and select .
On the Control Panel dialog box, double-click .
Click Windows Firewall.
On the Windows Firewall dialog box, click the Exceptions tab and then click .
Browse to the following programs:
For IBM MPI:
<ANSYS Inc installation path>\v160\CFX\bin\<CFX5OS>\solver-mpi.exe
<ANSYS Inc installation path>\v160\CFX\bin\<CFX5OS>\double\solver-mpi.exe
<ANSYS Inc installation path>\v160\commonfiles\MPI\IBM\<version number>\winx64\sbin\ibmmpiwin32service.exe
<ANSYS Inc installation path>\v160\commonfiles\MPI\IBM\<version number>\winx64\bin\mpirun.exe
<ANSYS Inc installation path>\v160\commonfiles\MPI\IBM\<version number>\winx64\bin\mpid.exe
The IBM MPI service listens to port 8636 by default. This can be changed during installation of IBM MPI from the product installer or using the following command:
cfx5parallel -setportkey-ibmmpi-serviceIn all cases, you must ensure the chosen port is accessible.
After each selection, click to add that program to the list of programs that are permitted to pass through the Windows firewall.
When you are finished, click to close the Windows Firewall dialog box, then close the and the .
To set up running Microsoft Windows HPC 2008 or Microsoft Windows HPC 2012, steps must be taken so that:
The installed software is on a shared location that can be accessed by all hosts
The working directory is a shared location
A mechanism is provided so that all local paths can be properly resolved.
To prepare for running ANSYS CFX with Microsoft Windows HPC 2008 or Microsoft Windows HPC 2012, you can follow the steps below (terms in angle brackets < > should be replaced with the required entry):
Install ANSYS Licensing, ANSYS Workbench and ANSYS CFX on the head node.
Share the installation directory. For example, on a typical installation
share C:\Program Files\ANSYS Inc as
\\ where
<HeadNodeName>\ANSYS
Inc is
the name of the head node.<HeadNodeName>
Install the ANSYS prerequisites on all of the cluster nodes. You can do this either by:
Executing the following commands directly on each node:
\\<HeadNodeName>\ANSYS Inc\v182\prereq\2008MFCvcredist_x64.exe /q \\<HeadNodeName>\ANSYS Inc\v182\prereq\2008vcredist_x64.exe /q \\<HeadNodeName>\ANSYS Inc\v182\prereq\2008vcredist_x86.exe /q \\<HeadNodeName>\ANSYS Inc\v182\prereq\2010vcredist_x64.exe /q \\<HeadNodeName>\ANSYS Inc\v182\prereq\2010vcredist_x86.exe /q \\<HeadNodeName>\ANSYS Inc\v182\prereq\2012vcredist_x64.exe /quiet \\<HeadNodeName>\ANSYS Inc\v182\prereq\2012vcredist_x86.exe /quiet \\<HeadNodeName>\ANSYS Inc\v182\prereq\dotNetFx40.exe /q \\<HeadNodeName>\ANSYS Inc\v182\prereq\vcredist_x64_SP1.exe /Q \\<HeadNodeName>\ANSYS Inc\v182\prereq\vcredist_x86_SP1.exe /Q
or by using the clusrun command on the
headnode to execute these commands on all the nodes (refer to your
Microsoft Windows HPC documentation for details).
Share the working directory on the head node. For example, share
C:\Users\<UserName>
as
\\
where <HeadNodeName>\<UserName> is the
user name.<UserName>
Alternatively, share the working directory of the submitting machine. For
example share
C:\Users\
on the submitting machine as
<UserName>\\
where <SubmitHostName>\<UserName> is
the name of the submitting machine.<SubmitHostName>
If the submitting machine is different from the server head node, it is
necessary to install ‘HPC Pack 2008 R2 Client Utilities’ on the
submitting machine. This is available for download from Microsoft. This sets
the CCP_HOME environment variable, enabling the
‘Submit to Windows HPC Queue’ start method in the
CFX-Solver Manager.
On the submitting machine, create
%USERPROFILE%\.cfx\cfxccs_options.txt with the
following content to define the required job submission options:
PATHMAP=C:\Program Files\ANSYS
Inc;\\ |
PATHMAP=
C:\Users\
or
PATHMAP=C:\Users\
if the working directory has been shared from the submitting
machine. |
CLUSTERHOST=
to be used when submitting jobs from machines other than the
headnode. |
ACCOUNT=
to be used when submitting jobs using different credentials, where
and
are the domain and user names of another user,
respectively. |
PROCESSORSPERSOLVER=2 an optional setting
(default setting is 1) that allocates the number of cores per
partition. This is typically used on hosts that are limited by
memory bandwidth such as Xeon-based machines. |
Set up ANSYS Workbench for a network as described in Network Installation and Product Configuration.
To submit a job:
Start the CFX-Solver Manager (either using CFX standalone or from ANSYS Workbench) on the submitting machine using the software installed on the headnode.
Ensure that the Run Mode on the Run
Definition tab of the CFX-Solver Manager is set to
Submit to Windows HPC Queue. Otherwise, set up the
CFX-Solver Manager as normal.
Click on the CFX-Solver Manager to start the simulation. The first time a simulation is run, supply the required credentials that are prompted (this prompt may appear behind the CFX-Solver Manager window). You may elect to have these credentials saved to avoid repeated prompting of these credentials.
%USERPROFILE% is typically
C:\Users\<UserName>
on Microsoft Windows HPC Server 2008.
To use on ANSYS Workbench with Parameters and Design Points, start the
CFX-Solver Manager ensure that the Run Mode on the
Run Definition tab of the CFX-Solver Manager is
set to Submit to Windows HPC Queue, set the number
of processes, then click Save Settings. When the
user clicks Update all Design Points each parameter
or design point will be solved on the cluster.