2. ANSYS CFX Windows Parallel Setup

ANSYS CFX-Solver must be installed on all hosts that are to be used for a parallel run.

Local parallel runs use Intel MPI (default) or IBM MPI for communication.

To run distributed parallel (where slave processes run on hosts other than the host with the master process), the desired MPI service must be installed, configured, and started on all involved hosts. In addition, you must make host information available to the master host. These tasks are discussed in Setting Up IBM MPI for Windows or Setting Up Intel MPI for Windows.

2.1. Parallel Setup for Windows

When the ANSYS CFX distributed parallel starts, it needs to know where to find the CFX-Solver executable on every host. You must make host information available to the master host by one of these mechanisms:

  • (Preferred mechanism) The hostinfo.ccl file on the master host is configured with any required information about slave hosts.

    The hostinfo.ccl file registers information such as the Installation Root and Host Architecture String parameters for hosts used for distributed parallel runs. The hostinfo.ccl file is created automatically when installing ANSYS CFX. By default, it contains details of the host on which it is installed.

    For hosts that have the same architecture as the machine that runs the master process, registration in the hostinfo.ccl file is optional; if a particular host is not specified then all of its properties are assumed to be the same as those of the machine that runs the master process.

    For machines that have a machine architecture different from the machine that runs the master process, the hostinfo.ccl file must include information about the properties that differ from those of the machine that runs the master process.

    See hostinfo.ccl file for more information.

    If your machine previously had ANSYS CFX installed, then ensure that remote host interrogation is disabled either by deleting the environment variable CFX_SOLVE_DISABLE_REMOTE_CHECKS or (equivalently) by setting that environment variable to a value of 1. This environment variable can be specified in your Windows environment or your user cfx5rc.txt file or site cfx5rc.txt file. See Resources Set in cfx5rc Files in the CFX Introduction for details.

  • Have ANSYS CFX determine the required information for itself.

    If ANSYS CFX is to determine the required information for itself, then remote- or secure-shell access must be available from the master host (the system on which parallel runs will be started) to slave hosts (systems on which the ANSYS CFX-Solver will actually run). You must be able to execute an rsh or ssh command on the remote host without typing in your password. Unless you already have your Windows systems set up to allow this type of access, this method of setting up distributed parallel is not recommended.

    You will also have to set the environment variable CFX_SOLVE_DISABLE_REMOTE_CHECKS to a value of 0; this enables remote host interrogation. This environment variable can be specified in your Windows environment or your user cfx5rc.txt file or site cfx5rc.txt file. See Resources Set in cfx5rc Files in the CFX Introduction for details.

2.1.1. hostinfo.ccl file

In order to use the Distributed Parallel modes effectively, the file hostinfo.ccl must exist in the <CFXROOT>/config/ directory of the ANSYS CFX installation on the master host and be made readable by all users of the software. This file is a database containing information about the available hosts and where ANSYS CFX has been installed on each of them. The file is used by the ANSYS CFX-Solver when constructing a parallel run.

The hostinfo.ccl file is written using the CFX Command Language. It defines a set of HOST DEFINITION objects, one for each available host. For example:

SIMULATION CONTROL:
  EXECUTION CONTROL:
    PARALLEL HOST LIBRARY:
      HOST DEFINITION: hostname1
        Installation Root = C:\Program Files\ANSYS Inc\v%v\CFX
        Host Architecture String = winnt-amd64
      END # HOST DEFINITION hostname1
    END # PARALLEL HOST LIBRARY
  END # EXECUTION CONTROL
END # SIMULATION CONTROL

Note:  If you manually create your hostinfo.ccl file, you must ensure that the SIMULATION CONTROL...END wrapper is present.


None of the values for each host are mandatory. For example, the following empty host definition is perfectly valid:

HOST DEFINITION: hostname1
END

Host names are limited to the set of valid CCL object names. In particular, they must not contain full stops (.) or underscores (_) and must start with a letter.

If a hostinfo.ccl file does not already exist when ANSYS CFX is installed, one will be created containing the installation host. You can add hosts to the hostinfo.ccl file using the cfx5parhosts utility, or by modifying the file using a text editor.

For most installations, it will be necessary only to supply the Installation Root parameter, which should point to the <CFXROOT> directory in which ANSYS CFX is installed. On mixed networks, you may find it useful to supply the Number of Processors and/or Relative Speed parameters. A little time may be saved at startup by giving the Host Architecture String parameter explicitly, for example, using the cfx5parhosts utility.

The available parameters are as follows:

Installation Root

This is set to the <CFXROOT> installation directory on this host. If it is set to the special string none, this indicates that there is no ANSYS CFX installation on the remote host, which can sometimes be useful if only the solver binary is available.

Host Architecture String

This should be set to the actual architecture <arch> of the remote host. ANSYS CFX will use this value to select the most appropriate solver executable for this host. These strings can be obtained by giving the command <CFXROOT>/bin/cfx5info -os on the host in question. If the shorter <os> values are given in this position, the generic solver executable will always be used for this host.

Number of Processors

As implied, this is the number of processors on the machine. It is used for display purposes only and can be safely omitted.

Relative Speed

The Relative Speed is a real number that is used by the ANSYS CFX-Solver when partition sizes are calculated. The ratio of relative speeds of each host is used to determine the size of each partition. As an example, consider a parallel run involving two machines, one with a relative speed of 2.0 and the other with a relative speed of 1.0. The faster machine would work on a partition size twice as large as the slower machine in this case.

The numbers themselves are arbitrary; as a guide you may want to set 1.0 for a 1 GHz processor, 0.75 for a 750 MHz processor, and so on. If a relative speed entry is not entered, a default of 1.0 is assumed for that host.

You can obtain relative speed values using the cfx5parhosts utility.

Remote Host Name

To include hosts in the parallel run with names that contain, for example, underscore characters, you can add the “Remote Host Name” parameter to the HOST DEFINITION with the correct network name for the host and use a readable alias as the name of the object.

Solver Executable

A solver executable may be explicitly supplied, if necessary. This is usually only required when using Installation Root = none, and is recommended for advanced users only. The following substitutions are made on the string:

%rroot directory of the installation
%pparallel suffix for the executable
%vversion of ANSYS CFX being run
%ooperating system string
%aarchitecture subdirectory specification, for example, linux/double

If it is not supplied, this parameter has the default value %r/bin/%a/solver%p.exe

2.1.1.1. Adding Hosts for Parallel Processing with the cfx5parhosts Utility

You can add new hosts to ANSYS CFX's database for parallel runs using the cfx5parhosts utility; this is done by running:

CFXROOT\bin\cfx5parhosts argument list

where argument list corresponds to one or more of the arguments listed below.

ArgumentDescription
-add host[,host,...]

Add information about the named host(s) to the file. This assumes that ANSYS CFX is installed in the same directory on each listed host as on the host on which you are running.

host may be specified as [user@]hostname[:cfx-5 root] if the user name or the ANSYS CFX installation root directory differs from the local host.

To add a set of separately-installed Windows hosts to the hostinfo.ccl file, where the installation may be in a different place on each host, the recommended method is to gather the hostinfo.ccl files created on each host by the installation process, and merge them together using the -merge switch.

-benchmark Runs a standard benchmark case, and fills in the Relative Speed for the local host. The benchmark case will usually take less than 5 minutes to run.
-file fileUse the specified file as the hostinfo.ccl file.
-merge fileMerge host information from the named file.
-no-update After modifying the file, write back the information available without attempting to fill in any missing pieces.
-strict Used with -update. Normally, hosts that exist on the network but cannot be connected to with rsh or ssh are included in the resulting file with a comment. This switch will exclude these hosts.
-system Use the system host file. This is the default.
-update Updates the specified host information file. If any hosts do not have an architecture specified in the existing hostinfo.ccl file (for example, because it was added via the -add switch), it will connect to the host and query it to fill in the Host Architecture String parameter. This is the default behavior. Note that if the Installation Root parameter is incorrect for the host, it will use the standard system commands to guess a generic architecture string. This can happen if you use -add to include a host with a different installation directory than the local one.
-user Use the per-user host file.

2.2. Setting Up Intel MPI for Windows

To install, configure, and start the Intel MPI service on a given host, install Intel MPI using the installation launcher. After installing Intel MPI in this way, the software is configured and started. For more information, see Product Installation with Client Licensing.

2.2.1. Enabling Intel MPI Parallel Through a Windows Firewall

To enable Intel MPI parallel processing through a Windows firewall:

  1. Click Start and select Control Panel.

  2. On the Control Panel dialog box, double-click Security Center.

  3. Click Windows Firewall.

  4. On the Windows Firewall dialog box, click the Exceptions tab and then click Add Program.

  5. Browse to the following programs:

    • For Intel MPI:

      • <CFXROOT>\bin\<CFX5OS>\solver-mpi.exe

      • <CFXROOT>\bin\<CFX5OS>\double\solver-mpi.exe

      • <CFXROOT>\..\commonfiles\MPI\Intel\<version number>\winx64\bin\smpd.exe

      • <CFXROOT>\..\commonfiles\MPI\Intel\<version number>\winx64\bin\mpiexec.exe

    After each selection, click OK to add that program to the list of programs that are permitted to pass through the Windows firewall.

  6. When you are finished, click OK to close the Windows Firewall dialog box, then close the Windows Security Center and the Control Panel.

2.3. Setting Up IBM MPI for Windows

To install, configure, and start the IBM MPI service on a given host, install IBM MPI using the installation launcher. After installing IBM MPI in this way, the software is configured and started. For more information, see Product Installation with Client Licensing.


Note:
  • IBM MPI service requires that all hosts are on the same domain and that the user has an account for that domain.

  • IBM MPI service requires that all machines in the cluster have identical MPI installation directory paths.

  • IBM MPI uses the –cpu_bind option by default and binds processes to cores using a cyclic order. This leads to better performance but, on multi-user clusters having scheduling systems that do not enforce exclusive use of cluster nodes, this may cause problems because multiple jobs may be bound to the same CPU on the same node. In this case, the –cpu_bind argument to mpirun should be removed from the IBM MPI start methods in the start_methods.ccl file in the <CFXROOT>/etc directory.


2.3.1. Enabling IBM MPI Parallel Through a Windows Firewall

To enable IBM MPI parallel processing through a Windows firewall:

  1. Click Start and select Control Panel.

  2. On the Control Panel dialog box, double-click Security Center.

  3. Click Windows Firewall.

  4. On the Windows Firewall dialog box, click the Exceptions tab and then click Add Program.

  5. Browse to the following programs:

    • For IBM MPI:

      • <ANSYS Inc installation path>\v160\CFX\bin\<CFX5OS>\solver-mpi.exe

      • <ANSYS Inc installation path>\v160\CFX\bin\<CFX5OS>\double\solver-mpi.exe

      • <ANSYS Inc installation path>\v160\commonfiles\MPI\IBM\<version number>\winx64\sbin\ibmmpiwin32service.exe

      • <ANSYS Inc installation path>\v160\commonfiles\MPI\IBM\<version number>\winx64\bin\mpirun.exe

      • <ANSYS Inc installation path>\v160\commonfiles\MPI\IBM\<version number>\winx64\bin\mpid.exe

    The IBM MPI service listens to port 8636 by default. This can be changed during installation of IBM MPI from the product installer or using the following command:

    cfx5parallel -setportkey-ibmmpi-service

    In all cases, you must ensure the chosen port is accessible.

    After each selection, click OK to add that program to the list of programs that are permitted to pass through the Windows firewall.

  6. When you are finished, click OK to close the Windows Firewall dialog box, then close the Windows Security Center and the Control Panel.

2.4. Setting up and Running Microsoft Windows HPC 2008 or HPC 2012

To set up running Microsoft Windows HPC 2008 or Microsoft Windows HPC 2012, steps must be taken so that:

  • The installed software is on a shared location that can be accessed by all hosts

  • The working directory is a shared location

  • A mechanism is provided so that all local paths can be properly resolved.

To prepare for running ANSYS CFX with Microsoft Windows HPC 2008 or Microsoft Windows HPC 2012, you can follow the steps below (terms in angle brackets < > should be replaced with the required entry):

  1. Install ANSYS Licensing, ANSYS Workbench and ANSYS CFX on the head node.

  2. Share the installation directory. For example, on a typical installation share C:\Program Files\ANSYS Inc as \\<HeadNodeName>\ANSYS Inc where <HeadNodeName> is the name of the head node.

  3. Install the ANSYS prerequisites on all of the cluster nodes. You can do this either by:

    • Executing the following commands directly on each node:

      \\<HeadNodeName>\ANSYS Inc\v182\prereq\2008MFCvcredist_x64.exe /q
      \\<HeadNodeName>\ANSYS Inc\v182\prereq\2008vcredist_x64.exe /q
      \\<HeadNodeName>\ANSYS Inc\v182\prereq\2008vcredist_x86.exe /q
      \\<HeadNodeName>\ANSYS Inc\v182\prereq\2010vcredist_x64.exe /q
      \\<HeadNodeName>\ANSYS Inc\v182\prereq\2010vcredist_x86.exe /q
      \\<HeadNodeName>\ANSYS Inc\v182\prereq\2012vcredist_x64.exe /quiet
      \\<HeadNodeName>\ANSYS Inc\v182\prereq\2012vcredist_x86.exe /quiet
      \\<HeadNodeName>\ANSYS Inc\v182\prereq\dotNetFx40.exe /q
      \\<HeadNodeName>\ANSYS Inc\v182\prereq\vcredist_x64_SP1.exe /Q
      \\<HeadNodeName>\ANSYS Inc\v182\prereq\vcredist_x86_SP1.exe /Q
    • or by using the clusrun command on the headnode to execute these commands on all the nodes (refer to your Microsoft Windows HPC documentation for details).

  4. Share the working directory on the head node. For example, share C:\Users\<UserName> as \\<HeadNodeName>\<UserName> where <UserName> is the user name.

    Alternatively, share the working directory of the submitting machine. For example share C:\Users\<UserName> on the submitting machine as \\<SubmitHostName>\<UserName> where <SubmitHostName> is the name of the submitting machine.

  5. If the submitting machine is different from the server head node, it is necessary to install ‘HPC Pack 2008 R2 Client Utilities’ on the submitting machine. This is available for download from Microsoft. This sets the CCP_HOME environment variable, enabling the ‘Submit to Windows HPC Queue’ start method in the CFX-Solver Manager.

  6. On the submitting machine, create %USERPROFILE%\.cfx\cfxccs_options.txt with the following content to define the required job submission options:

    PATHMAP=C:\Program Files\ANSYS Inc;\\<HeadNodeName>\ANSYS Inc
    PATHMAP= C:\Users\<UserName>;\\<HeadNodeName>\<UserName> or PATHMAP=C:\Users\<UserName>;\\<SubmitHostName>\<UserName> if the working directory has been shared from the submitting machine.
    CLUSTERHOST=<HeadNodeName> to be used when submitting jobs from machines other than the headnode.
    ACCOUNT=<OtherUserDomain>\<OtherUserName> to be used when submitting jobs using different credentials, where <OtherUserDomain> and <OtherUserName> are the domain and user names of another user, respectively.
    PROCESSORSPERSOLVER=2 an optional setting (default setting is 1) that allocates the number of cores per partition. This is typically used on hosts that are limited by memory bandwidth such as Xeon-based machines.
  7. Set up ANSYS Workbench for a network as described in Network Installation and Product Configuration.

To submit a job:

  1. Start the CFX-Solver Manager (either using CFX standalone or from ANSYS Workbench) on the submitting machine using the software installed on the headnode.

  2. Ensure that the Run Mode on the Run Definition tab of the CFX-Solver Manager is set to Submit to Windows HPC Queue. Otherwise, set up the CFX-Solver Manager as normal.

  3. Click Start Run on the CFX-Solver Manager to start the simulation. The first time a simulation is run, supply the required credentials that are prompted (this prompt may appear behind the CFX-Solver Manager window). You may elect to have these credentials saved to avoid repeated prompting of these credentials.


Note:
  1. %USERPROFILE% is typically C:\Users\<UserName> on Microsoft Windows HPC Server 2008.

  2. To use on ANSYS Workbench with Parameters and Design Points, start the CFX-Solver Manager ensure that the Run Mode on the Run Definition tab of the CFX-Solver Manager is set to Submit to Windows HPC Queue, set the number of processes, then click Save Settings. When the user clicks Update all Design Points each parameter or design point will be solved on the cluster.



Release 18.2 - © ANSYS, Inc. All rights reserved.