4.8. Example: Setting Up a Multi-Node ANSYS RSM Cluster (ARC)

The example provided in this section contains detailed, step-by-step instructions for setting up a multi-node ANSYS RSM Cluster (ARC), and creating a cluster configuration that enables users to submit jobs to this cluster.


Note:  Multi-node ARC configuration requires system administrator or root permission and should only be performed by an IT administrator.


Scenario

Cluster Nodes

There are 4 machines available for scheduling and running jobs. Their names and roles are described below.

Note that if we wanted to use ARCMASTER to run jobs as well, we would simply need to install the Node service on that machine. For this example, we will assume that only EXECHOST1, EXECHOST2 and EXECHOST3 will be used for running jobs.


Note:  All nodes in an ANSYS RSM Cluster must run on the same platform (Windows or Linux). Instructions for both platforms are provided in this example.


Cluster Queues

The ARC will already have a local cluster queue for submitting jobs to the local machine, and a default cluster queue that can submit jobs to any of the execution nodes.

We are going to create a custom cluster queue named high_mem that will be dedicated to running jobs on EXECHOST3 only, which is the execution node with unrestricted resource allocation. We will also set the maximum number of jobs that can be run on this queue to 100.

Cluster Configuration

Once we have set up the ARC cluster, we will use the RSM Cluster Configuration application to create a cluster configuration named ARC. We will import the ARC cluster queues (local, local and high_mem) into the configuration and create RSM queues that map to these cluster queues.

Finally, we will make the ARC cluster configuration available to users so that the RSM queues defined in the configuration appear in client applications on their machines, enabling them to submit jobs to the RSM queues (which map to the ARC cluster queues). Jobs will be sent to ARCMASTER, where the Master service will dispatch jobs to the execution nodes.

Below is an overview of the ARC setup that we will be creating:

Step 1: Install the ARC Master Service on the Submit Host

Perform this step on the ARCMASTER machine, following the appropriate instructions below.

Installing the ARC Master Service on Windows

To install the ARC Master service on a Windows cluster head node, run the following command:

%AWP_ROOT182%\RSM\ARC\tools\winx64\installservice -arcmaster

The startup of the Master service is confirmed in the command window:

C:\Program Files\ANSYS Inc\v182\RSM\ARC\tools\winx64>installservice -arcmaster
Start installing ARC Master Service ...
Configuring RSM...

Configuring default RSM and ARC Port numbers for Release 18.2 ...
Done!

User has Administrator privileges

----------------------------------------------------------------------------------
Installing RSMArcMasterService182...
Microsoft (R) .NET Framework Installation utility Version 4.6.79.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Running a transacted installation.

Beginning the Install phase of the installation.
See the contents of the log file for the C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.exe 
assembly's progress.
The file is located at C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.InstallLog.
Installing assembly 'C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.exe'.
Affected parameters are:
   logtoconsole =
   assemblypath = C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.exe
   logfile = C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.InstallLog
Installing service RSMArcMasterService182...
Service RSMArcMasterService182 has been successfully installed.
Creating EventLog source RSMArcMasterService182 in log Application...
Used assemblypath parameter: C:\Program Files\ANSYS Inc\v182\RSM\ARC\

The Install phase completed successfully, and the Commit phase is beginning.
See the contents of the log file for the C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.exe 
assembly's progress.
The file is located at C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.InstallLog.
Committing assembly 'C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.exe'.
Affected parameters are:
   logtoconsole =
   assemblypath = C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.exe
   logfile = C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcMasterService.InstallLog

The Commit phase completed successfully.

The transacted install has completed.


The Ansys RSM Cluster Master Service Release 18.2 service is starting.
The Ansys RSM Cluster Master Service Release 18.2 service was started successfully.

Installing the ARC Master Service on Linux

When installing an ARC cluster service on Linux, you must determine if you want to install the service as a daemon that will start the service automatically when the machine is booted, or if you want to start the service manually via a startup script. Use only one of these methods.

When an ARC cluster service is started manually, it runs as a process for the user who initiated the service. A manually started ARC cluster service is stopped each time the machine is rebooted. After a reboot you must restart the ARC cluster service by running the startup script.

To manually start the ARC Master service on a Linux cluster head node, run the following command:

$AWP_ROOT182/RSM/ARC/tools/linx64/arcmaster start

To install the ARC Master service as a daemon, run the following command:

$AWP_ROOT182/RSM/ARC/tools/linx64/install_daemon arcmaster

Step 2: Install the ARC Node Service on the Execution Hosts

Perform this step on EXECHOST1, EXECHOST2 and EXECHOST3, following the appropriate instructions below.


Note:  If you wanted to use the master node (ARCMASTER) as an execution node, you would install the Node service on that node as well.


Installing the ARC Node Service on Windows

To install the ARC Node service on a Windows cluster head node, run the following command:

%AWP_ROOT182%\RSM\ARC\tools\winx64\installservice -arcnode

The startup of the Node service is confirmed in the command window:

C:\Program Files\ANSYS Inc\v182\RSM\ARC\tools\winx64>installservice -arcnode
Start installing ARC Node Service ...
Configuring RSM...

Configuring default RSM and ARC Port numbers for Release 18.2 ...
Done!

User has Administrator privileges

----------------------------------------------------------------------------------
Installing RSMArcNodeService182...
Microsoft (R) .NET Framework Installation utility Version 4.6.79.0
Copyright (C) Microsoft Corporation.  All rights reserved.


Running a transacted installation.

Beginning the Install phase of the installation.
See the contents of the log file for the C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.exe 
assembly's progress.
The file is located at C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.InstallLog.
Installing assembly 'C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.exe'.
Affected parameters are:
   logtoconsole =
   assemblypath = C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.exe
   logfile = C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.InstallLog
Installing service RSMArcNodeService182...
Service RSMArcNodeService182 has been successfully installed.
Creating EventLog source RSMArcNodeService182 in log Application...
Used assemblypath parameter: C:\Program Files\ANSYS Inc\v182\RSM\ARC\

The Install phase completed successfully, and the Commit phase is beginning.
See the contents of the log file for the C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.exe 
assembly's progress.
The file is located at C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.InstallLog.
Committing assembly 'C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.exe'.
Affected parameters are:
   logtoconsole =
   assemblypath = C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.exe
   logfile = C:\Program Files\ANSYS Inc\v182\RSM\ARC\bin\ArcNodeService.InstallLog

The Commit phase completed successfully.

The transacted install has completed.


The Ansys RSM Cluster Node Service Release 18.2 service is starting.
The Ansys RSM Cluster Node Service Release 18.2 service was started successfully.



----------------------------------------------------------------------------------

Installing the ARC Node Service on Linux

If you have chosen to run ARC cluster services manually, run the following command:

$AWP_ROOT182/RSM/ARC/tools/linx64/arcnode start

If you have chosen to install ARC cluster services as daemons, run the following command:

$AWP_ROOT182/RSM/ARC/tools/linx64/install_daemon arcnode

Step 3: Associate the Execution Nodes with the Master Node

Once cluster services have been installed we need to configure the execution nodes (EXECHOST1, EXECHOST2 and EXECHOST3) to report to the master node (ARCMASTER). This can be done from any of the cluster nodes.


Tip:  Running this command from the master node (instead of the execution node that you are configuring) is a good way to test that the firewall is allowing traffic as needed.


  1. On any cluster node, run the following command in the %AWP_ROOT182%/RSM/ARC/tools/winx64 or $AWP_ROOT182/RSM/ARC/tools/linx64 directory as appropriate:

    arcconfig node modify exechost1 -mn arcmaster

    Note the modified value for the Associated Master setting (Windows shown):

    C:\Program Files\ANSYS Inc\v182\RSM\ARC\tools\winx64>arcconfig node modify exechost1 -mn arcmaster
    
    Execution node configuration before update:
    
    
      Exec Node Name      Associated Master    Port      Max Cores    Max Memory    Max Disk
    ========================================================================================
    
        exechost1            exechost1         13181         *            *            *
    
        * Indicates that resources have not been set up.  Any resource request will be accepted.
    
    
    Execution node config updated: exechost1
    
    Current execution node setup after modification:
    
    
      Exec Node Name      Associated Master    Port      Max Cores    Max Memory    Max Disk
    ========================================================================================
    
        exechost1            arcmaster         13181         *            *            *
    
        * Indicates that resources have not been set up.  Any resource request will be accepted.
  2. Repeat for the other two nodes:

    arcconfig node modify exechost2 -mn arcmaster

    arcconfig node modify exechost3 -mn arcmaster

Step 4: Configure Execution Nodes

On EXECHOST1 and EXECHOST2 we will restrict the amount of resources that can be allocated to jobs on those machines. Each machine will be able to use a maximum of 10GB of disk space and 40MB of memory.

  1. On EXECHOST1, run the following command in the %AWP_ROOT182%/RSM/ARC/tools/winx64 or $AWP_ROOT182/RSM/ARC/tools/linx64 directory as appropriate:

    arcconfig node modify exechost1 -r d:10gb,m:40mb

    Note the change in the Max Memory and Max Disk values for this node:

    C:\Program Files\ANSYS Inc\v182\RSM\ARC\tools\winx64>arcconfig node modify exechost1 -r d:10gb,m:40mb
    
    Execution node configuration before update:
    
    
      Exec Node Name           Associated Master     Port      Max Cores    Max Memory    Max Disk
    ================================================================================================
    
         exechost1                 arcmaster         13182         *            *            *
    
        * Indicates that resources have not been set up.  Any resource request will be accepted.
    
    
    Execution node config updated: exechost1
    
    Current execution node setup after modification:
    
    
      Exec Node Name           Associated Master     Port      Max Cores    Max Memory    Max Disk
    ================================================================================================
    
         exechost1                 arcmaster         13182         *           40Mb         10Gb
    
        * Indicates that resources have not been set up.  Any resource request will be accepted.
  2. On EXECHOST2, run the following command:

    arcconfig node modify exechost2 -r d:10gb,m:40mb

Step 5: Cache the Password for Accessing the Cluster

Now that the cluster nodes are set up, we can cache the password that we want RSM to use when accessing the cluster. When no arguments are used, the arccredentials command will prompt you to enter the password for the currently logged-in user:

C:\Program Files\ANSYS Inc\v182\RSM\ARC\tools\winx64>arccredentials

Caching credentials for: ANSYS\atester

Password:

If we wanted RSM to use a different account to log in to the cluster, we would run the following command instead:

arccredentials -a Username

Step 6: Add a Custom Cluster Queue

We are going to create and enable a custom cluster queue named high_mem that will be dedicated to running jobs on EXECHOST3. This is the only machine with unrestricted resource allocation, making it ideal for larger jobs. We will also set the maximum number of jobs that can run on the high_mem queue to 100.

  1. On the ARCMASTER machine, run the following command in the %AWP_ROOT182%/RSM/ARC/tools/winx64 or $AWP_ROOT182/RSM/ARC/tools/linx64 directory as appropriate:

    arcconfig queue add high_mem

    An updated queue list is displayed:

    C:\Program Files\ANSYS Inc\v182\RSM\ARC\tools\winx64>arcconfig queue add high_mem
    
    Queue updated: high_mem
    
    Current queues after addition:
    
    
      Name     Status   Priority   Start Time   End Time   Max Jobs      Allowed Machines       Allowed Users
    =========================================================================================================
    
    default    Active      0        00:00:00    23:59:59      *     exechost1,exechost2,exechost3    all
     local     Active      0        00:00:00    23:59:59      *               arcmaster              all
    high_mem  Disabled     0        00:00:00    23:59:59      *                 None                 all
    
      * Indicates that resources have not been set up.  Any resource request will be accepted.

    Notice that the new queue is disabled, and has no machines assigned to it.

  2. Run the following command to enable the queue, add the EXECHOST3 machine to it, and set the maximum number of jobs to 100:

    arcconfig queue modify high_mem -e true -m exechost3 -n 100

    Note the change in the Status, Max Jobs, and Allowed Machines values:

    C:\Program Files\ANSYS Inc\v182\RSM\ARC\tools\winx64>arcconfig queue modify high_mem -e true -m exechost3 -n 100
    
    Queue list before update:
    
    
      Name     Status   Priority   Start Time   End Time   Max Jobs      Allowed Machines       Allowed Users
    =========================================================================================================
    
    default    Active      0        00:00:00    23:59:59      *     exechost1,exechost2,exechost3    all
     local     Active      0        00:00:00    23:59:59      *               arcmaster              all
    high_mem  Disabled     0        00:00:00    23:59:59      *                 None                 all
    
      * Indicates that resources have not been set up.  Any resource request will be accepted.
    
    
    Queue updated: high_mem
    
    Current queues after modification:
    
    
      Name     Status   Priority   Start Time   End Time   Max Jobs      Allowed Machines       Allowed Users
    =========================================================================================================
    
    default    Active      0        00:00:00    23:59:59      *     exechost1,exechost2,exechost3    all
     local     Active      0        00:00:00    23:59:59      *               arcmaster              all
    high_mem   Active      0        00:00:00    23:59:59     100              exechost3              all
    
      * Indicates that resources have not been set up.  Any resource request will be accepted.

At this point the ANSYS RSM Cluster is set up and ready to receive job submissions from client machines.

Step 7: Create a Cluster Configuration

In order for RSM client machines to be able to submit jobs to the ARC submit host (ARCMASTER), we must create a cluster configuration in RSM that establishes communication between the client and submit host, specifies the file transfer method, and specifies RSM queues that map to the ARC cluster queues.

  1. Launch the RSM Cluster Configuration application by selecting Start > All Programs > ANSYS 18.2 > RSM Cluster Configuration 18.2.

  2. Click  , or right-click in the Clusters list and select Add Cluster.

  3. On the Cluster tab, specify a name for the cluster configuration, the name of the cluster submit host, and select ARC for the cluster type. Since SSH is not being used to communicate with the submit host, ensure that Able to directly submit and monitor cluster jobs is selected:

  4. Click Apply, then select the File Management tab. Referring to Specifying File Management Properties, select the desired file transfer method, then specify the job execution working directory.

  5. Click Apply, then select the Queues tab.

    Click   to import the cluster queues that are defined on the ARCMASTER machine (default, local and high_mem).

    Since we are accessing ARCMASTER for the first time, we may be asked to enter the credentials that RSM will use to access that machine:

  6. Enter a User Name and Password that can be used to access ARCMASTER, then click OK. The specified credentials will be cached, enabling any user using this configuration to submit jobs to ARCMASTER.

  7. For each Cluster Queue in the list, you can specify a unique RSM Queue name if you want (by default, the RSM queue name matches the cluster queue name). RSM queues are what users see in client applications when they choose to submit jobs to RSM. You can also choose which queues you want to enable for users, and submit a test job to each RSM queue by clicking Submit in the Test column.

  8. Click Apply to complete the configuration.

Step 8: Make the Cluster Configuration Available to Users

Since we named our cluster configuration ARC, a file named ARC.rsmcc has been created in the cluster configuration directory. This directory also contains an RSM queue definition file named queues.rsmq. You will need to make these files available to users who will be submitting jobs to the cluster via RSM. You can do this by making the cluster configuration directory a shared directory, and instructing users to point their own cluster configuration directory setting to this shared directory. Alternatively, users can copy the cluster configuration files that you have created to the appropriate directory on their local machines.

With either of these options, if users were to launch the RSM Configuration application on their machines, they would see the ARC cluster configuration automatically added to their Clusters list. They could then start submitting jobs to the RSM queues that were defined in this configuration. RSM queues are linked to the ARC cluster configuration, which enables jobs to be submitted to ARCMASTER for scheduling.

There are two options for making the ARC.rsmcc and queues.rsmq files available to users:

Option 1: Share the cluster configuration directory

This method ensures that all users have the most accurate and up-to-date cluster configuration information, as files are centrally stored and managed.

Once the cluster configuration directory has been shared, users should set the cluster configuration directory on their local machines to the path of the shared cluster configuration directory. For example, the share path might be something like \\machineName\Share\RSM for Windows users, or /clusternodemount/share/RSM for Linux users. They will follow the steps in Changing the Cluster Configuration Directory.


Note:  One potential drawback of this method is that users may not be able to access to the shared cluster configurations if the host goes offline or cannot be accessed for some reason (for example, if a user is working off-site and does not have access to the network). In this case RSM will automatically switch the cluster configuration directory back to the default cluster configuration directory on their local machines. This means that users will, at a minimum, be able to submit jobs to ARC clusters already installed on their local machines using the localhost cluster configuration that is generated in the default cluster configuration directory when RSM is installed.


Option 2: Have users copy cluster configuration files to their local machines

If you are a user looking to access cluster configurations that have been defined by your RSM or cluster administrator, you can do so by setting your cluster configuration directory to the shared cluster configuration directory that was set by the administrator (see Option 1 above). Alternatively you can copy the cluster configuration database to the appropriate directory on your machine.

As a user, you will need to:

  1. Obtain the ARC.rsmcc and queues.rsmq files from the RSM or cluster administrator. If the administrator has put the files in a shared directory that you can access, you can retrieve them directly from there.

  2. On your local machine, copy the files into your default cluster configuration directory.

    By default, the directory in which the cluster configurations are stored resolves to the following location:

    Windows: %APPDATA%\ANSYS\v182\RSM

    The path to this directory might be C:\users\%username%\appdata\Roaming\Ansys\V182\RSM, where %username% is the name of the RSM or cluster administrator.

    Linux: ~/.ansys/v182/RSM

    On Linux, ~ is the home directory of the account under which RSM is being run.


Note:  If any of the shared files that you are copying have the same name as files in your local cluster configuration directory, you will need to rename your local files if you do not want them to be overwritten. For example, you may want to rename your localhost.rsmcc file to mylocalhost.rsmcc to distinguish it from the remote cluster's localhost.rsmcc file, as its settings may be different.

Alternatively, to avoid this issue altogether:

  1. Create a new folder on your local machine (for example, C:\SharedRSMConfig).

  2. Copy the cluster configurations from the network share to that folder.

  3. Use the RSM Utilities application to set the JobManagement ConfigurationDirectory setting to the new folder. See Changing the Cluster Configuration Directory.



Release 18.2 - © ANSYS, Inc. All rights reserved.