8.2.3. Configuring File Transfer by OS Type and Network Share Availability

Remote job execution on a cluster usually requires the transfer of files to and from a cluster directory. This section goes over the different options that are available for cluster-side and client-side integration in terms of how you can copy the files.

When creating a cluster configuration, settings on the File Management tab are used to specify information about the cluster staging directory and job execution directory. The following sections contain example configuration settings for different scenarios:

For each scenario, the cluster staging directory can be:

Also, for each scenario, you can be using different types of communication:

8.2.3.1. Windows Client to Windows Cluster

In the following two scenarios, a Windows RSM client is integrated with a Windows cluster. Windows to Windows SSH is not supported natively.

8.2.3.1.1. Windows-to-Windows, Staging Visible

In this scenario, the Windows client can “see” the Windows cluster staging area via a network share or mapped drive.

  1. On the Cluster tab, set the appropriate Custom cluster type as described in Customizing Cluster-Side Integration or Customizing Client-Side Integration.

  2. On the File Management tab:

    • Select Operating system file system transfer to existing network share (Samba, CIFS, NFS). With this option, RSM will (attempt to) copy jobs to and from the cluster staging directory using the native Windows OS “copy” commands.

      In the Cluster staging network share field, specify the path to the shared file system as the RSM client sees it. A Windows client will see the shared file system as a UNC path (for example, \\Share_Host_Name\Share\Directory).

      All of the nodes and the client machine (the one submitting the job) should be able to access this share.

    • If jobs will run In a scratch directory local to the execution node, specify the path of the desired local scratch space on the cluster. This local scratch directory must be exactly the same location on all of the nodes, should not be shared, and should be in the form of D:\storage\RsmTemp.

8.2.3.1.2. Windows-to-Windows, Staging Not Visible

In this scenario, the Windows client cannot “see” the Windows cluster staging area; either there is a firewall or this storage is otherwise not allowed to be directly accessed by the users.

  1. On the Cluster tab, set the appropriate Custom cluster type as described in Customizing Cluster-Side Integration or Customizing Client-Side Integration.

  2. On the File Management tab:

    • Select RSM internal file transfer mechanism, then specify the path to the staging area in the Cluster staging directory field (for example, Z:\storage\RsmTempShare). With this option, RSM will use TCP sockets to stream files from the client machine to the submit host machine. This method is slightly slower than the OS copy method noted above in Windows-to-Windows, Staging Visible.

      When transferring files to a single node cluster, it may not be necessary for the staging directory to be a shared path (for example, a UNC path).

      This directory must be visible to all cluster nodes.

    • If jobs will run In a scratch directory local to the execution node, specify the path of the desired local scratch space on the cluster. This local scratch directory must be exactly the same location on all of the nodes, should not be shared, and should be in the form of D:\storage\RsmTemp.

8.2.3.2. Windows Client to Linux Cluster

In the following two scenarios, a Windows RSM client is integrated with a Linux cluster. Windows to Linux SSH is supported, so additional commentary about SSH is included in these sections.

8.2.3.2.1. Windows-to-Linux, Staging Visible

In this scenario, the Windows client can “see” the Linux cluster staging area via a Samba UNC or mapped drive.

  1. On the Cluster tab:

  2. On the File Management tab:

    • Select Operating system file system transfer to existing network share (Samba, CIFS, NFS). With this option, RSM will (attempt to) copy jobs to and from the cluster staging directory using the native Windows OS “copy” commands.

      In the Cluster staging network share field, specify the path to the shared file system as the RSM client sees it. A Windows client will see the shared file system as a UNC path (for example, \\machine\STAGING).

      In the Cluster staging directory field, specify the path to the shared file system as the cluster sees it (for example, /staging on a Linux machine). This maps the client-visible path to the cluster-visible path. For example, the Windows client sees \\machine\STAGING which is a Linux Samba share of /staging.

      All of the nodes and the client machine (the one submitting the job) should be able to access this share.

    • If jobs will run In a scratch directory local to the execution node, specify the path of the desired local scratch space on the cluster. This local scratch directory must be exactly the same location on all of the nodes, should not be shared, and should be in the form of /path/to/cluster/nodes/individual/scratch.

8.2.3.2.2. Windows-to-Linux, Staging Not Visible

In this scenario, the Windows RSM client cannot “see” the Linux cluster staging area.

  1. On the Cluster tab:

  2. On the File Management tab:

    • Select RSM internal file transfer mechanism, then specify the path to the staging area in the Cluster staging directory field (for example, /path/to/shared/cluster/directory). With this option, RSM will use TCP sockets to stream files from the client machine to the submit host machine. This method is slightly slower than the OS copy method noted in Windows-to-Linux, Staging Visible.

      When transferring files to a single node cluster, it may not be necessary for the staging directory to be a shared path.

      This directory must be visible to all cluster nodes.

    • If using local scratch, specify the path of the desired (cluster) local scratch space on the cluster. This local scratch directory must be exactly the same location on all of the nodes and should be in the form of /path/to/cluster/nodes/individual/scratch.

  3. You will need to copy an example transfer script (or modify/write your own) such as transfer_<YourKeyword>.py as shown in Creating Copies of Sample Code Using a Custom Client Keyword.

8.2.3.3. Linux Client to Linux Cluster

In the following two scenarios, a Linux Client machine is integrated with a Linux cluster. Linux to Linux SSH is supported, so additional commentary about SSH is included in these sections.

8.2.3.3.1. Linux-to-Linux, Staging Visible

In this scenario, the Linux RSM client can “see” the Linux cluster staging area because the staging area is mounted on the client machines.

  1. On the Cluster tab:

  2. On the File Management tab:

    • Select Operating system file system transfer to existing network share (Samba, CIFS, NFS). With this option, RSM will (attempt to) copy jobs to and from the cluster staging directory using the native Linux OS “copy” commands.

      In the Cluster staging network share field, specify the path to the shared file system as the RSM client sees it. A Linux client will mount the cluster staging directory (for example, /mounts/cluster1/staging).

      In the Cluster staging directory field, specify the path to the shared file system as the cluster sees it (for example, /staging).

      All of the nodes and the client machine (the one submitting the job) should be able to access this share.

    • If jobs will run In a scratch directory local to the execution node, specify the path of the desired local scratch space on the cluster. This local scratch directory must be exactly the same location on all of the nodes, should not be shared, and should be in the form of /path/to/cluster/nodes/individual/scratch.

8.2.3.3.2. Linux-to-Linux, Staging Not Visible

In this scenario, the Linux Client cannot “see” the Linux cluster staging area.

  1. On the Cluster tab:

  2. On the File Management tab:

    • Select RSM internal file transfer mechanism, then specify the path to the staging area in the Cluster staging directory field (for example, /path/to/shared/cluster/directory). With this option, RSM will use TCP sockets to stream files from the client machine to the submit host machine. This method is slightly slower than the OS copy method noted in Linux-to-Linux, Staging Visible.

      When transferring files to a single node cluster, it may not be necessary for the staging directory to be a shared path.

      This directory must be visible to all cluster nodes.

    • If using local scratch, specify the path of the desired (cluster) local scratch space on the cluster. This local scratch directory must be exactly the same location on all of the nodes and should be in the form of /path/to/cluster/nodes/individual/scratch.

  3. You will need to copy an example transfer script (or modify/write your own) such as transfer_<YourKeyword>.py as shown in Creating Copies of Sample Code Using a Custom Client Keyword.


Release 18.2 - © ANSYS, Inc. All rights reserved.