This section describes:
A data management process is used to both stage and retrieve files for jobs submitted to RSM. It is a Java process that runs on the cluster submit host as the same user who submits the RSM job.
When a user starts a job in EKM, the user’s data management process writes job
data to the job data directory, which is specified in the
jobSourceRootPath setting in the
ekm.xml file. The cluster submit host accesses this
directory when managing the job.
In order for the data management process to execute properly, the job data directory must meet the following requirements:
The directory must exist before EKM is deployed and users begin running jobs, and must be accessible by the cluster submit host.
All users who will be submitting jobs must have read/write permission on this directory.
For more information about the jobSourceRootPath setting, see Specifying Remote Process
Policies (<remoteProcess>).
To see how the data management process works, see the example in The Creation and Structure of Job Directories.
When users set up a batch job in EKM, they have the option
of choosing a working directory for the job. If they do not choose
one, EKM will stage job files in the job data directory. This
directory is specified in the jobSourceRootPath setting in the ekm.xml file.
If a shared job data directory is being used, EKM will
create a workspace directory under the
job data directory (if not already present), followed by a username directory, and time stamp directory.
As an example in Windows:
In the ekm.xml file, the jobSourceRootPath is
set to \\HEADNODE\HPCTemp. This is a UNC path to
a shared directory named HPCTemp on the cluster
submit host. This translates to C:\HPCTemp on that
machine. This is also the cluster staging
directory.
John Smith (with user name jsmith) is signed in to the Default workspace.
When John runs a job, jsmith’s data
management process creates the following directory for the job:
C:\HPCTemp\Default\jsmith\20_May_2015_10.39.18_AM
When John selects input files for the job, the files are uploaded to the
20_May_2015_10.39.18_AM directory. The
cluster submit host accesses this directory to manage the job. Since the
working directory resides under the cluster staging directory
(C:\HPCTemp), the files do not need to be
transferred to the cluster staging directory.
Every time John starts a new job, a new subdirectory
named with the job’s time stamp will be created under the C:\HPCTemp\Default\jsmith directory.
When another user, Susan Thomas, starts a job, a directory will be created for her job under the job data directory in a similar way:
C:\HPCTemp\Default\sthomas\20_May_2015_11.40.26_AM
In a Linux configuration, if home directories are being used instead of a shared job directory, only a time stamp directory will be created under the job data directory, as shown in the following example:
/net/server/home/jsmith/jobRoot/20_May_2015_10.39.18_AM
For information about specifying the jobSourceRootPath setting, see Specifying Remote Process
Policies (<remoteProcess>).
Note: If using a shared job data directory, it is recommended that you create the workspace subdirectory up front, and ensure that all users who will be submitting jobs have read/write permission on that subdirectory. Additional steps may be needed depending on your security requirements. For more information, see Access and Permission Requirements of the Job Data Directory.
Before job submission begins, you must create the job data directory
that you have specified in the jobSourceRootPath setting, and ensure that it is accessible to all users who will
be submitting jobs. EKM will create workspace, user, and time
stamp subdirectories as described in The Creation and Structure of Job Directories. There are important considerations to be made regarding the job
data directory and its subdirectories. If you are working in Linux,
additional steps may be required depending on the directory that you
have chosen, and the level of security that you desire.
In Windows:
The job data directory must be a shared directory,
specified by a UNC path in the jobSourceRootPath setting.
Make sure that this directory is open to all users who will be running RSM jobs.
It is recommended that you create the workspace subdirectory up front, and ensure that all
users have read/write permission on this directory, so that each
user’s data management process will be able to create a user
directory for that user under the workspace directory. For example, if
your job data directory path is
\\HEADNODE\HPCTemp, and users will be
submitting jobs from the Default workspace, create
a Default directory under the
HPCTemp directory
(\\HEADNODE\HPCTemp\Default), and set
appropriate permissions on the Default
directory.
In Linux:
The use of home directories (via a relative path) is the recommended way of handling users’ job data in Linux. Home directories provide each user with guaranteed access to his or her own job data, while preventing unwanted access by other users. The home directory is accessible by RSM, and the permission requirements are satisfied because users have read/write access on their own home directories.
You may choose to use a shared job directory (via an absolute path) instead of home directories. This may be less secure, however, because users could potentially have access to each other’s data. If you do choose to use a shared job directory, it is recommended that you create the workspace subdirectory up front, and ensure that all users have read/write access on this subdirectory. Although not necessary, for tighter security you may also want to create user subdirectories up front and grant each user full read/write permissions on his or her user subdirectory rather than the workspace directory. Keep in mind, however, that if you remove read/write permissions on the workspace directory, you will need to create user subdirectories for any new users who will be added to EKM later on, and set up appropriate permissions on those directories.
It is important to ensure
that all users have at least access (execute) permission on all parent
folders in the chain, right up to the top level. Consider the path /someshare/folder1/jobRoot/Default. If you granted users
read/write permissions on the jobRoot and Default folders, but did not grant them access permission
on folder1, job submission would fail for
those users.
For information about specifying the jobSourceRootPath setting, see Specifying Remote Process
Policies (<remoteProcess>).