The performance meter allows you to report the wall clock time elapsed during a computation, as well as message-passing statistics. Since the performance meter is always enabled, you can access the statistics by displaying them after the computation is completed. To view the current statistics, click Usage in the Parallel ribbon tab (Timer group box).
Parallel → Timer
→ Usage
Performance statistics will be displayed in the console.
To clear the performance meter so that you can eliminate past statistics from the future report, click Reset in the Parallel ribbon tab (Timer group box).
Parallel → Timer
→ Reset
The following example demonstrates how the current parallel statistics are displayed in the console:
Performance Timer for 1 iterations on 4 compute nodes Average wall-clock time per iteration: 4.901 sec Global reductions per iteration: 408 ops Global reductions time per iteration: 0.000 sec (0.0%) Message count per iteration: 801 messages Data transfer per iteration: 9.585 MB LE solves per iteration: 12 solves LE wall-clock time per iteration: 2.445 sec (49.9%) LE global solves per iteration: 27 solves LE global wall-clock time per iteration: 0.246 sec (5.0%) AMG cycles per iteration: 64 cycles Relaxation sweeps per iteration: 4160 sweeps Relaxation exchanges per iteration: 920 exchanges Total wall-clock time: 4.901 sec
A description of the parallel statistics is as follows:
Average wall-clock time per iteration describes the average
real (wall clock) time per iteration.
Global reductions per iteration describes the number of global
reduction operations (such as variable summations over all processes). This requires
communication among all processes.
A global reduction is a collective operation over all processes for the given job that reduces a vector quantity (the length given by the number of processes or nodes) to a scalar quantity (for example, taking the sum or maximum of a particular quantity). The number of global reductions cannot be calculated from any other readily known quantities. The number is generally dependent on the algorithm being used and the problem being solved.
Global reductions time per iteration describes the time per
iteration for the global reduction operations.
Message count per iteration describes the number of messages
sent between all processes per iteration. This is important with regard to communication
latency, especially on high-latency interconnects.
A message is defined as a single point-to-point, send-and-receive operation between any two processes. This excludes global, collective operations such as global reductions. In terms of domain decomposition, a message is passed from the process governing one subdomain to a process governing another (usually adjacent) subdomain.
The message count per iteration is usually dependent on the algorithm being used and the problem being solved. The message count and the number of messages that are reported are totals for all processors.
The message count provides some insight into the impact of communication latency on parallel performance. A higher message count indicates that the parallel performance may be more adversely affected if a high-latency interconnect is being used. Ethernet has a higher latency than Myrinet or Infiniband. Therefore, a high message count will more adversely affect performance with Ethernet than with Infiniband.
To check the latency of the overall cluster interconnect, refer to Checking Latency and Bandwidth.
Data transfer per iteration describes the amount of data
communicated between processors per iteration. This is important with respect to
interconnect bandwidth.
Data transfer per iteration is usually dependent on the algorithm being used and the problem being solved. This number generally increases with increases in problem size, number of partitions, and physics complexity.
The data transfer per iteration may provide some insight into the impact of communication bandwidth (speed) on parallel performance. The precise impact is often difficult to quantify because it is dependent on many things including: ratio of data transfer to calculations, and ratio of communication bandwidth to CPU speed. The unit of data transfer is a byte.
To check the bandwidth of the overall cluster interconnect, refer to Checking Latency and Bandwidth.
LE solves per iteration describes the number of linear systems
being solved per iteration. This number is dependent on the physics (non-reacting versus
reacting flow) and the algorithms (pressure-based versus density-based solver), but is
independent of mesh size. For the pressure-based solver, this is usually the number of
transport equations being solved (mass, momentum, energy, and so on).
LE wall-clock time per iteration describes the time
(wall-clock) spent doing linear equation solvers (that is, multigrid).
LE global solves per iteration describes the number of
solutions on the coarsest level of the AMG solver where the entire linear system has
been pushed to a single processor (n0). The system is pushed to
a single processor to reduce the computation time during the solution on that level.
Scaling generally is not adversely affected because the number of unknowns is small on
the coarser levels.
LE global wall-clock time per iteration describes the time
(wall-clock) per iteration for the linear equation global solutions.
AMG cycles per iteration describes the average number of
multigrid cycles (V, W, flexible, and so on) per iteration.
Relaxation sweeps per iteration describes the number of
relaxation sweeps (or iterative solutions) on all levels for all equations per
iteration. A relaxation sweep is usually one iteration of Gauss-Siedel or ILU.
Relaxation exchanges per iteration describes the number of
solution communications between processors during the relaxation process in AMG. This
number may be less than the number of sweeps because of shifting the linear system on
coarser levels to a single node/process.
Time-step updates per iteration describes the number of
sub-iterations on the time step per iteration.
Time-step wall-clock time per iteration describes the time per
sub-iteration.
Total wall-clock time describes the total wall-clock
time.
The most relevant quantity is the Total wall clock
time. This quantity can be used to gauge the parallel performance (speedup
and efficiency) by comparing this quantity to that from the serial analysis.
You can check the latency and bandwidth of the overall cluster interconnect, to help identify any issues affecting ANSYS Fluent scalability, by clicking and in the Parallel ribbon tab (Network group box).
Parallel → Network
→ Latency
Depending on the number of machines and processors being used, a table containing information about the communication speed for each node will be displayed in the console. The table will also summarize the minimum and maximum latency between two nodes.
Consider the following example when checking for latency:
Latency (usec) with 1000 samples [1.83128 sec] ------------------------------------------ ID n0 n1 n2 n3 n4 n5 ------------------------------------------ n0 48.0 48.2 48.2 48.3 *50 n1 48.0 48.2 48.3 48.3 *48 n2 48.2 48.2 48.8 49.1 *53 n3 48.2 48.3 *49 48.6 48.5 n4 48.3 48.3 49.1 48.6 *50 n5 49.7 48.5 *53 48.5 49.7 ------------------------------------------ Min: 47.9956 [n0<-->n1] Max: 52.6836 [n5<-->n2] ------------------------------------------
Important: In the above table, (*) is the maximum value in that row. The smaller the latency, the better.
Six processors (n0 to n5) are spawned. The latency between n0 and n1 is 48.0
. Similarly, the latency between n1 and n2 is 48.2
. The minimum latency occurs between n0 and n1 and the maximum latency
occurs between n2 and n5, as noted in the table. Checking the latency is particularly
useful when you are not seeing expected speedup on a cluster.
Parallel → Network
→ Bandwidth
In addition to checking for latency, you can check your bandwidth. A table containing information about the amount of data communicated within one second between two nodes is displayed in the console. The table will also summarize the minimum and maximum bandwidth between two nodes.
Consider the following example when checking for bandwidth:
Bandwidth (MB/s) with 5 messages of size 4MB [4.36388 sec] -------------------------------------------- ID n0 n1 n2 n3 n4 n5 -------------------------------------------- n0 111.8 *55 111.8 97.5 101.3 n1 111.8 69.2 98.7 111.7 *51 n2 54.7 69.2 72.9 104.8 *45 n3 111.8 98.7 72.9 64.0 *45 n4 97.6 111.7 104.8 *64 76.9 n5 101.2 50.9 45.5 *45 76.9 -------------------------------------------- Min: 45.1039 [n5<-->n3] Max: 111.847 [n0<-->n3] --------------------------------------------
Important: In the above table, (*) is the minimum value in that row. The larger the bandwidth, the better.
The bandwidth between n0 and n1 is 111.8 MB/s. Similarly, the bandwidth between n1 and n2 is 69.2 MB/s. The minimum amount of bandwidth occurs between n3 and n5 and the maximum occurs between n0 and n3, as noted in the table. Checking the bandwidth is particularly useful when you cannot see good scalability with relatively large cases.