Cluster network configuration includes both operating system configuration (Linux), and JGroups clustering software configuration. EKM uses JGroups clustering software to provide clustering capabilities. The configuration of JGroups is set in the JBoss configuration.
In order to optimize JGroups network performance on Linux operating systems, configuration of the operating system is recommended. If the operating system is not configured to optimize JGroups network performance, you may observe warnings in the JBoss logs, similar to the following:
2016-08-17 10:42:49,058 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the send buffer of socket ManagedDatagramSocketBinding was set to 1MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) 2016-08-17 10:42:49,059 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the receive buffer of socket ManagedDatagramSocketBinding was set to 20MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux) 2016-08-17 10:42:49,059 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the send buffer of socket ManagedMulticastSocketBinding was set to 1MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max send buffer in the OS correctly (e.g. net.core.wmem_max on Linux) 2016-08-17 10:42:49,060 WARN [org.jgroups.protocols.UDP] (MSC service thread 1-1) JGRP000015: the receive buffer of socket ManagedMulticastSocketBinding was set to 25MB, but the OS only allocated 131.07KB. This might lead to performance problems. Please set your max receive buffer in the OS correctly (e.g. net.core.rmem_max on Linux)
These warnings indicate that the operating system is unable to allocate buffers for network traffic in the size requested by JGroups. This will limit JGroups network performance. The following steps can be used to configure the operating system to optimize JGroups network performance:
As the root user, modify the file: /etc/sysctl.conf.
Configure maximum values for buffer sizes as follows:
# Allow a 25MB UDP receive buffer for JGroups net.core.rmem_max = 26214400 # Allow a 1MB UDP send buffer for JGroups net.core.wmem_max = 1048576
As the root user, run the following command to refresh the configuration and allow the new buffer sizes to take effect:
sysctl -p
Without these configurations, EKM will still operate normally, but JGroups network performance will not be optimized.
JGroups configuration is highly specialized, and should only be performed by IT professionals with advanced networking experience. It is not necessary under normal circumstances to edit the JGroups configuration, but if you need to do so, the information in this section will help you.
JGroups configuration is based upon “stacks” which
define the set of network protocols and configuration settings that
are used by JGroups for network communications. JGroups stacks are
defined in the domain configuration (ekm.xml), in the jgroups configuration element,
as shown below (portions omitted):
<subsystem xmlns="urn:jboss:domain:jgroups:4.0"> ... <stack name="ekm"> <transport type="UDP" socket-binding="jgroups-udp-ekm"/> ...
The JGroups stack used by EKM is specified by a system property in the host controller
configuration (ekm-host.xml), as shown below (portions omitted). The
default stack used by EKM is ekm:
<servers>
<server name="node1" group="ekm-server-group">
<socket-bindings port-offset="${env.JBOSS_PORT_OFFSET}" />
<system-properties>
...
<property name="ekm.jgroups.stack" value="ekm" />JGroups stacks may contain configuration settings that refer to socket bindings that are
defined in the socket-bindings configuration element in the domain
configuration (ekm.xml):
<socket-binding-groups>
<socket-binding-group name="ha-sockets" default-interface="public">
...
<socket-binding name="jgroups-udp-ekm" interface="private" port="55201"
multicast-address="${jboss.default.multicast.
address:230.0.0.4}" multicast-port="45689" />
<socket-binding name="jgroups-udp-fd-ekm" interface="private" port="54201" />To edit the JGroups configuration for EKM, follow these steps:
On each EKM cluster node, stop the EKM server.
Edit the JBoss configuration (see JBoss Domain Mode and Configuration).
Configure the JGroups stack and/or socket bindings used by EKM according to your needs.
You may also define
a new JGroups stack for use by EKM, and configure EKM to
use this stack by changing the value of the ekm.jgroups.stack system property in the host controller configuration.
Important: The JBoss configuration contains additional JGroups stack definitions and socket-bindings which are used by the JBoss application server for other services. Be careful to change only the JGroups stack and associated socket bindings used by EKM.
On each EKM cluster node, start the EKM server.
The default configuration of JGroups for EKM uses UDP multicast for cluster network communications. If more than one EKM server[15] is installed on the same LAN, it may be necessary to configure the UDP multicast address and/or port uniquely for each EKM server, so that each EKM server only receives UDP multicast traffic destined for it, and not traffic destined for other EKM servers.
To change the JGroups multicast configuration of the EKM server:
Follow the steps in JGroups Configuration, with these additional instructions:
In step 3, configure the socket-binding named jgroups-udp-ekm. Change the value
of the multicast-port attribute to a unique
port number that is not used by any other EKM servers on the same
LAN.