The following is an overview of the changes to clustering functionality found in the release of WildFly 8. To read the announcement see WildFly 8 Final is released! · WildFly.
New Features
Undertow Support
WildFly introduced a new web container called Undertow. WildFly adds both web session clustering support (see below) and mod_cluster integration with Undertow (see further below).
New Web Session Clustering Implementation
[WFLY-406] Redesign web session clustering - JBoss Issue Tracker
The web session clustering implementation found in AS7 has been around more-or-less in its current form for ages (since JBoss AS 3.2!) and suffered from a number of issues:
- The code base is tightly coupled to the web subsystem, i.e. JBoss Web.
- Sessions are effectively store in 2 places, once within the session manager, and again within the distributed cache. This creates a multitude of corner cases which can result in stale session data following failover. There are a number of open issues relating to this.
- The current implementation involves concurrent access to 2 data structures - consequently, an external locking mechanism is necessary to prevent deadlocking.
- Infinispan atomic maps, on which the AS7 implementation relies, have proven to be problematic.
- Early/Late request pipeline access to the session (e.g. from a request listener) results in multiple Infinispan batches (i.e. multiple cache transactions) per request, resulting in poor performance.
- The implementation in AS7 does not play well with shared cache store configurations (e.g. where sessions are stored in a shared database or data grid).
The new implementation addresses the above issues in the following ways:
- The new implementation introduces a proper session manager SPI, an Infinispan implementation of that SPI, and separate integration modules for Undertow vs JBoss Web that simply delegate to the SPI.
- Sessions are implemented as a facade over one or more cache entries - the container’s session manager itself does not retain a separate reference to each HttpSession.
- Pessimistic locking of cache entries effectively ensures that only a single client on a single node ever accesses a given session at any given time.
- The new implementation utilizes cache entry grouping instead of atomic maps to ensure that multiple cache entries belonging to the same session are co-located.
- The new implementation ensures that session operations within a request only ever use a single batch/transaction. This results in fewer RPCs per request.
- The new implementation supports write-through cache stores as well as passivation-only cache stores.
The new web session clustering implementation deprecates/reinterprets much of the configuration from jboss-web.xml.
<max-active-sessions/>
Previously, session creation would fail if an additional session would cause the number of active sessions to exceed the value specified by <max-active-sessions/>.
In the new implementation, <max-active-sessions/> is used to enable session passivation. If session creation would cause the number of active sessions to exceed <max-active-sessions/>, then the oldest session known to the session manager will passivate to make room for the new session.
<passivation-config/>
This configuration element and its sub-elements are no longer used in WildFly.
<use-session-passivation/>
Previously, passivation was enabled via this attribute.
In the new implementation, passivation is enabled by specifying a non-negative value for <max-active-sessions/>.
<passivation-min-idle-time/>
Previously, sessions needed to be active for at least a specific amount of time before becoming a candidate for passivation. This could cause session creation to fail, even when passivation was enabled.
The new implementation does not support this logic and thus avoids this DoS vulnerability.
<passivation-max-idle-time/>
Previously, a session would be passivated after it was idle for a specific amount of time.
The new implementation does not support eager passivation - only lazy passivation. Sessions are only passivated when necessary to comply with <max-active-sessions/>.
<replication-config/>
The new implementation deprecates a number of sub-elements:
<replication-trigger/>
Previously, session attributes could be treated as either mutable or immutable depending on the values specified by <replication-trigger/>:
- SET treated all attributes as immutable, requiring a separate HttpSession.setAttribute(...) to indicate that the value changed.
- SET_AND_GET treated all session attributes as mutable.
- SET_AND_NON_PRIMITIVE_GET recognized a small set of types (i.e. strings and boxed primitives) as immutable, and assumed that any other attribute was mutable.
The new implementation replaces this configuration option with a single, robust strategy. Session attributes are assumed to be mutable unless one of the following is true:
- The value is a known immutable value:
- null
- java.util.Collections.EMPTY_LIST, EMPTY_MAP, EMPTY_SET
- The value type is or implements a known immutable type:
- Boolean, Byte, Character, Double, Float, Integer, Long, Short
- java.lang.Enum, StackTraceElement, String
- java.io.File, java.nio.file.Path
- java.math.BigDecimal, BigInteger, MathContext
- java.net.InetAddress, InetSocketAddress, URI, URL
- java.security.Permission
- java.util.Currency, Locale, TimeZone, UUID
- The value type is annotated with @org.wildfly.clustering.web.annotation.Immutable
<use-jk/>
Previously, the instance-id of the node handling a given request was appended to the jsessionid (for use by load balancers such as mod_jk, mod_proxy_balancer, mod_cluster, etc.) depending on the value specified for <use-jk/>.
In the new implementation, the instance-id, if defined, is always appended to the jsessionid.
<max-unreplicated-interval/>
Previously, this configuration option was an optimization that would prevent the replicate of a session’s timestamp if no session attribute was changed. While this sounds nice, in practice it doesn't prevent any RPCs, since session access requires cache transaction RPCs regardless of whether any session attributes changed.
In the new implementation, the timestamp of a session is replicated on every request. This prevents stale session meta data following failover.
<snapshot-mode/>
Previously, one could configure <snapshot-mode/> as INSTANT or INTERVAL.
Infinispan’s replication queue renders this configuration option obsolete.
<snapshot-interval/>
Only relevant for <snapshot-mode>INTERVAL</snapshot-mode>. See above.
<session-notification-policy/>
Previously, the value defined by this attribute defined a policy for triggering session events.
In the new implementation, this behavior is spec-driven and not configurable.
New Clustered Single Sign-On Implementation
[WFLY-2404] Port SingleSignOn Valve to Undertow auth mechanism - JBoss Issue Tracker
The cluster single sign-on implementation found in AS7 has been around almost as long as the distributed web session implementation. It was also too tightly coupled to JBoss Web, and suffered from the duplicate data problem, where authentication information was stored both in memory in a local hash map, and again in the distributed cache. Additionally, a single sign-on entry would store a user's credentials, i.e. the authentication input; rather than the user's identity, i.e. the authentication result. Lastly, single sign-on needed to be configured differently if your web applications were distributable, than if they were local (i.e. non-distributable).
The new implementation similarly creates a single sign-on manager SPI, which serves to abstract the clustering logic from the web container integration code. The user identities, along with the set of web applications to which a user is authenticated, is stored only within the distributed cache. This information is stored in a separate cache instance per host, unlike the previous implementation that use a single cache instance for all hosts. There is also no additional configuration required to enable clustered SSO. This behavior is enabled automatically if the server is started using an HA profile.
Single sign-on is configured for a specific host, within the Undertow subsystem. E.g.:
<subsystem xmlns="urn:jboss:domain:undertow:1.0"> <!-- ... --> <server name="default-server"> <http-listener name="default" socket-binding="http"/> <host name="default-host" alias="localhost"> <location name="/" handler="welcome-content"/> <single-sign-on domain="localhost"/> </host> </server> <!-- ... --> </subsystem>
The domain attribute, if unspecified, defaults to the name of the containing host. This value corresponds to the domain of the cookie used to store the SSO identifier on the client.
New @Stateful Session EJB Clustering Implementation
[WFLY-2298] Redesign @Stateful EJB clustering - JBoss Issue Tracker
[WFLY-2419] SFSBs should be distributable by default in HA profiles - JBoss Issue Tracker
[WFLY-2363] Deprecate @Clustered EJB annotation - JBoss Issue Tracker
The SFSB clustering implementation found in AS7 was designed back in 2008 for use with JBoss Cache. It utilizes a cache instance per EJB class and a global group cache for storing reference groups of SFSB instances. While only in use since AS 7.1, the translation to Infinispan introduced a number of issues:
- The current implementation is very slow. Each EJB invocation results in 4 Infinispan transactions! Cache.get(...) and Cache.release(...) each involve 2 transactions, 1 for the bean cache and 1 for the group cache.
- An external locking mechanism is used to guard the SFSB against access from other nodes between Cache.get(...) and Cache.release(...)
- Because the current design uses 2 separate cache instances, with separate transaction contexts, data is not correctly isolated
- The current design uses a global group cache, which results in classloader leaks when a server has multiple EJB deployments (MarshalledValues in the group cache, which survives undeployment, can still reference the classloader of an undeployed EJB)
- The lifecycle of the global group cache is not managed by the MSC and thus is prone to race conditions by multiple SFSB deployments
- State changes that occur to the SFSB during activation/passivation callbacks are not correctly replicated.
- The @Clustered annotation currently has 2 orthogonal meanings:
- If applied to a @Stateful EJB, then a specific cache will be used (as defined by the EJB subsystem)
- If applied to a @Stateless or @Stateful EJB, remote EJB clients will have a dynamic view of the EJB's topology.
- Eager passivation is unnecessarily expensive, and makes the eventual SFSB removal slower.
WildFly 8 introduces a new clustering implementation that addresses all of the issues identified above. It uses a single Infinispan cache instance per deployment unit. Depending on your environment, you may see up to a 4x performance improvement.
WildFly 8 no longer requires @Stateful session EJBs to use the @Clustered annotation to enable clustering behavior. By default, if WildFly is started using an HA profile, the state of your SFSBs will be replicated. Disabling this behavior is achievable on a per-EJB basis by annotating your bean using @Stateful(passivationCapable=false), which is new to the EJB 3.2 specification; or globally, via the ejb3 subsystem.
Mod_cluster Support for Undertow
Support for new web container Undertow is realized within the mod_cluster subsystem, in the 'undertow' module. Since Undertow is built on top of completely different architecture than Tomcat-based JBoss Web, the behavior of metrics is a little different:
- 'busyness' used to represent the thread pool usage, which doesn't translate cleanly to Undertow's architecture. It now represents number of currently being processed requests. You are supposed to set capacity explicitly.
- 'mem' metric has been dropped since it did not reflect the actual memory usage, read the discussion on MODCLUSTER-288 (you can reintroduce a similar metric using custom metric implementation).
Mod_cluster has been upgraded to version 1.3.0.Final which includes number fixes and improvements. It is recommended to upgrade Apache httpd with the latest native modules as well to synchronize versions used by WildFly 8 (newer Apache httpd modules are always backwards compatible with older container implementation).
Current limitation: mod_cluster will now only register the default server, instead of all the configured Undertow servers. See WFLY-2876 for updates.
Public API for Clustering Services
WildFly introduces a refined public clustering API for use by applications. Gone are the days of the cumbersome HAPartition and CoreGroupCommunicationService. The new services are designed to be lightweight, easily injectable, with no external dependencies.
org.wildfly.clustering.group.Group
The Group service provides a mechanism to view the cluster topology for a channel or cache, and to be notified when the topology changes. A cache group, defined as the set of nodes in a cluster on which a given cache is deployed, is always the same as, or a subset of, the corresponding channel group.
e.g.
@Resource(lookup = "java:jboss/clustering/group/channel-name") private Group channelGroup;
org.wildfly.clustering.dispatcher.CommandDispatcher
The CommandDispatcherFactory service provides a mechanism to create a dispatcher for executing commands on nodes in the cluster. The resulting CommandDispatcher is a command-pattern analog to the reflection-based GroupRpcDispatcher from previous AS releases.
e.g.
@Resource(lookup = "java:jboss/clustering/dispatcher/channel-name")
private CommandDispatcherFactory factory;
public void foo() {
String context = "Hello world!";
try (CommandDispatcher<String> dispatcher = this.factory.createCommandDispatcher(context)) {
dispatcher.executeOnCluster(new StdOutCommand());
}
}
public static class StdOutCommand implements Command<Void, String> {
@Override
public Void execute(String context) {
System.out.println(context);
return null;
}
}
New Singleton Service Builder API
AS 7.2 introduced singleton services - a mechanism for installing an MSC service such that it will only start on one node in the cluster at a time. While not terribly difficult to implement, the installation process suffered from a couple shortcomings:
- Installing multiple singleton services within a single deployment caused the deployer to hang.
- Installing a singleton service required the user to specify several private module dependencies in /META-INF/MANIFEST.MF
WildFly introduces a new public API for building singleton services that simplifies the process significantly.
- The SingletonServiceBuilder implementation install its services so they will start asynchronously, preventing deadlocking of the MSC.
- Requires no private module dependencies within /META-INF/MANIFEST.MF.
e.g.
public class MyServiceActivator implements ServiceActivator {
private static final String CONTAINER_NAME = "server";
private static final String CACHE_NAME = "default";
@Override
public void activate(ServiceActivatorContext context) {
ServiceName name = ServiceName.of(...);
MyService service = new MyService(...);
ServiceName factoryServiceName = SingletonServiceBuilderFactory.SERVICE_NAME.append(CONTAINER_NAME, CACHE_NAME);
ServiceController<?> factoryService = context.getServiceRegistry().getRequiredService(factoryServiceName);
SingletonServiceBuilderFactory factory = (SingletonServiceBuilderFactory) factoryService.getValue();
factory.createSingletonServiceBuilder(name, service)
.electionPolicy(new SimpleSingletonElectionPolicy())
.requireQuorum(2)
.build(context.getServiceTarget())
.addDependency(...)
.install()
;
}
}
Cross-site Replication Support
WildFly 8 now includes support for cross-site replication, a mechanism which allows maintaining hot-backups of cache data across sites (distinct clusters). If your organization maintains multiple WildFly clusters, you can arrange for data in an Infinispan cache created and modified at one site to be backed up at one or more remote sites, where the remote backup caches are updated after each local cache operation.
Cross-site replication can be set up with a few very simple changes to the Infinispan and JGroups subsystem configurations. Changes to the jgroups subsystem include adding the RELAY2 protocol to enable inter-site communication. Changes to the Infinispan subsystem include defining which caches should be backed up and where.
For example, suppose your organization has three WildFly clusters in New York (NYC), London (LON) and San Francisco (SFO) and you would like key data in the New York site (stored in cache "dataCache" in cache container "dataContainer") to be backed up to London and San Francisco. In the subsystem configuration files on site NYC, as mentioned, you would need to make changes to both the jgroups and the Infinispan subsystems. We illustrate the changes required to hosts in the NYC site which want to participate in the backup scheme.
Firstly, to enable inter-site communication for applications using the UDP stack, the JGroups subsystem needs the following changes:
<stack name="udp"> <transport type="UDP" socket-binding="jgroups-udp"> <protocol type="PING"/> ... <protocol type="RSVP" <relay site="NYC"> <remote-site name="LON" stack="tcp" cluster="bridge"/> <remote-site name="SFO" stack="tcp" cluster="bridge"/> </relay> </stack>
The new <relay/> child element of <stack/> adds the RELAY2 inter-site communication protocol to a jgroups stack. The relay element defines the local site via the site attribute. Child remote-site elements are used to reference the sites for which inter-site communication should be enabled. Inter-site communication is based on defining what might be thought of as a "cluster of sites": each site is a member of the named cluster and can specify a common transport to be used for inter-site communication. In this case, the cluster used for inter-site communication is called "bridge" and uses the "tcp" stack definition for sending backup operations from a local site to it's remote backup sites.
Secondly, to define the local cache to be backed up and where the remote backups should live, the Infinispan subsystem needs these changes:
<cache-container name="dataContainer"> <replicated-cache name="dataCache" ...> ... <backups> <backup site="LON" failure-policy="WARN" strategy="SYNC" timeout="10000" enabled="true"/> <backup site="SFO" failure-policy="WARN" strategy="ASYNC" timeout="10000" enabled="true"/> </backups> ... </replicated-cache> </cache-container>
Within the cache definition for "dataCache", we use one backup element for each remote backup required. The backup element is used to configure the operation of the remote backup via its attributes. The site attribute specifies the remote site on which the backup cache resides. The cache container and the cache used for remote backup must be created by the remote site administrator and have the same names as on the local site; if this is not convenient, a backup-for element can be used to define that a differently named cache is to be used as the backup cache. The strategy attribute is used to control whether inter-site communication is synchronous or asynchronous. The failure-policy controls what happens to the local cache operation if the backup operation fails (e.g. WARN logs a warning message locally but otherwise lets the local operation succeed). The related timeout attribute defines the time within which the backup operation should complete before being marked as failed. The enabled attribute is used to activate and deactivate the backup.
The above configuration scenario is completed by adding the appropriate <relay/> elements to the jgroups subsystem configurations of the hosts in the LON and SFO clusters. Because data is only being backed up from NYC to LON and SFO, no <backup/> elements are required at LON and SFO.
Cluster Monitoring Subsystem (deferred until WF9)
https://issues.jboss.org/browse/WFLY-1430
WildFly 8 includes a new subsystem called clustering-monitor. The main purpose of this new subsystem is twofold:
- to allow detailed statistics monitoring of clusters and the applications running on those clusters
- to provide a location within the management API for clustering diagnostic tools
The clustering-monitor subsystem is being introduced in WildFly 8 with statistics monitoring of clusters and their deployed applications implemented, and will have additional features added in subsequent releases.
The resources in the clustering-monitor are organized as follows:
/subsystem=clustering-monitor /subsystem=clustering-monitor/cluster=* /subsystem=clustering-monitor/cluster=*/deployment=* /subsystem=clustering-monitor/cluster=*/deployment=*/web=WEB /subsystem=clustering-monitor/cluster=*/deployment=*/bean=*
The addressable resources are as follows:
- cluster=X represents a jgroups channel named X
- cluster=X/deployment=Y represents an J2EE application deployment Y on a channel X
- cluster=X/deployment=Y/web=WEB represents a web session cache used by a J2EE application deployment Y on a channel X
- cluster=X/deployment=Y/bean=Z represents a bean session cache for a @Clustered bean Z used by a J2EE application deployment Y on a channel X.
Each addressable resource provides its own set of attributes and operations, which reflect the key information when monitoring cluster status and performance. For example:
[standalone@localhost:9990 /] deploy ~/maven/distributable.war [standalone@localhost:9990 /] deploy ~/maven/my-ejb-project/target/my-ejb-project-1.0-SNAPSHOT.jar [standalone@localhost:9990 /] /subsystem=clustering-monitor/cluster= ejb web [standalone@localhost:9990 /] /subsystem=clustering-monitor/cluster=web : / [standalone@localhost:9990 /] /subsystem=clustering-monitor/cluster=web/deployment=distributable.war/ bean web [standalone@localhost:9990 /] /subsystem=clustering-monitor/cluster=web/deployment=distributable.war/web=WEB:read-resource(include-runtime=true) { "outcome" => "success", "result" => { "cache-view" => [("lenovo/web" => "[lenovo/web]")], "distribution" => [("lenovo/web" => "cache entries: 0")], "operation-stats" => [("lenovo/web" => "get(hits): 0, get(misses): 0, puts 0, remove(hits): 0, remove(misses): 0")], "rpc-stats" => [("lenovo/web" => "RPC count: 0, RPC failures: 0")], "txn-stats" => [("lenovo/web" => "prepares: 0, commits: 0, rollbacks: 0")] } } [standalone@localhost:9990 /] /subsystem=clustering-monitor/cluster=ejb/deployment=my-ejb-project-1.0-SNAPSHOT.jar/ bean web [standalone@localhost:9990 /] /subsystem=clustering-monitor/cluster=ejb/deployment=my-ejb-project-1.0-SNAPSHOT.jar/bean=CounterBean:read-resource(include-runtime=true) { "outcome" => "success", "result" => { "cache-view" => [("lenovo/ejb" => "[lenovo/ejb]")], "distribution" => [("lenovo/ejb" => "cache entries: 0")], "operation-stats" => [("lenovo/ejb" => "get(hits): 0, get(misses): 0, puts 0, remove(hits): 0, remove(misses): 0")], "rpc-stats" => [("lenovo/ejb" => "RPC count: 0, RPC failures: 0")], "txn-stats" => [("lenovo/ejb" => "prepares: 0, commits: 0, rollbacks: 0")] } } Component Upgrades
Infinispan 6.0
WildFly integrates the latest release stream from Infinispan. Notable improvements include:
- ISPN-3290 Improved cache store interoperability
- ISPN-2806 Faster file cache store implementation
- ISPN-2772 Unification of REPL and DIST modes
Distribution mode is now the default mode for the web and ejb caches. This should enable WildFly to scale well to a large number of nodes using the default configuration, without sacrificing performance for small clusters (containing 4 nodes or less).
JGroups 3.4
WildFly also includes the latest release from JGroups. This release introduces a new FORK protocol, which effectively can transform a linear protocol stack into a protocol tree. While not currently leveraged in WildFly, this has the potential to simplify port requirements, minimizing resources by allowing all clustering services to run under a single channel.
mod_cluster 1.3
Mod_cluster 1.3 includes numerous stability improvements. It is recommended to upgrade the httpd modules prior to updating the container. The versions are backwards compatible.
For questions about clustering and WildFly please, please use the community forums at Clustering.
Comments