Wildfly EJB clustering ClassNotFoundException

[stephen.lorenz]

Hi,

We are trying to setup a 2 server cluster in Wildfly 8.1.0.  Our application consists of a web tier (war) and ejb tier (ear).  We have added the <distributable/> element to our war web.xml file and we have annotated our ejb's as @Clustered (though we understand this is deprecated).  In front of this web application we are running a round robin load balancer.  We are running into strange behavior once we attempt to login to our web application.  It appears that some requests succeed and others fail.  The ones that fail appear to be hitting the other instance from the one we hit while logging in.  The exception thrown by the failing instance is intriguing:

2014-06-17 23:36:39,057 ERROR [io.undertow.request] (default task-11) Blocking request failed HttpServerExchange{ GET /example_web/secure/dashboard.xhtml}: java.lang.IllegalArgumentException: java.lang.ClassNotFoundException: com.example.ejb.AuditManager$1778645733$Proxy$_$$_Weld$Proxy$ from [Module "deployment.example_web.war:main" from Service Module Loader]

  at org.wildfly.clustering.web.infinispan.session.MarshalledValueSessionAttributeMarshaller.read(MarshalledValueSessionAttributeMarshaller.java:48)

  at org.wildfly.clustering.web.infinispan.session.MarshalledValueSessionAttributeMarshaller.read(MarshalledValueSessionAttributeMarshaller.java:33)

  at org.wildfly.clustering.web.infinispan.session.coarse.CoarseImmutableSessionAttributes.getAttributes(CoarseImmutableSessionAttributes.java:46)

  at org.wildfly.clustering.web.infinispan.session.coarse.CoarseImmutableSessionAttributes.getAttributeNames(CoarseImmutableSessionAttributes.java:51)

  at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findListeners(InfinispanSessionManager.java:320)

  at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.triggerPostActivationEvents(InfinispanSessionManager.java:309)

  at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:164)

  at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)

  at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:677)

  at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:707)

  at io.undertow.servlet.spec.ServletContextImpl.updateSessionAccessTime(ServletContextImpl.java:711)

  at io.undertow.servlet.spec.HttpServletResponseImpl.responseDone(HttpServletResponseImpl.java:522)

  at io.undertow.servlet.spec.HttpServletResponseImpl.sendError(HttpServletResponseImpl.java:137)

  at io.undertow.servlet.spec.HttpServletResponseImpl.sendError(HttpServletResponseImpl.java:142)

  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:273)

  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:227)

  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:73)

  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:146)

  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:177)

  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:727)

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_45]

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_45]

  at java.lang.Thread.run(Thread.java:744) [rt.jar:1.7.0_45]

Caused by: java.lang.ClassNotFoundException: com.example.ejb.AuditManager$1778645733$Proxy$_$$_Weld$Proxy$ from [Module "deployment.example_web.war:main" from Service Module Loader]

  at org.jboss.modules.ModuleClassLoader.findClass(ModuleClassLoader.java:213) [jboss-modules.jar:1.3.3.Final]

  at org.jboss.modules.ConcurrentClassLoader.performLoadClassUnchecked(ConcurrentClassLoader.java:459) [jboss-modules.jar:1.3.3.Final]

  at org.jboss.modules.ConcurrentClassLoader.performLoadClassChecked(ConcurrentClassLoader.java:408) [jboss-modules.jar:1.3.3.Final]

  at org.jboss.modules.ConcurrentClassLoader.performLoadClass(ConcurrentClassLoader.java:389) [jboss-modules.jar:1.3.3.Final]

  at org.jboss.modules.ConcurrentClassLoader.loadClass(ConcurrentClassLoader.java:134) [jboss-modules.jar:1.3.3.Final]

  at java.lang.Class.forName0(Native Method) [rt.jar:1.7.0_45]

  at java.lang.Class.forName(Class.java:270) [rt.jar:1.7.0_45]

  at org.jboss.marshalling.ModularClassResolver.resolveClass(ModularClassResolver.java:102)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadClassDescriptor(RiverUnmarshaller.java:943)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1239)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

  at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1712)

  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1628)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1269)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

  at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1712)

  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1628)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1269)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:272)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadMapObject(RiverUnmarshaller.java:195)

  at org.jboss.marshalling.river.RiverUnmarshaller.readMapData(RiverUnmarshaller.java:819)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:682)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

  at org.jboss.as.clustering.marshalling.SimpleMarshalledValue.get(SimpleMarshalledValue.java:101)

  at org.jboss.as.clustering.marshalling.SimpleMarshalledValue.get(SimpleMarshalledValue.java:45)

  at org.wildfly.clustering.web.infinispan.session.MarshalledValueSessionAttributeMarshaller.read(MarshalledValueSessionAttributeMarshaller.java:46)

  ... 22 more

This seems to be a serialization error but we cannot figure out what we are doing wrong.  The class in question is marked as Stateless and implements Serializable.

We have tested the example, https://github.com/arun-gupta/wildfly-samples/tree/master/clustering/http, and this war works find in our cluster.  Unfortunately, this example does not include any ejb's to compare to our project.

Thanks!

[rhusar]

Let me try to find out what you might be doing wrong or whether this is a potential bug.

Meanwhile make sure you are running the exact same application build on both servers.

In front of this web application we are running a round robin load balancer. 

That's not a very good idea, it's fine for testing, but you should use a balancer that supports sticky sessions (e.g. mod_cluster or mod_jk).

e have annotated our ejb's as @Clustered (though we understand this is deprecated).

That should be no harm, but you can get rid of it.

[stephen.lorenz]

Thank you for your reply.  Both servers should be running the same exact applications as we are running in domain mode.  But that brings up a good point that I have noticed since the original post; this seems to happen after performing a hot deploy of the war while running in domain mode.  It is as if the SLSBs are being serialized and deserialized on hot deploy.  Is that possible?  But many of these SLSB cannot be serialized as they contain references to an entity manager.  (I have tried declaring the entity manager field as transient but with no luck.)

However, the other thing that concerns me is your advice about not running our load balancing in round robin mode.  We are running our applications in Amazon Web Services using their Elastic Load Balancer (ELB).  The ELB supports stick sessions when you terminate your SSL on the ELB, which we are unable to do due to the sensitive medical information passing through our application.  I guess we could still add an apache server (as you recommend) but that seems to add one more point of failure for our HA application.  Are there known issues with running in round robin mode or is it just bad practice?

I am continuing to debug this issue by incrementally creating a new test application and trying to see if/when it breaks.

Thanks