NPE with SFSB in cluster

[valsaraj007]

I am getting following error when I checked wildfly-8.2.0 cluster of 2 nodes with Clusterbench ejbservlet call:

2015-12-01 15:53:31,894 ERROR [io.undertow.request] (default task-19) UT005023: Exception handling request to /clusterbench/ejbservlet: java.lang.NullPointerException

  at org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler.readResolve(EnterpriseBeanProxyMethodHandler.java:161) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_07]

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_07]

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_07]

  at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_07]

  at org.jboss.marshalling.reflect.SerializableClass.callReadResolve(SerializableClass.java:413)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1287)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)

  at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1746)

  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1659)

  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1607)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1286)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)

  at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1746)

  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1659)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1286)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)

  at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1746)

  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1659)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1286)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:224)

  at org.jboss.marshalling.river.RiverUnmarshaller.readFields(RiverUnmarshaller.java:1746)

  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1659)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1286)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)

  at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:149)

  at org.jboss.marshalling.river.BlockUnmarshaller.readObject(BlockUnmarshaller.java:135)

  at org.jboss.marshalling.MarshallerObjectInputStream.readObjectOverride(MarshallerObjectInputStream.java:53)

  at org.jboss.marshalling.river.RiverObjectInputStream.readObjectOverride(RiverObjectInputStream.java:307)

  at java.io.ObjectInputStream.readObject(ObjectInputStream.java:363) [rt.jar:1.7.0_07]

  at java.util.HashMap.readObject(HashMap.java:1155) [rt.jar:1.7.0_07]

  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [rt.jar:1.7.0_07]

  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) [rt.jar:1.7.0_07]

  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) [rt.jar:1.7.0_07]

  at java.lang.reflect.Method.invoke(Method.java:601) [rt.jar:1.7.0_07]

  at org.jboss.marshalling.reflect.SerializableClass.callReadObject(SerializableClass.java:307)

  at org.jboss.marshalling.river.RiverUnmarshaller.doInitSerializable(RiverUnmarshaller.java:1638)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadNewObject(RiverUnmarshaller.java:1286)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:276)

  at org.jboss.marshalling.river.RiverUnmarshaller.doReadObject(RiverUnmarshaller.java:209)

  at org.jboss.marshalling.AbstractObjectInput.readObject(AbstractObjectInput.java:41)

  at org.jboss.as.clustering.marshalling.SimpleMarshalledValue.get(SimpleMarshalledValue.java:101)

  at org.jboss.as.clustering.marshalling.SimpleMarshalledValue.get(SimpleMarshalledValue.java:45)

  at org.wildfly.clustering.web.infinispan.session.MarshalledValueSessionAttributeMarshaller.read(MarshalledValueSessionAttributeMarshaller.java:46)

  at org.wildfly.clustering.web.infinispan.session.MarshalledValueSessionAttributeMarshaller.read(MarshalledValueSessionAttributeMarshaller.java:33)

  at org.wildfly.clustering.web.infinispan.session.coarse.CoarseImmutableSessionAttributes.getAttributes(CoarseImmutableSessionAttributes.java:46)

  at org.wildfly.clustering.web.infinispan.session.coarse.CoarseImmutableSessionAttributes.getAttributeNames(CoarseImmutableSessionAttributes.java:51)

  at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findListeners(InfinispanSessionManager.java:319)

  at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.triggerPostActivationEvents(InfinispanSessionManager.java:308)

  at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:163)

  at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)

  at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:688) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.servlet.spec.HttpServletRequestImpl.getSession(HttpServletRequestImpl.java:364) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at org.jboss.weld.servlet.SessionHolder.requestInitialized(SessionHolder.java:47) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]

  at org.jboss.weld.servlet.HttpContextLifecycle.requestInitialized(HttpContextLifecycle.java:212) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]

  at org.jboss.weld.servlet.WeldInitialListener.requestInitialized(WeldInitialListener.java:160) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]

  at io.undertow.servlet.core.ApplicationListeners.requestInitialized(ApplicationListeners.java:216) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:260) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_07]

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_07]

  at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_07]

Caused by: an exception which occurred:

  in object of type org.jboss.weld.bean.proxy.EnterpriseBeanProxyMethodHandler

  in field methodHandler

  in object of type org.jboss.weld.bean.proxy.InjectionPointPropagatingEnterpriseTargetBeanInstance

  in field beanInstance

  in object of type org.jboss.weld.bean.proxy.ProxyMethodHandler

  in field methodHandler

  in object of type org.jboss.test.clusterbench.ejb.stateful.LocalStatefulSB$Proxy$_$$_Weld$EnterpriseProxy$

  in field instance

  in object of type org.jboss.weld.context.SerializableContextualInstanceImpl

  in object of type java.util.HashMap

2015-12-01 15:53:31,908 ERROR [io.undertow.servlet.request] (default task-19) UT015005: Error invoking method requestDestroyed on listener class org.jboss.weld.servlet.WeldInitialListener: java.lang.NullPointerException

  at org.jboss.weld.context.AbstractBoundContext.deactivate(AbstractBoundContext.java:71) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]

  at org.jboss.weld.context.http.HttpRequestContextImpl.deactivate(HttpRequestContextImpl.java:72) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]

  at org.jboss.weld.servlet.HttpContextLifecycle.requestDestroyed(HttpContextLifecycle.java:282) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]

  at org.jboss.weld.servlet.WeldInitialListener.requestDestroyed(WeldInitialListener.java:143) [weld-core-impl-2.2.6.Final.jar:2014-10-03 10:05]

  at io.undertow.servlet.core.ApplicationListeners.requestDestroyed(ApplicationListeners.java:225) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:304) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:247) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:76) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:166) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.server.Connectors.executeRootHandler(Connectors.java:197) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

  at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:759) [undertow-core-1.1.0.Final.jar:1.1.0.Final]

  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_07]

  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_07]

  at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_07]

Any suggestions regarding this?

[pferraro]

Are you able to reproduce the issue against WF 9.0.x?

[valsaraj007]

Not yet migrated to wildfly 9. Is this fixed in 9?

[lprimak]

I believe this is a Weld bug. 

At least as of Weld 2.2.16 there are issues deserializing objects in a cluster, especially when upgrading apps.

I am trying to debug this as we speak, maybe I'll find the cause.

[lprimak]

This patch to Weld: rolling upgrades... catch all exceptions (i.e. NPE) in readResolve so… · flowlogix/core@7bdbb45 · GitHub

will ignore the error and start a new session for the user instead of having a fatal error in the app

[lprimak]

Correction... pull request: https://github.com/weld/core/pull/1268

[valsaraj007]

Thank you!

How can I apply this patch to existing wildfly-8.2? I found there is section to apply patch in wildfly admin console but never tried it. In the above link, I saw code changed but there is nothing like download. Can you please suggest how to apply the patch to 8.2?

[lprimak]

I run GlassFish, so I don't know how to apply this patch to WildFly, but my guess is to just build weld from source (follow the instructions)

and then replace the appropriate weld JAR inside WildFly distribution.

This patch will get you the login screen as opposed to a request failure.  I think there is something deeper going on in your situation,

which this patch will not solve.

It's not the best practice to store SFSB inside the web session.  Better way is to use @SessionScoped beans.

Unless you are using extended PersistentContext, there is really no difference between the two (if you use @Transactional annotation)

I think you may have better luck then. 

[mkouba]

Hi Valsaraj, Lenny,

I like the fix except that I don't understand why InvalidObjectException should cause an HTTP session invalidation. Is it defined/described somewhere? I haven't found anything and although it seems quite reasonable it's probably implementation-specific. Also note that even with the fix we would not get rid of the error. We've also found WFLY-5208 which seems to have similar symptoms. We're going to investigate the case more in depth.

[manovotn]

Hi,

could you please be more specific about the environment, app and setup you use?

I tried to reproduce the issue, but failed. It all seems to work.

Here is a breakdown of what I did:

I used the app from this repo. I buildt it in EE 6 and EE 7 versions (and tried both).

I downloaded clean Wildfly 8.2 and set up a cluster of two nodes (only changing offset for secondary node).

Then I deployed the app to both nodes and accessed the servlet with /clusterbench/ejbservlet

Everything worked as expected and the servlet responded correctly.

Could you, please, give me more details on how to reproduce this issue?

[lprimak]

Martin,

If you look at the exception stack, this is happening during deserialization of session replication:

...

at org.wildfly.clustering.web.infinispan.session.InfinispanSessionManager.findSession(InfinispanSessionManager.java:163)

  at org.wildfly.clustering.web.undertow.session.DistributableSessionManager.getSession(DistributableSessionManager.java:115)

  at io.undertow.servlet.spec.ServletContextImpl.getSession(ServletContextImpl.java:688) [undertow-servlet-1.1.0.Final.jar:1.1.0.Final]

...

The InvalidObjectException causes session deserialization to fail, and thus causes the current session to be unavailable.  The default action for this in most cases is to start a new session, or in the user's perspective it's most likely to send the user to a login page.

In the current implementation, this causes 500 error or worse, just empty request.

This behavior is the same in both WildFly and GlassFish.

[manovotn]

Lenny Primak wrote:

The InvalidObjectException causes session deserialization to fail, and thus causes the current session to be unavailable. The default action for this in most cases is to start a new session, ...

Do you have a link to some spec/documentation, which would explicitly state this behavior? I mean, it would make perfect sense, but it doesn't seem to be defined anywhere (at least I haven't managed to pinpoint it) and is therefore implementation specific.

[lprimak]

I don't know anywhere in any specs where it says it, but that doesn't mean it doesn't.

We can call it "de facto" standard

[manovotn]

I tested this behaviour with Wildfly and it does work as you described.

I took a cluster of two nodes and overriden the readResolve method to throw the exception.

Then I deployed it and forced session replication which leads to this exception.

I was immediately given a new session.

This means we can probably take it as de-facto standard.