NullPointerException in HttpSessionProvider (3.0-SNAPSHOT)

[nva]

Upgraded to 3.0-SNAPSHOT a few days ago and just had a chance to run our application for a few days. I found null pointer exceptions in the logs (full stack trace below). They happened in the middle of the night where there was no load, no clients and no scheduled jobs running. The effect was that the application hung, jetty didn't even serve the login page, only a full restart helped. I am not sure yet how to isolate and reproduce the problem, but perhaps the stacktrace provides some useful pointers.

I have 22 of these errors in last night's log, all happended within a period of about 5 minutes. Did not have any problems with Errai 2.4.

2013-09-02 02:35:20.454:WARN:oejs.ServletHandler:/dtt/in.48553-49991.erraiBus
java.lang.NullPointerException
        at org.jboss.errai.bus.server.HttpSessionProvider$HttpSessionWrapper.<init>(HttpSessionProvider.java:113)
        at org.jboss.errai.bus.server.HttpSessionProvider$SessionsContainer.createSession(HttpSessionProvider.java:79)
        at org.jboss.errai.bus.server.HttpSessionProvider.createOrGetSession(HttpSessionProvider.java:60)
        at org.jboss.errai.bus.server.HttpSessionProvider.createOrGetSession(HttpSessionProvider.java:42)
        at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.doGet(DefaultBlockingServlet.java:113)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:598)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:486)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.eclipse.jetty.server.Server.handle(Server.java:350)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
        at org.eclipse.jetty.server.AbstractHttpConnection.headerComplete(AbstractHttpConnection.java:890)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.headerComplete(AbstractHttpConnection.java:944)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:630)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:230)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
        at java.lang.Thread.run(Thread.java:722)

[edewit]

Hi Valentin,

This could be caused by the servlet that you are using, could you post your web.xml file?

Cheers,

     Erik Jan

[nva]

Hello Erik Jan, below the web.xml. Am using the DefaultBlockingServlet at the moment. Tried the Async several months ago and at that point it was agreed that the DefaultBlockingServlet is the better one to use. Has this changed in Errai 3.0?

The application is deployed into jetty 8.

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns="http://java.sun.com/xml/ns/javaee"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee
          http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
         metadata-complete="true"
         version="2.5">
   
    <context-param>
        <param-name>quartz:config-file</param-name>
        <param-value>quartz.properties</param-value>
    </context-param>
    <context-param>
        <param-name>quartz:shutdown-on-unload</param-name>
        <param-value>true</param-value>
    </context-param>
    <context-param>
        <param-name>quartz:wait-on-shutdown</param-name>
        <param-value>false</param-value>
    </context-param>
    <context-param>
        <param-name>quartz:start-scheduler-on-load</param-name>
        <param-value>true</param-value>
    </context-param>

    <listener>
         <listener-class>org.quartz.ee.servlet.QuartzInitializerListener</listener-class>
    </listener>
    <listener>
         <listener-class>com.nvcode.dtt.server.service.ContainerManagerService</listener-class>
    </listener>
    <listener>
         <listener-class>com.nvcode.dtt.server.service.HttpSessionListenerService</listener-class>
    </listener>
   
    <servlet>
        <servlet-name>ErraiServlet</servlet-name>
        <servlet-class>org.jboss.errai.bus.server.servlet.DefaultBlockingServlet</servlet-class>
        <init-param>
            <param-name>auto-discover-services</param-name>
            <param-value>true</param-value>
        </init-param>
        <load-on-startup>1</load-on-startup>
    </servlet>
   
    <servlet>
        <servlet-name>TotalValueServlet</servlet-name>
        <servlet-class>com.nvcode.dtt.server.servlet.TotalValueServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
   
    <servlet>
        <servlet-name>TokenFileDownloadServlet</servlet-name>
        <servlet-class>com.nvcode.dtt.server.servlet.TokenFileDownloadServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
   
    <servlet>
        <servlet-name>AbsoluteFileDownloadServlet</servlet-name>
        <servlet-class>com.nvcode.dtt.server.servlet.AbsoluteFileDownloadServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
   
    <servlet>
        <servlet-name>UploadServlet</servlet-name>
        <servlet-class>com.nvcode.dtt.server.servlet.FileUploadServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>

    <servlet-mapping>
        <servlet-name>ErraiServlet</servlet-name>
        <url-pattern>*.erraiBus</url-pattern>
    </servlet-mapping>
   
    <servlet-mapping>
        <servlet-name>TotalValueServlet</servlet-name>
        <url-pattern>totalValue</url-pattern>
    </servlet-mapping>
   
    <servlet-mapping>
        <servlet-name>TokenFileDownloadServlet</servlet-name>
        <url-pattern>tokendownload</url-pattern>
    </servlet-mapping>
   
    <servlet-mapping>
        <servlet-name>AbsoluteFileDownloadServlet</servlet-name>
        <url-pattern>filedownload</url-pattern>
    </servlet-mapping>
   
    <servlet-mapping>
        <servlet-name>UploadServlet</servlet-name>
        <url-pattern>upload</url-pattern>
    </servlet-mapping>
   
    <context-param>
        <param-name>login.config</param-name>
        <param-value>/WEB-INF/login.config</param-value>
    </context-param>

    <context-param>
        <param-name>users.properties</param-name>
        <param-value>/WEB-INF/users.properties</param-value>
    </context-param>

    <welcome-file-list>
        <welcome-file>dtt.html</welcome-file>
    </welcome-file-list>
   
    <session-config>
        <session-timeout>30</session-timeout>
    </session-config>
</web-app>

[edewit]

Hi Valentin,

Could you try to change to the StandardAsyncServlet?

Cheers,

     Erik Jan

[nva]

Hmm, this is what I added to the web.xml:

    <servlet>
        <servlet-name>ErraiServlet</servlet-name>
        <servlet-class>org.jboss.errai.bus.server.servlet.StandardAsyncServlet</servlet-class>
        <async-supported>true</async-supported>
    </servlet>

I also have the javax.servlet-api in the pom. In dev mode this is what I get, however:

[errai] wait for: org.jboss.errai.marshalling.client.api.MarshallerFramework
[errai] vote for: org.jboss.errai.marshalling.client.api.MarshallerFramework
[errai] bus initialization started ...
[errai] wait for: org.jboss.errai.bus.client.api.ClientMessageBus
[errai] IOC bootstrapper successfully initialized.
[errai] 10 beans successfully deployed.
[errai] bean manager now in service.
[errai] sending handshake message to remote bus
POST http://127.0.0.1:8888/out.7428-23773.erraiBus?z=0&clientId=7428-23773&wait=1&phase=connection 500 (org.mortbay.jetty.Request.startAsync()Ljavax/servlet/AsyncContext;) :8888/out.7428-23773.erraiBus?z=0&clientId=7428-23773&wait=1&phase=connection:1
[errai] attempting Tx reconnection -- attempt: 1

The bundled jetty doesn't seem to like the javax.servlet-api (3.1.0).

Tried the JettyContinuationsServlet too, got at least a login screen rendered, but the bus is not up. Of course, didn't change anything else, just the servlet definition.

[errai] wait for: org.jboss.errai.marshalling.client.api.MarshallerFramework

[errai] vote for: org.jboss.errai.marshalling.client.api.MarshallerFramework

[errai] bus initialization started ...

[errai] wait for: org.jboss.errai.bus.client.api.ClientMessageBus

[errai] IOC bootstrapper successfully initialized.

[errai] 10 beans successfully deployed.

[errai] bean manager now in service.

[errai] sending handshake message to remote bus

[errai] my queue session id: 1cbada5d18f68901ef08ac21d3e366e693289faa2edac96e4d572fa177d729d

[errai] transitioning to new handler: Long Polling

[errai] vote for: org.jboss.errai.bus.client.api.ClientMessageBus

[errai] bus federated and running.

no subscribers to deliver to for subject: com.nvcode.dtt.shared.PublicRPC:RPC -- Additional Details: <br/>org.jboss.errai.bus.client.api.base.NoSubscribersToDeliverTo: no subscribers to deliver to for subject: com.nvcode.dtt.shared.PublicRPC:RPC<br/>org.jboss.errai.bus.client.framework.ClientMessageBusImpl.send(ClientMessageBusImpl.java:781)<br/>  org.jboss.errai.bus.client.api.base.CommandMessage.sendNowWith(CommandMessage.java:343)<br/>  org.jboss.errai.bus.client.api.builder.DefaultRemoteCallBuilder$1.sendNowWith(DefaultRemoteCallBuilder.java:130)

[nva]

To summarise the status so far: I tried all the servlets described in the reference guide and only been able to get the DefaultBlockingServlet working in dev mode (GWT 2.5.1) with the bundled jetty.

[jfuerth]

Hi Valentin,

Your observations about the various ErraiServlet implementations match my understanding: only DefaultBlockingServlet and JettyContinuationsServlet will work with the version of Jetty that's embedded in GWT 2.5.1 Dev Mode. The fact that you ran into a different issue with JettyContinuationsServlet could mean that there's a bug in that servlet on the master (3.0-SNAPSHOT) branch, or it could be a different symptom of the same problem you're encountering when trying to use DefaultBlockingServlet.

The NullPointerException at HttpSessionProvider.java:113 means that either the Errai QueueID is null (unlikely), or that HttpSession.getId() from the Jetty-supplied HttpSession impl is returning null. I've made a small modification code to HttpSessionProvider so we'll be able to tell which scenario we're dealing with. Can you update to the new 3.0-SNAPSHOT (once Jenkins is done building and publishing) and wait for this issue to recur? The line number in the new stack trace should point us in the right direction.

-Jonathan

[nva]

Hello Jonathan,

thanks! Will do. I can do a build from git's head and deploy that.

Actually, I did a 'bleeding edge' build yesterday and now found a new error (below) in my jetty log. Is this relevant or shall I rather put it into a separate discussion? Just to reiterate, this was built from what was in git yesterday, not the version published in the maven repo. All unit tests completed successfully. I can see the problem in the code, but I don't know the motivation behind using an ordered collection there, so I rather not attempt to write a patch.

java.lang.ClassCastException: org.jboss.errai.bus.server.ServerMessageBusImpl$RemoteMessageCallback cannot be cast to java.lang.Comparable
        at java.util.concurrent.ConcurrentSkipListMap.comparable(ConcurrentSkipListMap.java:663)
        at java.util.concurrent.ConcurrentSkipListMap.doPut(ConcurrentSkipListMap.java:848)                                                                            
        at java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(ConcurrentSkipListMap.java:1881)
        at java.util.concurrent.ConcurrentSkipListSet.add(ConcurrentSkipListSet.java:232)
        at java.util.AbstractCollection.addAll(AbstractCollection.java:334)
        at org.jboss.errai.bus.server.DynamicDeliveryPlan.<init>(DynamicDeliveryPlan.java:32)
        at org.jboss.errai.bus.server.DynamicDeliveryPlan.newDynamicDeliveryPlan(DynamicDeliveryPlan.java:36)
        at org.jboss.errai.bus.server.DeliveryPlan.newDeliveryPlanWith(DeliveryPlan.java:83)
        at org.jboss.errai.bus.server.ServerMessageBusImpl.createOrAddDeliveryPlan(ServerMessageBusImpl.java:586)
        at org.jboss.errai.bus.server.ServerMessageBusImpl.remoteSubscribe(ServerMessageBusImpl.java:636)
        at org.jboss.errai.bus.server.ServerMessageBusImpl$ServerBusMessageCallback.callback(ServerMessageBusImpl.java:1042)
        at org.jboss.errai.bus.server.DeliveryPlan.deliver(DeliveryPlan.java:51)
        at org.jboss.errai.bus.server.ServerMessageBusImpl.sendGlobal(ServerMessageBusImpl.java:284)
        at org.jboss.errai.bus.server.SimpleDispatcher.dispatchGlobal(SimpleDispatcher.java:46)
        at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:92)
        at org.jboss.errai.bus.server.service.ErraiServiceImpl.store(ErraiServiceImpl.java:109)
        at org.jboss.errai.bus.server.servlet.DefaultBlockingServlet.doPost(DefaultBlockingServlet.java:139)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:647)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:728)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:598)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:486)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:119)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:233)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1065)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:413)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:192)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:999)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:117)
        at org.eclipse.jetty.server.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:250)
        at org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:149)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:111)
        at org.eclipse.jetty.server.Server.handle(Server.java:350)
        at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:454)
        at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:900)
        at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:954)
        at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:851)
        at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:235)
        at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:77)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:606)
        at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:46)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:603)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:538)
        at java.lang.Thread.run(Thread.java:722)

[jfuerth]

Hi Valentin,

This is fortunate timing.. I actually came across DynamicDeliveryPlan last week when I was doing some unrelated work on the ServerMessageBus. Here's how it works:

For a given bus subject, subscribers are tracked in a DeliveryPlan object, which in turn keeps references to subscribers in an array. Every addition or removal of a subscriber entails creating a new array, and copying all the references to it. I don't know why this DeliveryPlan "array list" implementation is better than just using ArrayList like we do everywhere else in Errai.

In any case, after 100 mutations to the subscriber list, the DeliveryPlan creates a DynamicDeliveryPlan and replaces itself with that in the ServerMessageBusImpl. I don't know why the DynamicDeliveryPlan uses the ConcurrentSkipListMap. That probably never worked; I doubt our test suite ever trips the changeover from the regular DeliveryPlan to the DynamicDeliveryPlan.

IMO, the best way to solve this would be to eliminate DynamicDeliveryPlan and to just use a data structure with a reasonable time/space tradeoff over a wide range of subscriber list sizes--probably good old ArrayList!

-Jonathan

[jfuerth]

I've removed DynamicDeliveryPlan from Errai 2.4 and 3.0. Apparently you were the first to find it, and when you did, it didn't work anyway.

[nva]

Thanks Jonathan, no more ClassCastExceptions now!

On the NullPointerException front: I no longer get it in the logs, however, the sympthom is still the same: after a period of time the whole of jetty hangs and it no longer serves anything. The only change is the upgrade from Errai 2.3 (sorry, I said 2.4 before, my mistake, I was using the latest stable) to 3.0. Am not sure where to go from here. I quite like the BatchCaller now that I got it working and also need the server-to-server communication, so preferred to stay on 3.0. Do you have any suggestions?

Cheers,

V.

[jfuerth]

The only thing I did on the NPE front was to add explicit null checks to the two arguments that could have caused the NPE you posted at the beginning of this thread.

If you're not getting any exceptions now, that means the problem must have changed. Which is unfortunate.

I'd suggest you get a full thread dump from the frozen Jetty VM. Maybe we'll see where its worker threads are stuck. You can do this on the console with Ctrl-\ or you can use the VisualVM tool that comes with the JDK.

-Jonathan

[nva]

Thanks Jonathan! Will do the dump and take things from there.

The jetty 'hanging' happened on jetty8. Upgraded to the latest stable version of jetty9 now. That eliminates one possibility as well.

Cheers,

V.