12 Replies Latest reply on Apr 6, 2011 4:03 AM by jfclere

    Comet HTTPEventServlet HttpResponse/sockets messed up

    cshear

      I'm using JBoss AS 5.1GA. My servlet implements HttpEventServlet, and I implemented event() method, I followed similarly the sample code to  implement comet servlets at http://docs.jboss.org/jbossweb/3.0.x/aio.html, holding on HttpServletResponse objects for long HTTP GET requests and write msgs back to these HttpServletResponse objects when msgs are available.

       

      Here is my problem:

       

      After serveral clients login/logout/session timeout/invalidations, the server seems to mess up the HttpServletResponse objects/sockets: msg that is supposed to send to Client X is written to Client Y, via HttpServletResponse.getWriter().println() and subsequent flush(). And sometimes the msg are not delivered at all on the wire, because we can see wireshark does not show the data traffic for the client IP address. If client logs out and logs back in, then the msg can be received again for some time, and then it will fail again.

       

      I tried to debug the problem with jbossweb source code but unable to figure out why these HttpServletResponse objects/sockets appear to be messed up.

       

      I wonder if anyone experience similar problems with jbossweb, and could shed some light on this and let me know any workaround if any. I'm getting to a point where I have no way to go, and looking to abandon using jboss if these problems could not be solved. Your help is greatly appreciated.

        • 1. Comet HTTPEventServlet HttpResponse/sockets messed up
          cshear

          By the way, I also changed my servlet to use only HttpServlet, not implementing HttpEventServlet just to see if this is caused by HttpEventServlet, and I experienced the same problem - HttpServletResponse got messed up - msg to ClientX goes to ClientY. So something with Jbossweb is causing this issue.

          • 2. Comet HTTPEventServlet HttpResponse/sockets messed up
            jfclere

            hm you have a small sevlet to reproduce the issue?

            • 3. Comet HTTPEventServlet HttpResponse/sockets messed up
              cshear

              I kinda found the cause for this problem, but don't know a solution yet.

               

              The problem seems to be that our smart phone client for whatever reason occassionally sends TCP.RST (TCP reset msg) msg to the JBoss AS server (jboss web i.e. tomcat), and reset the server socket. But somehow JBoss server does not notify such an event to the application level (our servlet), so our application still holds the HttpServletResponse object and keeps writing stuff to the same object, yet the jboss server knows the connection is reset and reuse the HttpServletResponse object for other client connections, this causes messages destined for Client A may be delivered to Client B.

               

              I traced the HttpServletResponse.getWriter().println() to tomcat code in jbossweb and found sometimes the error code is true and the code simply returned without throwing any exception to the caller,  so the caller never knows there was a problem writing stuff to the network, yet there were no data on the wire for client to receive.

               

              I am very shocked that a simple TCP.RST msg at the tcp network level will cripple JBoss/Tomcat server.

               

              Right now we try to send a keep-alive mesage every 30 seconds to server so that TCP.RST msg won't be triggered to the server. But this is not a good and scalable solution.

               

              Any thoughts?

              • 4. Comet HTTPEventServlet HttpResponse/sockets messed up
                cshear

                My Dev platform is win7, deployment platform is ubuntu. It happens on both.

                Our client is android sending http requests.

                • 5. Comet HTTPEventServlet HttpResponse/sockets messed up
                  gsarno

                  Hello,

                  is there an update on this issue ?

                   

                  Thanks.

                  • 6. Comet HTTPEventServlet HttpResponse/sockets messed up
                    jfclere

                    I am still waiting for a sample application sample... I think the application doen't handle errors correctly.

                    • 7. Comet HTTPEventServlet HttpResponse/sockets messed up
                      cshear

                      If you can send TCP.RST msg to the jboss web server, you will notice that this will cause httpResponse.getWriter().println() to not receive any exception, and it seems to write successfully, but the there is nothing written to the wire. Also, if you do TCP.RST often enough, you will get the sockets written messed up because of the re-use of the underlying socket connections.

                       

                      Because the application does not get exceptions from the underlying application server for the TCP.RST msgs, the application has no way to handle this error scenario. So I think the underlying server(tomcat, or jbossweb) need to propagate this error condition to the applications.

                      • 8. Comet HTTPEventServlet HttpResponse/sockets messed up
                        jfclere

                        Again don't except httpResponse.getWriter().println() to return an error (that won't happend until the JVM + OS buffers are full) but wait for an ERROR event.

                        • 9. Comet HTTPEventServlet HttpResponse/sockets messed up
                          cshear

                          Our appplication does handle the ERROR event, by removing the connection from the HTTPResponse object so it won't write to it anymore, but at that time, damage has already been done. Handling ERROR event does not seem to help the sockets messed up situation.

                          • 10. Comet HTTPEventServlet HttpResponse/sockets messed up
                            jfclere

                            but at that time, damage has already been done. Handling ERROR event does not seem to help the sockets messed up situation.

                            It seems to me that your println() are blocking and you don't see the ERROR when it happens...

                            • 11. Comet HTTPEventServlet HttpResponse/sockets messed up
                              cshear

                              Our application basically follows the same code in the jbossweb Advanced IO example: http://docs.jboss.org/jbossweb/2.1.x/aio.html, where we do a printtln() followed by a flush(). I assume this is a blocking write. If this is not the correct way of using event servlet, what is the correct way? I searched the whole web and I'm unable to find a good tutorial that we can rely on without this critical issues. Can developers of jbossweb come up with solid examples/tutorials that exercises all the APIs that shows how to use them and synchronize them? Critical issues like this may adversely impact adoption of jbossweb for the rest of developer community. It needs to be made simpler to use, as a rule of thumb, when developing new technologies.

                               

                              You see what I mean by reading through all the potential warnings from jbossweb advanced IO page. It seems full of landmines:

                               

                              "The Servlet container will never, under any circumstance, invoke the event method of a Servlet    with an event pertaining to a particular connection concurrently. For example, if    a READ event is being processed, and some asynchronous thread closes the event using event.close(),    an END event will only be called after the Servlet has completed processing of the READ event.    However, the application must still synchronize any concurrent access it makes to an event or any    objects provided by the Servlet API. The event object remains allocated for the entire duration    of the connection, and can safely be used for synchronization purposes. For example, two possibly     concurrent writes done on the same connection should likely synchronize on the event object.  

                                 

                                  When the processing of the END or ERROR event is done, the connection may be closed, and no    additional data should be written on the connection. Unsynchronized asynchronous writes may    cause stability issues when this happens. It is valid however to add the proper synchronization    in the END or ERROR event processing (so that they wait for the asynchronous write loop to get    done rather than synchronizing all writes).  "

                               

                               

                              synchronized (connections) {

                                                  String[] pendingMessages = null;

                                                  synchronized (messages) {

                                                      pendingMessages = messages.toArray(new String[0]);

                                                      messages.clear();

                                                  }

                                                  // Send any pending message on all the open connections

                                                  // If a connection backlogs, there will be an exception

                                                  for (int i = 0; i < connections.size(); i++) {

                                                      try {

                                                          PrintWriter writer = connections.get(i).getWriter();

                                                          for (int j = 0; j < pendingMessages.length; j++) {

                                                              writer.println(pendingMessages[j] + "<br>");

                                                          }

                                                          writer.flush();

                                                      } catch (IOException e) {

                                                          log("IOExeption sending message", e);

                                                      }

                                                  }

                                              }

                              • 12. Comet HTTPEventServlet HttpResponse/sockets messed up
                                jfclere

                                Hm examples are examples... Fill free to contribute a better one.

                                 

                                While reading http://docs.jboss.org/jbossweb/2.1.x/aio.html you may see the following:

                                "The isWriteReady() method returns true when data may be written to the connection without      blocking (the flag becomes false       when the client is unable to accept data fast enough). When the flag becomes false,       the servlet must stop writing data...."