-
1. Comet HTTPEventServlet HttpResponse/sockets messed up
cshear Jan 24, 2011 9:14 PM (in response to cshear)By the way, I also changed my servlet to use only HttpServlet, not implementing HttpEventServlet just to see if this is caused by HttpEventServlet, and I experienced the same problem - HttpServletResponse got messed up - msg to ClientX goes to ClientY. So something with Jbossweb is causing this issue.
-
2. Comet HTTPEventServlet HttpResponse/sockets messed up
jfclere Jan 25, 2011 2:37 AM (in response to cshear)hm you have a small sevlet to reproduce the issue?
-
3. Comet HTTPEventServlet HttpResponse/sockets messed up
cshear Feb 10, 2011 1:24 PM (in response to jfclere)I kinda found the cause for this problem, but don't know a solution yet.
The problem seems to be that our smart phone client for whatever reason occassionally sends TCP.RST (TCP reset msg) msg to the JBoss AS server (jboss web i.e. tomcat), and reset the server socket. But somehow JBoss server does not notify such an event to the application level (our servlet), so our application still holds the HttpServletResponse object and keeps writing stuff to the same object, yet the jboss server knows the connection is reset and reuse the HttpServletResponse object for other client connections, this causes messages destined for Client A may be delivered to Client B.
I traced the HttpServletResponse.getWriter().println() to tomcat code in jbossweb and found sometimes the error code is true and the code simply returned without throwing any exception to the caller, so the caller never knows there was a problem writing stuff to the network, yet there were no data on the wire for client to receive.
I am very shocked that a simple TCP.RST msg at the tcp network level will cripple JBoss/Tomcat server.
Right now we try to send a keep-alive mesage every 30 seconds to server so that TCP.RST msg won't be triggered to the server. But this is not a good and scalable solution.
Any thoughts?
-
4. Comet HTTPEventServlet HttpResponse/sockets messed up
cshear Feb 11, 2011 6:56 PM (in response to cshear)My Dev platform is win7, deployment platform is ubuntu. It happens on both.
Our client is android sending http requests.
-
5. Comet HTTPEventServlet HttpResponse/sockets messed up
gsarno Mar 31, 2011 6:25 AM (in response to cshear)Hello,
is there an update on this issue ?
Thanks.
-
6. Comet HTTPEventServlet HttpResponse/sockets messed up
jfclere Apr 1, 2011 3:13 AM (in response to gsarno)I am still waiting for a sample application sample... I think the application doen't handle errors correctly.
-
7. Comet HTTPEventServlet HttpResponse/sockets messed up
cshear Apr 1, 2011 2:32 PM (in response to jfclere)If you can send TCP.RST msg to the jboss web server, you will notice that this will cause httpResponse.getWriter().println() to not receive any exception, and it seems to write successfully, but the there is nothing written to the wire. Also, if you do TCP.RST often enough, you will get the sockets written messed up because of the re-use of the underlying socket connections.
Because the application does not get exceptions from the underlying application server for the TCP.RST msgs, the application has no way to handle this error scenario. So I think the underlying server(tomcat, or jbossweb) need to propagate this error condition to the applications.
-
8. Comet HTTPEventServlet HttpResponse/sockets messed up
jfclere Apr 4, 2011 2:53 AM (in response to cshear)Again don't except httpResponse.getWriter().println() to return an error (that won't happend until the JVM + OS buffers are full) but wait for an ERROR event.
-
9. Comet HTTPEventServlet HttpResponse/sockets messed up
cshear Apr 4, 2011 3:03 PM (in response to jfclere)Our appplication does handle the ERROR event, by removing the connection from the HTTPResponse object so it won't write to it anymore, but at that time, damage has already been done. Handling ERROR event does not seem to help the sockets messed up situation.
-
10. Comet HTTPEventServlet HttpResponse/sockets messed up
jfclere Apr 5, 2011 3:47 AM (in response to cshear)but at that time, damage has already been done. Handling ERROR event does not seem to help the sockets messed up situation.
It seems to me that your println() are blocking and you don't see the ERROR when it happens...
-
11. Comet HTTPEventServlet HttpResponse/sockets messed up
cshear Apr 5, 2011 2:18 PM (in response to jfclere)Our application basically follows the same code in the jbossweb Advanced IO example: http://docs.jboss.org/jbossweb/2.1.x/aio.html, where we do a printtln() followed by a flush(). I assume this is a blocking write. If this is not the correct way of using event servlet, what is the correct way? I searched the whole web and I'm unable to find a good tutorial that we can rely on without this critical issues. Can developers of jbossweb come up with solid examples/tutorials that exercises all the APIs that shows how to use them and synchronize them? Critical issues like this may adversely impact adoption of jbossweb for the rest of developer community. It needs to be made simpler to use, as a rule of thumb, when developing new technologies.
You see what I mean by reading through all the potential warnings from jbossweb advanced IO page. It seems full of landmines:
"The Servlet container will never, under any circumstance, invoke the event method of a Servlet with an event pertaining to a particular connection concurrently. For example, if a READ event is being processed, and some asynchronous thread closes the event using event.close(), an END event will only be called after the Servlet has completed processing of the READ event. However, the application must still synchronize any concurrent access it makes to an event or any objects provided by the Servlet API. The event object remains allocated for the entire duration of the connection, and can safely be used for synchronization purposes. For example, two possibly concurrent writes done on the same connection should likely synchronize on the event object.
When the processing of the END or ERROR event is done, the connection may be closed, and no additional data should be written on the connection. Unsynchronized asynchronous writes may cause stability issues when this happens. It is valid however to add the proper synchronization in the END or ERROR event processing (so that they wait for the asynchronous write loop to get done rather than synchronizing all writes). "
synchronized (connections) {
String[] pendingMessages = null;
synchronized (messages) {
pendingMessages = messages.toArray(new String[0]);
messages.clear();
}
// Send any pending message on all the open connections
// If a connection backlogs, there will be an exception
for (int i = 0; i < connections.size(); i++) {
try {
PrintWriter writer = connections.get(i).getWriter();
for (int j = 0; j < pendingMessages.length; j++) {
writer.println(pendingMessages[j] + "<br>");
}
writer.flush();
} catch (IOException e) {
log("IOExeption sending message", e);
}
}
}
-
12. Comet HTTPEventServlet HttpResponse/sockets messed up
jfclere Apr 6, 2011 4:03 AM (in response to cshear)Hm examples are examples... Fill free to contribute a better one.
While reading http://docs.jboss.org/jbossweb/2.1.x/aio.html you may see the following:
"The isWriteReady() method returns true when data may be written to the connection without blocking (the flag becomes false when the client is unable to accept data fast enough). When the flag becomes false, the servlet must stop writing data...."