Stream Corrupted Exception on Server side

[vink]

I'm using Jboss 4.2.2.GA with Remoting 2.4.0.SP1. My product has multiple clients which are requesting SLSB. After several hours of running I'm getting following exception;

Please suggest.

Level
ERROR
Logger
org.jboss.remoting.transport.socket.ServerThread
Time
2008-12-24 19:58:31,921
Thread
WorkerThread#7[150.158.74.51:1586]
Message
WorkerThread#7[150.158.74.51:1586] exception occurred during first invocation
Properties
{{hostname,file}{application,central.log.31}{log4jid,1946}}
Throwable
java.lang.reflect.InvocationTargetException
at sun.reflect.GeneratedConstructorAccessor108.newInstance(Unknown Source)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
at org.jboss.remoting.transport.socket.ServerThread.createServerSocketWrapper(ServerThread.java:907)
at org.jboss.remoting.transport.socket.ServerThread.dorun(ServerThread.java:489)
at org.jboss.remoting.transport.socket.ServerThread.run(ServerThread.java:230)
Caused by: java.io.StreamCorruptedException: invalid stream header: 77011679
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:783)
at java.io.ObjectInputStream. (ObjectInputStream.java:280)
at org.jboss.remoting.loading.ObjectInputStreamWithClassLoader. (ObjectInputStreamWithClassLoader.java:97)
at org.jboss.remoting.serialization.impl.java.JavaSerializationManager.createInput(JavaSerializationManager.java:54)
at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:75)
at org.jboss.remoting.marshal.serializable.SerializableUnMarshaller.getMarshallingStream(SerializableUnMarshaller.java:58)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createInputStream(ClientSocketWrapper.java:179)
at org.jboss.remoting.transport.socket.ClientSocketWrapper.createStreams(ClientSocketWrapper.java:162)
at org.jboss.remoting.transport.socket.ClientSocketWrapper. (ClientSocketWrapper.java:66)
at org.jboss.remoting.transport.socket.ServerSocketWrapper. (ServerSocketWrapper.java:46)
... 6 more

[ron_sigal]

That doesn't look good. Do you see anything else in the server log? Do you have a log for the client side, and, if so, do you see anything going on at the same time you see the StreamCorruptedException on the server side?

For additional logging information, you can increase the server side log level for Remoting by adding, for example,

 <category name="org.jboss.remoting">
 <priority value="TRACE"/>
 </category>

to $JBOSS_HOME/server/$CONFIG/conf/jboss-log4j.xml.

[vink]

As suggested, i've set Remoting to TRACE level.

And, then I launched my clients (~32).
Observation;
- Server starts bouncing connections after few minutes of running
- The ping timedout for server completly after sometime.
- netstat tool shows many connections on 1098 & 1099 in TIME_WAIT state.

TCP 150.158.73.73:1099 150.158.72.87:3585 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.87:3628 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.124:1122 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.124:4901 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.124:4903 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.126:4798 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.126:4816 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.126:4820 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.149:2833 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.149:2903 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.158:1383 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.158:1523 TIME_WAIT
TCP 150.158.73.73:1099 150.158.72.158:1527 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4178 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4182 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4184 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4235 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4236 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4244 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4245 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4246 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4247 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4256 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4264 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4278 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4279 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4284 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4294 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4299 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4301 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4302 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4308 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4318 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4319 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4324 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4325 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4326 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4327 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4333 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.73:4414 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.84:2883 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.84:2948 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.101:1152 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.101:1155 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.101:1191 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.101:1197 TIME_WAIT
TCP 150.158.73.73:1099 150.158.73.101:1279 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.51:3653 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.51:3669 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.51:3675 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.51:3752 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.53:3306 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.53:3318 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.53:3321 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.53:3372 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.54:4703 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.54:4705 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.54:4805 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.113:3148 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.113:3158 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.117:2331 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.118:3305 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.118:3388 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.118:3395 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.118:3443 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.118:3448 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.118:3572 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.136:4404 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.136:4425 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.174:4491 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.174:4566 TIME_WAIT
TCP 150.158.73.73:1099 150.158.74.174:4667 TIME_WAIT
TCP 150.158.73.73:1099 150.158.75.156:3668 TIME_WAIT
TCP 150.158.73.73:4179 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4181 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4183 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4185 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4204 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4237 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4238 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4241 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4244 150.158.73.73:1099 TIME_WAIT
TCP 150.158.73.73:4248 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4249 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4250 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4251 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4252 150.158.73.73:3873 TIME_WAIT
TCP 150.158.73.73:4253 150.158.73.73:3873 TIME_WAIT
TCP 150.158.73.73:4260 150.158.73.73:3873 TIME_WAIT
TCP 150.158.73.73:4262 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4265 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4268 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4273 150.158.73.73:3873 TIME_WAIT
TCP 150.158.73.73:4280 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4283 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4285 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4288 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4295 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4297 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4303 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4304 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4305 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4307 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4309 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4311 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4317 150.158.73.73:3873 TIME_WAIT
TCP 150.158.73.73:4320 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4321 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4328 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4329 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4330 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4331 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4332 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4395 150.158.73.73:1098 ESTABLISHED
TCP 150.158.73.73:4415 150.158.73.73:1098 TIME_WAIT
TCP 150.158.73.73:4416 150.158.73.73:1098 ESTABLISHED

I've extra logs on my system, but I don't find a way to attach it with my reply. Pls suggest.

[ron_sigal]

"vink" wrote:

netstat tool shows many connections on 1098 & 1099 in TIME_WAIT state

Ports 1098 and 1099 are used by java RMI, not Remoting.

[ron_sigal]

I'm continuing the discussion about Vinay Kumar's (vink's) issues on the "Socket Read Timeout on stressing calls for validating produc" thread - http://www.jboss.com/index.html?module=bb&op=viewtopic&p=4199374#4199374.