Excessive UDP packes when serialising a session

[alt1]

Hi,

I have been looking at who JBoss serializes sessions and have noticed that when I place objects into the HttpSession that are not simple Java objects the UDP network traffic for a session increases >10 fold.

My setup includes a web application that has many classes in its "lib" directory. If I place one of these objects into the session and do a trace of the UDP packets I seem to not only get the object serialised itself but I also get the codebase of all the Jboss classes in the "tmp" deployment directory including classes found in the webapps lib. This dramatically increases the network overhead.

For example my session contains around 300 bytes of data and can be serialised and sent in a few network packets. However when tracing the UDP packets from JBosses serialisation it turns out that around 100 packets are sent with a total data size > 30K. This is for a single (small) session update!

I have traced this problem to the method

serializeSession() and unserializeSession()

in

ClusteredHTTPSessionBeanImpl.java

class. This seems to use MarshalledObject to wrap the session. I think this may be wrong since MarshalledObject states "classes are annotated with a codebase URL from where the class can be loaded (if available).."

It is this that causes the vast increase of network traffic for non simple-objects which have there codebase set. (Note I am using jboss-3.0.3_tomcat-4.0.5 but it also happens with jboss-3.0.3_tomcat-4.1.12 - which seem to have an codebase http server running at 8083- it does not happen when using Jetty but see below).

Is it necessary to use MarshalledObject wrapper? The comment in the code seems to indicate that there could be other ways of doing this?

Jetty/JBoss in 3.0.3 is broken and cannot deploy distributable webapps (see another thread in this forum). In 3.0.2 Jetty/JBoss only sends the serialised classes (but no code base) however it does send the session details on each modification of the session - which again produces poor network performance.

Andy.

[slaboure]

Which kind of codebase do you have? I should only have one URL in it.

I am curious to have an answer to this question while I agree that we don't need to serialize this information as part of the state.

I may take a look into this when I have time. I think someone already wrote a non-verbose marshaller, so it would be very easy to plug it in.

One of my next goals is to work again on the HttpSession clustering side to improve it.

[alt1]

Hi,

You are right the codebase is a single URL contained in the MarshalledObject object. However I have rebuild JBoss with an altered ClusteredHTTPSessionBeanImpl.java. It now writes the serialised stream to a filesystem object. This object turns out to be approx. 120K. The majority of the file is the complete classpath of the application replicated around 6 times. This includes items in tmp/deploy/server/all/deploy/ folder. The ones that have numbers pre-pended. Each element is space separated. It actually looks like a serialized array of Strings.

If I remove the MarshalledObject then the size drops right down to a few 100 bytes.

Everything then works OK (session replicated) until the first session expires. At which time the session object tries to be de-serialized (all tied up with the EJB lifecycle stuff). This fails since the Thread that is doing the session reaping does not have access to the classes contained in the serialized object and hence fails to load it. (Class loading failure!).

Oh dear :-(

[alt1]

Hi,

You are right the codebase is a single URL contained in the MarshalledObject object. However I have rebuild JBoss with an altered ClusteredHTTPSessionBeanImpl.java. It now writes the serialised stream to a filesystem object. This object turns out to be approx. 120K. The majority of the file is the complete classpath of the application replicated around 6 times. This includes items in tmp/deploy/server/all/deploy/ folder. The ones that have numbers pre-pended. Each element is space separated. It actually looks like a serialized array of Strings.

If I remove the MarshalledObject then the size drops right down to a few 100 bytes.

Everything then works OK (session replicated) until the first session expires. At which time the session object tries to be de-serialized (all tied up with the EJB lifecycle stuff). This fails since the Thread that is doing the session reaping does not have access to the classes contained in the serialized object and hence fails to load it. (Class loading failure!).

Oh dear :-(


(Reposted - not sure first post worked OK)

[slaboure]

Hello alt1,

That's interesting. What do you mean by "If I remove the MarshalledObject"?

And do you have the stacktrace of the classloading exception?

That's really interesting because it has two consequences:
1) if the size is soooo big, then we can really improve performance by replacing MO with something less verbose (easy)
2) if, when you remove MO, you get classloading exceptions, it *may* mean that it also fails now and the reason we don't receive an exception is that dynamic class downloading is used => VERY BAD in this case as it really impact the cluster performance.

If you can give me more information, I will work on this.

Cheers,


sacha

[alt1]

Hi,

The MarshalledObject, as I understand it, is only a "place holder" for an object. Looking at the code it is never used as a MarshalledObject only the fact that it implements the serializable interface. So all I did was just store and extract session object directly in "this.tmpSession". I have attached the very small patch that demos this.

I have also attached a stack trace when the session times out. This only occurs on timeout of the session.

Note that with the patch applied I have not tested that another member of the cluster can actually deserialize ithe session objects, but I have observed the serialized object on the network.

The patch will only work if the thread that does the deserialization has all the required classes - if not the bad situation of dynamic class downloading you described would occur for every serialization - in which case the patch will cause bad stack dumps other members of the cluster. oooppss.

Andy.

[slaboure]

OK, I see what you meant by removing MO. In fact, I use a MO for a simple reason: as long as you use sticky sessions, the replicated session is received by other nodes but *never* unmarshalled until it is really needed (this, to save CPU time creating potentially lots of small objects)

Thus, I will replace the MO by another, less verbose MO but I will still use it to save CPU.

Do you have more information WRT to "if the thread that does the deserialization has all the required classes"? If your applications is deployed on all replicated nodes, that should automatically work thanks to the UCL. Is that the case? I guess you got that CNF exception because you hadn't deployed the applications on all nodes, correct? In this case, the same exception would have occured with the MO but, as we use late-deserialization, only when the node would have tried to unserialize the session, not before (which means NEVER if the web application wasn't deployed on this node)

[alt1]

Hi,

The stack trace was obtained from the same machine that the app was running on. In fact I had only one instance of the JBoss/JVM running. I could tell it was trying to do the session replication because of the multicast network traffic.

"if the thread that does the deserialization has all the required classes" - I was thinking here that if the session timeout thread was not a part of any particular application i.e one was used to timeout all/any sessions running in a container, then it is possible that its class loaders would not have access to the apps classes. I must admit that I don't known fully how the Class Loaders work in JBoss but I have alot of experience in writing and using class loaders else where & can guess where things fall apart (I've been there!).

Since I don't get this stack trace when using the MO It maybe that even the same JVM is using the dynamic class loading stuff when timing sessions out???