java.io.NotSerializableException: java.lang.Object -Cluster

[catchupvijay]

Hello,

We are using JbossCache in a clustered environment. (JBoss, Ibatis, Oracle). We are unable to figure out the reason for an exception when the JBossCacheService tries to do replication on the other node.

[java.io.NotSerializableException: java.lang.Object]
We have ensured that all the objects stored in Collections implement the Serializable interface. Following is the detailed stack trace.
--------------------------------------------------------------------------------------

09:57:47,393 INFO [STDOUT] java.io.NotSerializableException: java.lang.Object
09:57:47,393 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1054)
09:57:47,393 INFO [STDOUT] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1332)
09:57:47,393 INFO [STDOUT] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1304)
09:57:47,393 INFO [STDOUT] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1332)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1304)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1332)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1304)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1332)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1304)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:278)
09:57:47,409 INFO [STDOUT] at java.util.ArrayList.writeObject(ArrayList.java:529)
09:57:47,409 INFO [STDOUT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
09:57:47,409 INFO [STDOUT] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
09:57:47,409 INFO [STDOUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
09:57:47,409 INFO [STDOUT] at java.lang.reflect.Method.invoke(Method.java:324)
09:57:47,409 INFO [STDOUT] at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:809)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1296)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:278)
09:57:47,409 INFO [STDOUT] at java.util.HashMap.writeObject(HashMap.java:978)
09:57:47,409 INFO [STDOUT] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
09:57:47,409 INFO [STDOUT] at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
09:57:47,409 INFO [STDOUT] at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
09:57:47,409 INFO [STDOUT] at java.lang.reflect.Method.invoke(Method.java:324)
09:57:47,409 INFO [STDOUT] at java.io.ObjectStreamClass.invokeWriteObject(ObjectStreamClass.java:809)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1296)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1247)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1052)
09:57:47,409 INFO [STDOUT] at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:278)
09:57:47,409 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.SessionBasedClusteredSession.writeExternal(SessionBasedClusteredSession.java:288)
09:57:47,409 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheService.externalizeSession(JBossCacheService.java:771)
09:57:47,409 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheService.putSession(JBossCacheService.java:229)
09:57:47,409 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.SessionBasedClusteredSession.processSessionRepl(SessionBasedClusteredSession.java:165)
09:57:47,409 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheManager.processSessionRepl(JBossCacheManager.java:606)
09:57:47,409 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.JBossCacheManager.storeSession(JBossCacheManager.java:375)
09:57:47,409 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.InstantSnapshotManager.snapshot(InstantSnapshotManager.java:38)
09:57:47,409 INFO [STDOUT] at org.jboss.web.tomcat.tc5.session.ClusteredSessionValve.invoke(ClusteredSessionValve.java:91)09:57:47,409 INFO [STDOUT] at org.jboss.web.tomcat.security.JaccContextValve.invoke(JaccContextValve.java:59)
09:57:47,409 INFO [STDOUT] at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:126)
09:57:47,409 INFO [STDOUT] at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:105)
09:57:47,409 INFO [STDOUT] at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:107)
09:57:47,409 INFO [STDOUT] at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:148)
09:57:47,409 INFO [STDOUT] at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:856)
09:57:47,409 INFO [STDOUT] at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.processConnection(Http11Protocol.java:744)
09:57:47,409 INFO [STDOUT] at org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:527)
09:57:47,409 INFO [STDOUT] at org.apache.tomcat.util.net.MasterSlaveWorkerThread.run(MasterSlaveWorkerThread.java:112)
09:57:47,409 INFO [STDOUT] at java.lang.Thread.run(Thread.java:534)

[brian.stansberry]

Somewhere in the object graph is an instance of java.lang.Object, which does not implement Serializable. Looks like the ref is held by a fairly deep object graph rooted in an ArrayList; the ArrayList is object stored as a session attribute.

[catchupvijay]

We investigated the application code and Ibatis sources.

1. All the application classes (Beans, Actionforms, Decorators) are serializable. (This takes care of all the objects in chain)

2. We suspected that Ibatis could be problem area and made all the classes serializable (For sanity sake)

Yet the java.io.NotSerializableException: java.lang.Object is seen.
Its becomming impossible to understand what could be the reason for this.

Help Appreciated.

[brian.stansberry]

The problem isn't that any application class or iBatis class isn't Serializable. It's that one of the fields of one of those classes is of type java.lang.Object. That field is not Serializable, because java.lang.Object isn't. You need to find that field.

[catchupvijay]

Well, I think this has got to do with the way Ibatis is implemented. Here's an example.

private List runQueryForList(SqlStatement localSql, Connection conn, Object parameterObject, int skipResults, int maxResults, RowHandler rowHandler) throws SQLException
{
List list;
if(log.isDebugEnabled())
{
log.debug("executeQueryForList: " + name);
conn = ConnectionLogProxy.newInstance(conn);
}
ErrorField errorField = new ErrorField();
if(rowHandler == null)
list = new ArrayList();
else
list = null;
ResultSet rs = null;
PreparedStatement ps = null;
String errorMessage = "Check the SQL statement.";
try
{
sqlMap.incrementExecutionThrottle(conn);
ps = createStatement(conn, localSql, parameterObject);
if(sqlMap.isDriverHintsEnabled() && maxResults != -1 && skipResults != -1)
{
ps.setFetchSize(maxResults + skipResults);
ps.setMaxRows(maxResults + skipResults);
}
errorMessage = "Check the Parameter Map (or inline parameters).";
applyParameterMap(localSql, ps, parameterObject, errorField);
errorField.errorField = null;
errorMessage = "Check the SQL statement.";
rs = ps.executeQuery();
for(int i = 0; i < skipResults; i++)
if(!rs.next())
break;

errorMessage = "Check the Result Map.";
int n = 0;
if(rowHandler == null)
{
for(; (maxResults == -1 || n < maxResults) && rs.next(); n++)
{
Object object = applyResultMap(conn, rs, null, errorField);
list.add(object);
}

errorField.errorField = null;
} else
{
for(; (maxResults == -1 || n < maxResults) && rs.next(); n++)
{
Object object = applyResultMap(conn, rs, null, errorField);
rowHandler.handleRow(object);
}

errorField.errorField = null;
}
}
catch(Throwable t)
{
Throwable tt = unwrapProxyException(t);
String msg = "";
if(errorField.errorField != null)
msg = "Error executing '" + name + "' in '" + resourceName + "'. " + errorMessage + " Check the '" + errorField + "' property. Cause: " + t;
else
msg = "Error executing '" + name + "' in '" + resourceName + "'. " + errorMessage + " Cause: " + t;
if(log.isErrorEnabled())
log.error(msg, t.fillInStackTrace());
if(t instanceof SQLException)
throw (SQLException)t;
else
throw new SQLException(msg);
}
finally
{
closeResultSet(rs);
closeStatement(ps);
sqlMap.decrementExecutionThrottle(conn);
}
notifyListeners();
return list;
}

BELOW IS THE applyResultMap METHOD

private Object applyResultMap(Connection conn, ResultSet rs, Object resultObject, ErrorField errorField)
throws SQLException
{ Object object = resultObject;
String typeName = null;
if(getResultMapName() != null)
{
ResultMap map = getSqlMap().getResultMap(getResultMapName());
typeName = map.getClassName();
if(object == null)
object = instantiate(map.getClassName());
ResultMapping mapping;
for(Iterator names = map.getMappedPropertyNames(); names.hasNext(); setBeanProperty(mapping, object, conn, rs, errorField))
{
String propertyName = (String)names.next();
mapping = map.getResultMapping(propertyName);
}

} else
if(resultClass != null)
{
typeName = resultClass;
if(object == null)
object = instantiate(resultClass);
if(object instanceof BaseValue)
{
ResultMapping mapping = new ResultMapping();
mapping.setPropertyName("value");
mapping.setColumnName("VALUE");
mapping.setColumnIndex(new Integer(1));
setBeanProperty(mapping, object, conn, rs, errorField);
} else
if(object instanceof Map)
{
ResultSetMetaData rsmd = rs.getMetaData();
int i = 0;
for(int n = rsmd.getColumnCount(); i < n; i++)
{
String columnName = rsmd.getColumnLabel(i + 1);
((Map)object).put(columnName, rs.getObject(i + 1));
}

} else
{
autoMapResultSet(conn, rs, object, errorField);
}
}
if(typeName != null && object != null && (object instanceof BaseValue) && !object.getClass().getName().equals(typeName))
object = StaticBeanProbe.getObject(object, "value");
return object;
}

[brian.stansberry]

From looking at that, it would seem the the type of the object put in the list would be controlled by the mapping iBatis is using. If some db field is mapped to java.lang.Object, that's what you'll get. Hard to see why a db field would be mapped to java.lang.Object, though.

[catchupvijay]

1. Based on some further research, I did not find any "java.lang.Object" declaration in the (Beans/VO, Action forms, Decorators) of the application classes.

2. The Mappings are clean and theres no DB -> field mapping of type Object.

3. Although, In "com.ibatis.db.sqlmap.cache.memory" package, MemoryCacheController has a inner class declaration "StrongReference", which has java.lang.Object declaration.

[smarlow]

It might help to add some debug print code to java.util.ArrayList.writeObject() (in java.util.ArrayList.java around line 529).

There are some hoops that you have to jump through to get the updated class ArrayList class file in to be used.

It should be worthwhile if you get information about which field it is that isn't serializable, as you will then know which one needs to change.

Don't do this in your production app server installation and don't leave the debug ArrayList in your production app server installation.

I hope makes sense as my sleeping pill is kicking in. :)