[tomjenkinson]

Unfortunately I haven't dealt with the H2 team directly before.

The only other suggestions I can think of is to try this with PostgreSQL itself, would that be an option?

Also, it is probably not going to help, but could you share your driver code so we can scan it for anything you might have missed?

[mauromol]

Honestly, I don't think that the PostgreSQL devs can help here

Our h2_driver class is very simple actually:

package com.arjuna.ats.internal.arjuna.objectstore.jdbc.jdbc;
import com.arjuna.ats.internal.arjuna.objectstore.jdbc.postgresql_driver;

public class h2_driver extends postgresql_driver
{
 @Override
    public String name ()
    {
        return "h2";
    }
}

The accessor is instead (javadoc removed):

package com.cardinis.cardinis.support.io.server.db;

import java.io.File;
import java.sql.Connection;
import java.sql.SQLException;
import java.util.Collection;
import java.util.HashSet;

import org.h2.jdbcx.JdbcDataSource;

import com.arjuna.ats.internal.arjuna.objectstore.jdbc.JDBCStoreEnvironmentBean;
import com.arjuna.ats.internal.arjuna.objectstore.jdbc.accessors.accessor;
import com.arjuna.common.internal.util.propertyservice.BeanPopulator;
import com.cardinis.cardinis.support.io.server.transaction.JBossTSLifecycleManager;
import com.ost.cardinis.util.LogManager;
import com.ost.util.Application;
import com.ost.util.log.CommonLogManager;

public class h2_accessor extends accessor
{
 public static final String DB_USER_NAME = "sa";
 public static final String DB_PASSWORD = "sa";

 private static String dBBaseUrl = null;
 static
 {
 setUseInMemoryDB(false);
 }

 private static boolean clearJustOnShutdown = false;

 private static int MAX_CONNECTIONS = BeanPopulator.getDefaultInstance(
 JDBCStoreEnvironmentBean.class).getJdbcPoolSizeMaximum();

 private static final Collection<Connection> allGivenConnections =
 new HashSet<Connection>((int) (MAX_CONNECTIONS / .75) + 1);

 private final JdbcDataSource dataSource;
 private final Collection<Connection> givenConnections =
 new HashSet<Connection>((int) (MAX_CONNECTIONS / .75) + 1);

 private static final Thread SHUTDOWN_HOOK =
 new Thread("Cardinis h2 accessor shutdown hook")
 {
 @Override
 public void run()
 {
 closeConnections(allGivenConnections);
 allGivenConnections.clear();
 };
 };

 public h2_accessor()
 {
 final JdbcDataSource ds = new JdbcDataSource();
 ds.setURL(getDBBaseUrl()
 + ";DB_CLOSE_ON_EXIT=TRUE;AUTO_SERVER=TRUE;AUTO_RECONNECT=FALSE;MODE=PostgreSQL");
 ds.setUser(DB_USER_NAME);
 ds.setPassword(DB_PASSWORD);
 dataSource = ds;
 JBossTSLifecycleManager.getInstance().register(this);
 }

 @Override
 public Connection getConnection() throws SQLException
 {
 final Connection result = dataSource.getConnection();
 givenConnections.add(result);
 return result;
 }

 @Override
 public void putConnection(final Connection conn)
 {
 try
 {
 if(!conn.isClosed())
 conn.close();
 }
 catch(final SQLException e)
 {
 LogManager
 .getInstance()
 .getLog()
 .warning(
 "exception while trying to close a returned H2 Object Store connection, or while checking if it is closed",
 getClass(), e);
 }
 }

 public void clear()
 {
 if(!clearJustOnShutdown)
 closeConnections(givenConnections);
 else
 allGivenConnections.addAll(givenConnections);
 givenConnections.clear();
 }

 private static void closeConnections(final Collection<Connection> connections)
 {
 for(final Connection connection : connections)
 try
 {
 if(!connection.isClosed())
 connection.close();
 }
 catch(final SQLException e)
 {
 CommonLogManager.getInstance().getLog().error(
 "error closing Object Store H2 connection", h2_accessor.class, e);
 }
 }

 public static void setClearJustOnShutdown(final boolean clearJustOnShutdown)
 {
 h2_accessor.clearJustOnShutdown = clearJustOnShutdown;
 if(clearJustOnShutdown)
 // se stiamo abilitato la chisuura allo shutdown, registra l'hook
 try
 {
 Runtime.getRuntime().addShutdownHook(SHUTDOWN_HOOK);
 }
 catch(final IllegalArgumentException e)
 {
 // già registrato, ignora richiesta
 }
 else
 {
 // altrimenti chiudi tutte le eventuali connessioni aperte
 SHUTDOWN_HOOK.run();
 // e deregistra l'hook
 Runtime.getRuntime().removeShutdownHook(SHUTDOWN_HOOK);
 }
 }

 public static String getDBBaseUrl()
 {
 return dBBaseUrl;
 }

 public static void setUseInMemoryDB(final boolean inMemory)
 {
 if(inMemory)
 dBBaseUrl = "jdbc:h2:mem:CardinisObjectStore";
 else
 dBBaseUrl =
 "jdbc:h2:" + new File(Application.getContextFolder()).getAbsolutePath()
 + "/jta/ObjectStore";
 }
}

The part that tracks and closes connections is due to JBTM-465.

Mauro.

[tomjenkinson]

Hi Mauro,

What I meant is, can you use Postgres, rather than H2?

Did the H2 team have any feedback for you?

Thanks,
Tom

[mauromol]

Hi Tom,

it's not easy to replace H2 with PostgreSQL in our environment, also because we have to reproduce the problem (i.e.: "broken" transactions).

I was able to write to the H2 mailing list at last:

http://groups.google.com/group/h2-database/browse_thread/thread/efdb6e132cac5549#

Now the problem is to reproduce the issue. Thomas Mueller (the author of H2) is not aware of such a problem and is not able to reproduce.

I tried myself to add dummy entries in the jbosststxtable table of the same H2 db instance JBossTS is using, like these:

insert into jbosststxtable values('1', 'ciao', 'bau', null)

insert into jbosststxtable values('1', 'ciao2', 'bau', null)

update jbosststxtable set typename='ciao3' where typename='ciao'

That is, I'm trying to add rows where just the TYPENAME column value is different or to update existing rows by changing only that column value, since the original error message seems to suggest that H2 is considering only (UIDSTRING, STATETYPE) as the primary key.

However, these simple queries do not produce any error.

Could you please suggest a "real" (or likely) sequence of SQL instructions that JBossTS usually produce when inserting rows in that table and when an entry expires?

Maybe the kind of SQL that is passed to H2 is causing the problem to happen...

Mauro.

[tomjenkinson]

Hi Mauro,

I have to say, that sequence looks fairly accurate.

You can see from this class what happens when an entry is moved:

com.arjuna.ats.internal.arjuna.recovery.ExpiredTransactionScanner::moveEntry(Uid newUid)

The state is read_committed, and if it exists, it is write_committed.of a new type, followed by a remove_committed of the original:

i.e.:

insert into jbosststxtable values('1', 'ciao', 'bau', null)

insert into jbosststxtable values('1', 'ciao2', 'bau', null)

delete from jbosststxtable where uidstring = 'bau' and typename='ciao' and statetype='1'

The only possible thing I can think happens is replayCompletion was happening at the same time as the expiry monitor fired, would it be possible for you to send the logs for when the error happened, please?

Tom

[mauromol]

Well, the complete log is very very long, because of the stack overflow problem (by the way, should I open a JIRA for this?).

If you tell me what I have to look for, I can do a search in it.

Thanks again,

Mauro.

[mmusgrov]

We need the log entries around the time the TM tried to move the transaction to the expired log. Search for "is assumed complete and will be moved" it will be around the point where the primary key violation ocurred.

[mauromol]

Here is a ZIP with the logs:

http://dl.dropbox.com/u/55072601/jbossts-logs.zip

In the first file you'll find the first time the problem occured and the stack overflow exception.

The second file is about the logs of the following day, after the webapp was restarted. The stack overflow exception seems not to be explicit, however there are still signs of it. In this second file you'll find a couple of iterations of the problem, until the "Removing old transaction status manager item" message.

I tried to remove all the log entries not related to JBossTS (they were few, indeed).

Mauro.

[tomjenkinson]

Hi Mauro,

I cooked up a unit test using your h2_driver (not your accessor though that could be added) to see if I could replicate it which I can't, please find it attached.

1. Does the test work for you?

    Assuming it does, can you try to run the test against a ***copy**** of your production database?

2. Are you without a doubt positive (I know you said you were on the google group forum but I have to re-ask as we really feel that insert error returning the wrong PK must be part of the answer) you are pointed at the same database when you "desc" the table versus when you are running. Can you print the full path url in your jdbc url to confirm this, e.g. in setUseInMemoryDB

3. Can you switch on H2 tracing? You can see this in my attached example.

4. Could you have more than one recovery manager process running?

5. Is it possible that the table was created by an earlier version of JBoss TS where the PK may have been different?

6. Can you try to run h2 as a server and connect over tcp and see if that has any different effect?

Overall, and assuming you are 100% it is the same table, we do feel that the version of H2 you are using has a bug somewhere as your desc'ing the table shows a PK that the JDBC insert isn't respecting and is erroring on.

Tom

[tomjenkinson]

Sorry, I just couldn't leave it at that, so added your accessor too, please find attached a revised version