Transaction and 2 threads

[mpulse]

Hello

So we have two threads :

Jmsthread --> onMessage(-) --> method send() --> message.setX() --> entitymanager.merge(message)

Receiverthread --> readPacket() --> em.find(ByIdmessage) --> message.setY() --> em.merge(message)

the first thread is in a transaction start with onMessage
the second too between a begin and commit transaction

the problem is in database we don't always get the good data, it depends the order of merge method sometimes we have X or Y with the good value
or X and Y it depends

Receiverthread start after method send().

we want the second merge start after the end of the transaction start with onMessage.


In jbossjta-properties.xml

<properties depends="jta" name="jdbc">
property name="com.arjuna.ats.jdbc.isolationLevel" value="TRANSACTION_SERIALIZABLE"/>
</properties>

[mpulse]

I'm trying to set and retrieve data on an EJB3 entity bean in two concurrent threads:

Jmsthread --> onMessage(-) --> method send() --> message.setX() --> entitymanager.merge(message)
The JMS provider starts a transaction with onMessage(). It changes some properties on an entity bean that is retrieved from a queue sends the contents through a TCP connection and merges it into the entity manager.
The other peer of the TCP connection responds and the response is evaluated by a receiverthread:

Receiverthread --> readPacket() --> em.find(ByIdmessage) --> message.setX() --> em.merge(message)

The receiverthread looks up the original message that belongs to the response, and modifies some properties on the original message, then merges it with the entitymanager (the same as the one used by the JMS provider).

Now, after this procedure the state of the entity is highly inconsistent. Due to race conditions, and probably bad transaction synchronization, dirty data is read by receiverthread and saved back to the entitymanager without causing a rollback. The properties changed by the JMS thread on the object are sometimes reflected in the object retrieved by the find() method, and sometimes aren't.
I have tried setting the transaction isolation to SERIALIZABLE, but that didn't make much of a difference.

Any hints?

[jhalliday]

"Mpulse" wrote:

Receiver thread start after method send().

we want the second merge start after the end of the transaction start with onMessage.

Then you need to make that syncronization requirement explicit in your code. Use JMS for the communication between threads too i.e. have the onMessage() signal the receiver to start by sending it a message. The sending of that message will be tied into the commit of the first tx, along with the em.merge Hence the receiver thread will never begin before the updated data it needs to read has been committed to the db. If that reduction in concurrency is too much, then have the second thread start as it does now but block before the merge until it gets the JMS message from the first thread. However, you'll not be certain of reading the committed data from the first transaction in this case, so the changes taking place in the two threads must be consistent or the second merge will fail.

"Mpulse" wrote:

dirty data is read by receiver thread

Dirty reads have very specific meaning in transactions. This means that transaction B is reading data that transaction A has changed but not yet committed. Is that really what you are seeing, or are you just getting the state before the changes?

"Mpulse" wrote:

The properties changed by the JMS thread on the object are sometimes reflected in the object retrieved by the find() method, and sometimes aren't.

That's allowed even under SERIALIZABLE. The serial ordering may be A,B or B,A depending on your thread scheduling. If you need to force one or other ordering then use the method described above. Or just do it all in a single transaction.

[mpulse]

Ok, maybe I should give a few more details about what we're trying to achieve. The application consists of 2 peers that exchange messages that need to be acknowledged by the receiving end.

The JMS thread does trigger a send-operation within its transaction and sets a SENT flag on the message (the entity bean) and calls merge.

At the same time, there is a Receiver thread that processes incoming messages from the peer as well as ACK messages as a response to the sent messages.

Hence, sending and receiving operations take place in different threads (the receiver has a dedicated thread that does nothing else and the sender can be any thread, e.g. the JMS thread). I can't block the receiver thread based on sending threads, because there might be messages in the receiving queue that need to be read, and the ACK from that particular sending operation can take a long time to arrive.

It can happen that the ACK message is being processed before the JMS thread transaction (setting the SENT flag) is committed. That means we can't see the SENT flag yet after retrieving the entity with em.find(), which is OK. The order of the commits are not important.

However, if the JMS transaction is not committed before the Receiver transaction starts, SENT flag will never be persisted so it looks like the transaction has been rolled back because of concurrent write operations on the same object that are in conflict. I don't see any rollback messages in the logs.

I keep seeing data not being persisted at all from both threads depending on which one gets to commit first. It looks like the first one to commit wins, and the changes of the second one are lost.

Thanks for your help.

[jhalliday]

Something odd here. You are passing an entity bean around as a message, but also using find to retrieve it?

[mpulse]

Yes, exactly. It's an entity bean for logging reasons. The receiver thread needs to use find() to look it up, because several hours can pass between the sending of the message and the acknowledgment by the peer and the 2 threads are totally disconnected.

The peer sends a transaction ID for the send operation that needs to be persisted with the original message to be able to track what happened to the message later on.

[jhalliday]

The bit that confuses me is: why pass the whole object if you are going to look it up anyway? Why not just pass the id needed for the lookup. Then, why merge something you looked up? The merge will overwrite the looked up object, including any changes made since the disconnected copy was taken. Either you have some rather odd business requirements or there is a design flaw here.