@Asynchronous & thread-pool

[fcorneli]

Just noticed that if @Asynchronous methods never return (for whatever reason), then the thread-pool runs out after a while (10 per default). The transaction manager gives a timeout, but the thread is never terminated. Don't know if this is the expected EJB3 behaviour.

17:39:31,378 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper) ARJUNA012117: TransactionReaper::check timeout for TX 0:ffff7f000001:29a8c921:52ee5c48:1290 in state  RUN
17:39:31,381 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012095: Abort of action id 0:ffff7f000001:29a8c921:52ee5c48:1290 invoked while multiple threads active within it.
17:39:31,381 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012108: CheckedAction::check - atomic action 0:ffff7f000001:29a8c921:52ee5c48:1290 aborting with 1 threads active!
17:39:31,383 WARN  [org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorImpl] (Transaction Reaper Worker 0) Transaction afterCompletion called by a background thread! Delaying action until the original thread can handle it. [status=4]
17:39:31,383 WARN  [com.arjuna.ats.arjuna] (Transaction Reaper Worker 0) ARJUNA012121: TransactionReaper::doCancellations worker Thread[Transaction Reaper Worker 0,5,main] successfully canceled TX 0:ffff7f000001:29a8c921:52ee5c48:1290

[wdfink]

This is the same behaviour as a standard EJB.

If an execution is blocked the thread is blocked also. There is no interruption. Normaly this is if a resource i.e. DB is blocked.

The only thig is that the transaction is marked for rollback.

[smarlow]

Could you try setting "hibernate.jta.track_by_thread" to false in your persistence.xml?

<properties>

  <property name="hibernate.jta.track_by_thread" value="false" /> 

</properties>

[neshone]

Hello,

I happened to come by the same issue. Maybe I can give you more information.

In my case, the issue manifests only when an @Asynchronous method in EJB-A is called from another @Asynchronous method in EJB-B.

When calls to an @Asynchronous method in EJB-A are NOT made from another @Asynchronous method, the thread limit is not an issue (I went up to 10000 calls when the VM ran out of memory).

I tried disabling the transactions with @TransactionAttribute(value=TransactionAttributeType.NOT_SUPPORTED), because I don't need them here and I thought maybe multiple transactions are causing the problem. This made the warnings go away, but the calls to the method still get blocked.

By the way, the actual EJB-A @Asynchronous method never gets to be executed, so I doubt the problem is in the method itself.

I tried referencing the EJB 3.1 spec, but I didn't find anything useful.

Any help will be greatly appreciated.

Best regards,

Nenad

[smarlow]

Nenad,

Do you also see the same warning?

1. WARN  [org.hibernate.engine.transaction.synchronization.internal.SynchronizationCallbackCoordinatorImpl] (Transaction Reaper Worker 0) Transaction afterCompletion called by a background thread! Delaying action until the original thread can handle it. [status=4] 

Which build of WildFly are you using?  The nightly build is here if you want to verify that the issue hasn't been resolved already. 

Thanks for responding about this issue. 

Scott

[smarlow]

I couldn't recreate this on the latest WildFly master code (should be the same as the nightly link I gave you above).  Any suggestions for what the below needs to do differently to recreate?

@Stateless

@Remote

@Asynchronous

public class TestBean1 {

    @Inject

        TestBean2 bean2;

    public void test() {

      bean2.test();

    }

}

@Stateless

@Remote

@Asynchronous

public class TestBean2 {

    public void test() {

    }

}

I don't seem to leak any threads when running the above code (we cycle through the thread pool and reuse the threads as expected).

[neshone]

Hello Scott,

I get these warnings (the specific line you quoted is not present):

gist:5929d63d8df48e60f4fd

Actually, I'm using JBoss AS 7.1. Just noticed this is the Wildfly discussion, I'm really sorry, I'm new to the forum. Does this mean that I'm stuck with this issue unless I switch to Wildfly?

Thanks for your help.

Best regards,

Nenad

[smarlow]

Nenad,

Still good to hear from you.  Yes, please try the latest WildFly build if you like (using above nightly build link).

For AS 7, you could try using our EAP 6 which is based on AS 7.  No more changes are expected for AS 7 but we will continue to address AS 7 problems in the EAP 6 product.

Best,

Scott

[neshone]

Scott,

I tried the EAP 6.2 GA, the issue is also present there. I'll think about testing the issue on the Wildfly.

Thanks a lot for the info.

Best regards,

Nenad

[neshone]

Hi Scott,

I tried the nightly build this morning, the issue is also present in wildfly.

On the side note, in your example, when testing, try calling the TestBean2 more than once. One call doesn't expose the problem. I think 10 should be enough, but I tested it with 100 calls.

Best regards,

Nenad

[smarlow]

Nenad,

Could you create a jira for this issue?  The link is https://issues.jboss.org/browse/WFLY

Thanks for your help,

Scott

[smarlow]

Hmm, I'm still not reproducing.  Could you also attach a failing test case to the jira or here.  You could make it similar to the test case for WFLY-2651 (see git link in jira). 

[neshone]

Hi Scott,

I tried a simpler example and I think I'm getting a slightly different behavior then the one I described earlier. i wrote two EJBs, similar to the ones in your example and invoked the first one from a servlet. Each EJB @Asynchronous method returns an instance of the future interface.

The servlet invokes the first EJB's @Asynchronous method in a for loop of 11 iterations and each time the second EJB @Asynchronous method is invoked 15 times from the first EJB:
servlet ----11 times----> EJB1 -----15 times -----> EJB2

After the invocations in both the servlet end the first EJB, I wait for each of the results with Future.get().

This time the @Asynchronous methods don't block the caller, but not all invocations end, so the caller stays blocked on one of the Future.get() calls (I think the @Asynchronous method never gets invoked).

This is what I got after dumbing down, my use case. I still think the method was blocking in the original use case, but I'll try to investigate some more and provide you with an example for that too, if I manage to create a simple one.

I've attached a simple use case that reproduces the bug.

Please let me know if you managed to reproduce the bug or if you find an error in my logic.

Thanks.

Best regards,

Nenad

[smarlow]

Hi Nenad,

I switched to using your servlet/bean example code (hacked it into my current standalone test since I'm setup to build that via maven already).

With a default WildFly configuration, I was able to run the servlet once and it completed.  When I ran your servlet code the second time, the servlet didn't complete.  See attached gotstuck.log if you like.

I killed the WildFly server and updated the EJB3 configuration to increase the number of threads available to run EJB requests and this helped.  See attached moreejbthreads.log which shows plenty of available threads (by increasing ejb max-threads).

<subsystem xmlns="urn:jboss:domain:ejb3:2.0">

...

<thread-pools>

       <thread-pool name="default">

           <max-threads count="1000"/>

           <keepalive-time time="100" unit="milliseconds"/>

      </thread-pool>

</thread-pools>

The max-threads should be increased to at least to account for the expected number of threads for a single request.  This is a great case to recognize and understand how to configure for.  Other feedback is welcome but this is what it looks like to me.

Could you try increasing the (ejb) max-threads pool size also?

Scott

[neshone]

Hi Scott,

We increased the thread pool size and everything completed, just like you said.

This is a bit confusing for me now. Is this the expected behavior (methods never get invoked under heavy load)? Shouldn't it just be distributed over available threads?

Aside from this, are there any negative side effects caused by increasing the maximum number of threads in the pool? Is there a link where I can get some more info on that, I'd like to better understand all this before deciding that it's OK for me to fix this in this way (for an example, what exactly is this pool used for?)?

I'd also like to state that I'd rather not switch to Wildfly and stay on JBoss 7.1.1. I just tested the code on Wildfly for our collective debugging purposes.

Thanks for the help.

Best regards,

Nenad