Is there a way to solve this without cluster?

[mglowacki]

I have a very powerful machine, with 80 cores and 32GB RAM, but when there is a traffic peak, it goes down. It hosts Apacje/JBoss/Postgres with deployed Seam web applications and EJB 3 for app. logic.

In thread dumps I see a lot of:

Thread: ajp-0.0.0.0-8009-14 : priority:5, demon:true, threadId:115, threadState:BLOCKED -

waiting on 0x7406ce55 (a java.util.WeakHashMap) org.jboss.naming.ENCFactory.getObjectInstance(ENCFactory.java:114)

and

Thread: ajp-0.0.0.0-8009-2 : priority:5, demon:true, threadId:103, threadState:BLOCKED

- waiting on 0x672e12af (a java.util.HashMap) org.jboss.metadata.web.jboss.JBossWebMetaData.getRunAsIdentity(JBossWebMetaData.java:839)

and

Thread: ajp-0.0.0.0-8009-14 : priority:5, demon:true, threadId:115, threadState:BLOCKED

- waiting on 0x7406ce55 (a java.util.WeakHashMap) org.jboss.naming.ENCFactory.getObjectInstance(ENCFactory.java:114)

As I understand, there is too many invocations of methods trying to access naming service. In Tomcat, I have set maxthreads on 8009 connector to 1200, however the dumps shows the problems starts when it's over 600-700.

I believe that setting up cluster should help, as the traffic will be distributed to several jboss instances. However, as I afraid cluster like a fire - is there anything more I can do?

Regards,

Michal

[rhusar]

Czesc Michal,

great analysis and very good catch. This reminds me of my issue back in 2010 where I analyzed the synchronization in ENC factory is suboptimal. Could you have a look at the issue https://issues.jboss.org/browse/JBPAPP-4787 whether it is what you are seeing too?

Cheers,

Rado

[rhusar]

Nearly forgot, the other 2 issues are:

https://issues.jboss.org/browse/JBPAPP-4786 - JBossWebMetaData.getRunAsIdentity

https://issues.jboss.org/browse/JBPAPP-4772 - EJB3 pool synchronization inefficiency; only open to partners

[rhusar]

As you see, these problems are not really fixable and havent been completely fixed.

Lets get to solution then. Really, solving this with "cluster" as in fully replicated states and etc does not make much sense. Clusters provide high availability but since you would be running all instances in the same machine, the failure of one component or network or whatnot of that particular machine makes your entire "cluster" unavailable. Therefore, I propose a  solution to use multiple standalone servers (well, depends on your app if thats doable). You can either do this via different port binding or create new virtual network interfaces and bind servers to them.

[mglowacki]

Yes, this is exactly what I would like to try - create two standalone jbosses and loadbalance them using Apache, so the visitors traffic is divided between two of them.

Regarding your issues, I think it's same as mine:

Thread: ajp-0.0.0.0-8009-813 : priority:5, demon:true, threadId:1431, threadState:RUNNABLE

org.jboss.naming.ENCFactory.getObjectInstance(ENCFactory.java:114)
- locked <0x7406ce55> (a java.util.WeakHashMap)
javax.naming.spi.NamingManager.getObjectInstance(Unknown Source)
javax.naming.spi.NamingManager.getContext(Unknown Source)
javax.naming.spi.ContinuationContext.getTargetContext(Unknown Source)
javax.naming.spi.NamingManager.getContinuationContext(Unknown Source)
org.jnp.interfaces.NamingContext.lookup(NamingContext.java:832)
org.jnp.interfaces.NamingContext.lookup(NamingContext.java:686)
javax.naming.InitialContext.lookup(Unknown Source)
org.jboss.seam.transaction.Transaction.getUserTransaction(Transaction.java:82)

and other threads waiting for weakhashmap. BTW, I am using JBoss 5.1 standalone, not EAP.

[rhusar]

So is there something I could help you with setting this up?

[mglowacki]

hehe, just promise this would help

[rhusar]

LOL!!!

I promise this will fix the issue of thread congestion resulting in blocking around EJB/Metadata. However there will be some performance tradeoff for running multiple instances plus this might introduce new issues. I think its worth a try and its not that much work to set that up.

[wdfink]

I suppose this solution will help (for a while) I have had a similar configuration using EJB.

For this we setup multiple instances on one power system. It will solve the 'sync' stuff but if the JBoss app scales you might bring the DB in behind under heavy load .

Will say as more performance you have as more you will need