EJB big instalation 5000 EJB's and performance problem

[mireksz]

Hello,

I have problem with Jboss 5.0.0 GA and outofmemory when I deploy my application.

When I deploy with 1.000 of ejb serwer starts in 10 minutes but with 5.000 I got outofmemory

If some have expirience with that big applications??

[jaikiran]

Don't use JBoss AS 5.0.0. Instead download JBoss AS 5.1.0 and apply our latest EJB3 plugin (which has many performance fixes) over it http://www.jboss.org/ejb3/ejb3plugin.html

[mireksz]

Ok,

I downloaded Jboss 5.1 and install EJB3 plugin but I have still problem.

Now when I deploy my application with 1.000 EJB it took 2 minutes(600MB heap)

with 2.500 EJB took 15minutes(1.2GB heap)

with 5.000 I got outofmemory after 2hours

I simple generates ejb from program. Each ejb have 4 injection of another services. I trying simulate real app conditions.

Maybe there is some limitation of number of ejb

[jaikiran]

Mirek Sz wrote:

with 2.500 EJB took 15minutes(1.2GB heap)

You mean it deployed successfully with 2,500 EJBs in 15 minutes?

Mirek Sz wrote:

with 5.000 I got outofmemory after 2hours

Did the deployment fail for 5,000 EJBs? And it took 2 hours for the deployment to complete/fail?

Do you have the actual logs? And what params have you passed to the JVM for memory configuration? Which Java vendor and version?

[mireksz]

I using JDK 6_20,VIsta 32bit, I run Jboss with

set JAVA_OPTS=%JAVA_OPTS% -Xms128m -Xmx1700m -XX:MaxPermSize=256m

When I deploy 5.000 EJBs deployment fail with outofmemory in javaVisualVM was (1.7GB heap,I cann'ot add more on 32bit system)

with 2.500 application deploy successfull and work  after 15 minutes

[jaikiran]

What kind of EJBs are those? Are you just generating them to see how many EJBs can be deployed or are those some real app EJBs? Are those based on annotations? Basically, you will have to tell us a bit more about your application.

Also, since you are trying to do a huge deployment, I would recommend that you patch one of the JBoss AS files for this fix https://jira.jboss.org/browse/JBAS-6987. You'll just have to comment out a section in JBOSS_HOME/server/< servername>/conf/bootstrap/deployers.xml:

<!-- The holder for deployers that do real deployment -->
   <bean name="Deployers" class="org.jboss.deployers.plugins.deployers.DeployersImpl">
   <!-- Uncomment the following @org.jboss.aop.microcontainer.aspects.jmx.JMX to enable registering
    MC beans as MBeans. These MBeans provide debug level information about which deployers were selected for
    processing the deployment and what attachments were added to the deployment. More details http://www.jboss.org/index.html?module=bb&op=viewtopic&t=155074&start=10#4233874
    -->
    <!--
      <annotation>@org.jboss.aop.microcontainer.aspects.jmx.JMX(name="jboss.deployment:service=DeployersImpl", exposedInterface=org.jboss.deployers.plugins.deployers.DeployersImplMBean.class, registerDirectly=true)</annotation>
      -->

(see the commented section above. In AS5, that's uncommented by default. Comment it out to improve memory usage).

[mireksz]

This is all(EJB 3.0) Stateless beans with one remote Statefull Facade. I make deployment by ejb-jar.xml file(no annotation).

I genering 5.000 EJB's beacause we have old system which we have to write in new technology.

In our database existing 1.800 tables in standard way we generate DAO + Service per table which mean

two EJB's per table = 3.600 + another not standard services

I trying to choose technology EJB 3.x vs Spring and Servers for them

[mireksz]

I comment this line in /conf/bootstrap/deployers.xml

And app with 2.500 EJB's from 1.2 GB decreese to 600MB !!! but deployment time still 15minutes

[jaikiran]

Mirek Sz wrote:

This is all Stateless beans with one remote Statefull Facade. I make deployment by ejb-jar.xml file(no annotation).

I genering 5.000 EJB's beacause we have old system which we have to write in new technology.

I our database existing 1.8000 tables in standard way we generate DAO + Service per table which mean

two EJB's per table = 3.600 + another not standard services

I don't know anything about your application architecture or other details. So I can't comment. But a Session bean per table isn't really a good idea.

As for the deployment time of the application, without looking at the real application, I can't really say where it's spending time. Maybe you could try to profile the deployment through some profiler (like JProfiler).

[peterj]

A few comments and observations:

Add the -XX:+HeapDumpOnOutOfMemoryError option to the command line (add it to JAVA_OPTS in the run script). This option will dump the contents of the heap when you get an out of memory error. You can read the heap dump with the jvisualvm. As a word of warning, you will need about 2-3 times as much heap for running jvisualvm as for the amount of heap space dumped. As an example, if your max heap is 1.2GB, then you will need to set max heap to 3.6GB for jvisualvm, or it too will run out of heap space. You can then determine the amount of space each EJB takes up. I have dealt with some of these COBOL-to-Java translators before and they use up an enormous amount of memory to hold often very simple data - I have seen as much as 128 bytes just to hold a basic PIC 9 field.

I doubt very seriously that without doing a complete rewrite (which I am sure you don't want to do at this time) that you will be able to run with a 32-bit JVM. You will have to move up to a 64-bit OS and a 64-bit JVM and give the app more heap space. And make sure you assign lots of heap space (you will need a lot of RAM) and that you use the CMS collector, otherwise a standard major collection will bring the app to a halt for several minutes.

[peterj]

One more thought:

I suspect that the large jump in startup time between 2500 EJBs and 5000 EJBs is that the memory is so filled that you are doing nothing but major collections which take a long time, especially if you have very little garbage data to collect. I suspect that if you assigned a decently-sized heap, say 3GB or larger) that the startup time would be more linear (around 30 minutes). Of course you would have to turn on gc statistics to verify this.

[hifly81]

If you have RAM try to set Xms equal to Xmx and assign to Xms and Xmx around 3GB. But of course tuning is something that you must always experiment live

[hifly81]

sorry I red now that you have a vista 32 bit so no way to add more Ram than 1,7GB to the heap.... swith to linux it's hard to mange 5000 ejb with a windows 32 bit

[mireksz]

I making test with openeejb container and 2.500 EJBs deploy in 5 minutes(150MB).

Also I tryied spring and xml factory. It starts with 20 secund with 2.500 beans