10,000 Recorc Per Second (In EJB 3.0)

[grdzeli_kaci]

hi all,
i have some mission critical tasks into my project, is it possible to persist 10 000 record per seconds,

1. AS - JBoss Application Server 4.0.4GA
2. Database - Oracle 10G 10.2.0.1
3.EJB - 3.0 Framework
4.OS - SunOS 5.10
4.Server - Memory: 16G phys mem, 31G swap, 16 CPU,

i know that i need performace
here is my configurations about performance

1. JVM Config Into JBoss

JAVA_OPTS="-server -Xmx3168m -Xms2144m -Xmn1g -Xss256k -d64 -XX:PermSize=128m -XX:MaxPermSize=256m
 -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000
 -XX:ParallelGCThreads=20 -XX:+UseConcMarkSweepGC -XX:+UseParNewGC
 -XX:SurvivorRatio=8 -XX:TargetSurvivorRatio=90 -XX:MaxTenuringThreshold=31 -XX:+AggressiveOpts
 -verbose:gc -XX:+PrintGCTimeStamps -XX:+PrintGCDetails -XX:+PrintTenuringDistribution

2. Also i have one simple Stlateless Session Bean

@Stateless
@Remote(UsageFasade.class)
public class UsageFasadeBean implements UsageFasade {

 @PersistenceContext(unitName = "CustomerCareOracle")
 private EntityManager oracleManager;
 @TransactionAttribute(TransactionAttributeType.REQUIRED)
 public long createUsage(UsageObject usageObject, UserContext context)
 throws UserManagerException, CCareException {
 try {
 oracleManager
 .createNativeQuery("INSERT INTO USAGE "
 + " (ID, SESSION_ID, SUBSCRIBER_ID, RECDATE, STARTDATE, APPLIEDVERSION_ID, CHARGINGPROFILE_ID, TOTALTIME, TOTALUNITS, IDENTIFIERTYPE_ID, IDENTIFIER, PARTNO, CALLTYPE_ID, USAGETYPE, APARTY, BPARTY, CPARTY, IMEI, SPECIFICCALLTYPE, APN, SOURCELOCATION, SMSCADDRESS, MSC_ID, ENDREASON, USAGEORIGIN, BILL_ID, CONTRACT_ID) "
 + " VALUES(SEQ_USAGE_ID.NEXTVAL, NULL, NULL, SYSDATE, SYSDATE, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL, NULL) ");
 return 1;
 } catch (Exception e) {

 }
 }
}

3. and into client side i have 200 Threads, each of them tried to call this method 50 times
my result is that i can persist 10000 record in 20 seconds, without hibernate, with hibernate i got worst result :(,

also i hear that it is good idea to use JDBC 3.0 driver for performance,
i download newest oracle jdbc jar file from oracle site
http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/jdbc_10201.html

is this jar file JDBC 3.0 driver ?

is it any more performance tuning into JBoss or EJB with entity beans?

can anybody help me ? or is there any doc which can help me ?

Regards,
Paata,

[fhh]

If all you want to do is to insert 10.000 meaningless records than write a stored procedure.

If not - well, tell us what you want to do.

Regards

felix

[genman]

Let me just say that you first need to understand what it takes to get Oracle to insert 10,000 records per second.

[fhh]

Insering 10.000 rows into an empty table is not that difficult. Any Oracle database should be able to handle that.

Regards

Felix

[oskar.carlstedt]

Hi!!

Using EJBs here is probably not the best way to go. Doing so will cause the container to create a lot of insert statements - too many to get the performance you want. If I'm wrong here, let me know.

I did something like this a couple of years ago with Oracle 8.1.6. If you don't care about all transactions a good way might be to create a data file (normally a comma seprataed flat file), which you later do an import on. Here you get performance, but if it is an good design. I don't think so. Good design and performance are often something that is hard to combine.

Another way is to create a stored procedure that you call when you have, let say 5000 records to insert.

One thing to think of is the data transport. Using Oracle from Java you go from client(s) to a java server, to PL/SQL to Oracle DB.

Kind Regards
Oskar

[oskar.carlstedt]

...

one more thing. Whater decision you make here. Make sure you are using the native Oracle driver.

//Oskar

[grdzeli_kaci]

hi all,
great tnx for your posts,

yesterday i tried to count real amount of records and i got i need 2000 operation in one second, (operation means some selects and 5 insert into separate tables)

fhh

If all you want to do is to insert 10.000 meaningless records than write a stored procedure.
If not - well, tell us what you want to do.

i thought on this idea, but i don't need business logic into database, it is business request.

genman

Let me just say that you first need to understand what it takes to get Oracle to insert 10,000 records per second

i tried to insert 10 000 record into oracle database from pl/sql developer over network and it takes 5 minutes.

oskar.carlstedt Post

Using EJBs here is probably not the best way to go. Doing so will cause the container to create a lot of insert statements - too many to get the performance you want. If I'm wrong here, let me know.

you are right, it is not correct way to persist 2000 record using entity beans and hibernate,

but i thought, maybe is there any performance configuration which can increase performance,

Make sure you are using the native Oracle driver

http://www.oracle.com/technology/software/tech/java/sqlj_jdbc/htdocs/jdbc_10201.html

[oskar.carlstedt]

It is possible to configure hibernate to do bulk operation. Perhaps you've already tried it? If not, take a look a http://www.hibernate.org/hib_docs/v3/reference/en/html_single/#queryhql-bulk

//Oskar

[fhh]

i thought on this idea, but i don't need business logic into database, it is business request.

I strongly disagree: All business logic that can convienently handled by the database should be handled by the database. It is a zillion times faster, easier to implement and easier to maintain. There is no point in moving data in and out of the database for nothing.

If you think that is my idiosyncratic position read the following statement from Gavin King:

The main example of where ORM - and indeed Java - is not suitable is the case of processing data in bulk. It is simply never going to be efficient to fetch millions of rows from the database, into your JVM, and then update them one at a time. Don't use Java for this. Use a stored procedure.

http://www.javaperformancetuning.com/news/interview041.shtml

Regards

Felix

[snau2005]

Hi,
Few month ago I made some test and achieved more than 5000/s inserts on average pc (actually there was 2 pc's in case with oracle, one with jboss another with oracleand 1 pcin case with postgresql, both resutls quite equal). These inserts was simple, dummy inserts.
It's importat to set batch updates, can be 1000 or more


very important to switch off logging


use oci driver (in case with oracle)

also i used few threads (best speed with 8 threads, i thing this depends on pc (how many processors and so on)), everything was done i one transaction, also you can play with flush().

[grdzeli_kaci]

hi snau2005,

very important to switch off logging

of course i did it.

use oci driver (in case with oracle)

is it helpfull ?? do you think that oci driver is so faster then thin driver ?

also i used few threads (best speed with 8 threads, i thing this depends on pc (how many processors and so on)), everything was done i one transaction, also you can play with flush().

of course i use Thread, Thread will be created on client side and all of them calls my stateless session beam method, i tried to create 50 Threads, and my pc is enough strong, it has 16 CPU, and 16 GB Memory, Sun Solaris,

is it good idea that i use oracle cluster or JBoss Cluster ?

and i cant found any documentation about stored procedures into ejb 3.0, can anybody show me it ?



Regards,
Paata.

[snau2005]

use oci driver (in case with oracle)

is it helpfull ?? do you think that oci driver is so faster then thin driver ?

I'm not sure in case of perfomance (thin driver had some bugs and i switched to oci (but this was long ago, now maybe thin also ok)

of course i use Thread, Thread will be created on client side and all of them calls my stateless session beam method, i tried to create 50 Threads, and my pc is enough strong, it has 16 CPU, and 16 GB Memory, Sun Solaris,

As I unterstood, every client Thread creates (calls ejb witch creates TR) his own transaction? so in reality you have 10000 transactions with one insert statement, thats of cource i think should be slow. In my case I called only one time ejb , so 1 TR was created then 8 Threads started to persist and flush.

and i cant found any documentation about stored procedures into ejb 3.0, can anybody show me it ?

Example to call stored function

try {
 HibernateEntityManager hem = ((HibernateEntityManager)em.getDelegate());
 Session session = hem.getSession();
 Connection connection = session.connection();

 CallableStatement proc = connection.prepareCall("{ ? = call getNBVZSInfo(?, ?) }");
 proc.registerOutParameter(1, Types.VARCHAR);
 proc.setString(2, traderCode);
 proc.setString(3, userName);
 proc.execute();
 result = proc.getString(1);

 log.info("Info: " + result);
 } catch (Exception e) {
 log.error("getNBVZSInfo, error: " + e);
 }

and oracle functions looks like:

FUNCTION getNBVZSInfo(traderCode varchar2, userName varchar2) RETURN varchar2 is
 rez varchar2(4000);
 begin

 rez := 'hoho and the bottle of rum';
 return rez;
end;

But again if bussines logic alows procedure to store 10000 , so one ejb call also can store 10000 in one call and if you will call oracle procedure 10000 time thats also will be slow.

[oskar.carlstedt]

Hi again!

Here is some information from a project I attended a couple of years ago. We operated on a Sun Starfire (Sun Enterprise 10000), totally 32 cpus and 32GB RAM, our domain had 12 cpus and 12 GB RAM. A lot has hapend since then, but still, such computers are not fast when it comes to simple things like database inserts. They are used for extreme load and may calulate a lot of complex stuff.

We hade Oracle on the starfire. On another server we had a Java program doing import stuff to the database.

In our case, on the Starfire, the application perfomed much better when using 5-10 threads than using 20 or 30 of them. (I've read about other persons complaining about how Java 1.4.x scales on multi cpu machines). Note that we used Java 1.4.x here. I don't know about how Java 1.5 or Java 1.6 behave on multi cpu machines.

When we changed the environment to a normal Compaq D180 server (the pizza box) with only 2 Intel Xeon, 1GB RAM having better clock frequency, we got much better performance. I can't remember exactly how much better, but I know we are talking about several 100 percents. We had Java on one server and Oracle on antoher server (also a pizza box, but 4GB RAM). These machines used Windows 2000 Server Std. Edition.

Then we tried to seutp Oracle on a 8 Xeon cpu Dell server 8GB RAM, with Windows 2000 Server Ent. Ed (you had to have ent. ed. to be able to use all processors and all memory). This machine had lower processor speed than the piza box. This configuration gave us worse performance compared to the 2 pizza boxes.

On all machines we were in contact with Oracle about how to setup the Oracle to get the best performance. One thing Otracle learned me, but this was version 8.1.6, was to setup the chunk size of the disks. When you raid your disks, you shall have as specific chunk size. At least on windows. You shall also have a special raid configuration. From the beginning we had raid 10, but as I can remember you shall use something else.

Best insert performance:
1. import of flat file
2. creating a stored procedure wher you sent a kind of comma separated string.
3. Java - Oracle simple sql prepared statements
4. Java - using simple statements.
...

What about the network between database and java. Are they running on separate machines. If not you will have a slower system.

Finally, Yes, the OCI drivers are much faster. Ther are communicating directly with the Oracle DB. The non OCI drivers are normally working with the PL/SQL in Oracle - an extra layer on top of the database.

Regards
Oskar


PS...
From Java, rember to turn off auto commit. Ds.

[grdzeli_kaci]

great thx for your posts, they are so useful for me,

you are right snau2005, oci driver is more faster then thin driver but not very good for my issue,
ok thx for your code about stored procedures, i will try it, if i could not fing any other way,

i don't want stoberd procedures and bech inserts becouse in this way i won't have transaction management on my hand,

and at last i tried to use Thread on server side, i created 10 threads and each of them tried to persist 1000 object into database, and i tried to flush after 50 records,

i wrote my thread class into server side and use it into stateless session bean, and set entittymanager class into Thread class and into run method i tryed to persist

public class ThreadMy extends Thread {
 EntityManager entityManager;
 List<UsageObject> usagesObjs;
 public ThreadMy(EntityManager entityManager,
 List<UsageObject> usagesObjs) {
 this.usagesObjs = usagesObjs;
 this.entityManager = entityManager;
 this.setPriority(Thread.MAX_PRIORITY);
 this.start();
 }

 public void run() {
 try {
 int count = 0;
 for (int i = 0; i < usagesObjs.size(); i++) {
 UsageObject usageObject = usagesObjs.get(i);
 entityManager.persist(usageObject);
 if (i / 50 >= 0) {
 entityManager.flush();
 count = 0;
 }
 count++;
 }

 } catch (Exception e) {
 e.printStackTrace();
 }
 }
}

and into my stateless sessionbean i have created 10 Threads

but i got an error like this

18:11:35,684 ERROR [STDERR] javax.persistence.TransactionRequiredException: EntityManager must be access within a transaction
18:11:35,685 ERROR [STDERR] at org.jboss.ejb3.entity.ManagedEntityManagerFactory.verifyInTx(ManagedEntityManagerFactory.java:149)
18:11:35,685 ERROR [STDERR] at org.jboss.ejb3.entity.TransactionScopedEntityManager.persist(TransactionScopedEntityManager.java:174)
18:11:35,685 ERROR [STDERR] at com.magti.client.test.ThreadMy.run(ThreadMy.java:81)

entityManager.persist(usageObject); // this is ThreadMy.java:81

how i can fix it ?
can anybody tell me ?


ok and another thing, thank you also for your post oskar.carlstedt

i am not an administrator of the network and systems into my company, i don't know servers very well, am a programmer.


you thing it is more useful to use windows servers instead of solaris servers ?
i thik if operation system is well configured, i think solaris is enough for my program and for this task.

it is not right way that i requst my company to bye this servers which you tell me (Compac D180 or Dell Server) for my program,

and what you say about best insert performance, i 100 % agree with you,
i want good design and high performance together :) ;), maybe it is impossible, but i want try all ways, after all maybe is use comma separated files and stored procedures,


PS ..
of course i turn off auto commit into my java program,



Regards,
Paata.

[oskar.carlstedt]

Hi again!

What I actually mean is that normally it is better to have a few really fast computers instead of a single powerful machine. And I understand your situation, you have this machine and you wan't to use it.

Do you know if there are other IO-intensive tasks running on that machine at the same time? Normally you just don't buy such a server to insert 10000 rows/sec in an oracle database. Are you "alone" on your server?

//Oskar