10,000 Recorc Per Second (In EJB 3.0)| JBoss.org Content Archive (Read Only)

15. Re: 10,000 Recorc Per Second (In EJB 3.0)

snau2005 Apr 22, 2007 12:43 PM (in response to grdzeli_kaci)

and into my stateless sessionbean i have created 10 Threads

but i got an error like this

entityManager.persist(usageObject); // this is ThreadMy.java:81

how i can fix it ?
can anybody tell me ?

Do not use in Threads main entity manager, call in each Thread another sessionbean which will do what now Thread does (persist) (or call the same, but different method).

16. Re: 10,000 Recorc Per Second (In EJB 3.0)

grdzeli_kaci Apr 23, 2007 2:51 AM (in response to grdzeli_kaci)

hi all,
:) i have some news :) i tried to divide my program to some parties, and i recount my amount of needed persistable records and i got 1000 per second,

after some performance configuration i got 400 record per second using Thread into server side,

i have inserted 10,000 record in 28 seconds using 50 Threads
i have inserted 10,000 record in 51 seconds using 5 Threads

hi snau2005
i did Threads ad you say,

hi oskar.carlstedt,
OK i have discuss my issue to my chief and i will have two powerful server for my program, one will be for Oracle Database (with 8 CPU, 32 GB Memory, SunOS) and second will be for my JBoss Application Server,

know if on the same machine i got 400 record per second, is it possible after divide this task i got 1000 record per second ??

i have not two Development Server for Testing :(

oooh and am will not be "alone" on my database server, some users may be using some another schemas or database users, my JBoss User is not only one user fir my datrabase,

ok, if i got 1000 record per second it is enough for me,

i don't want stored procedures and some other things, if it is possible,
i tried to have good design pattern ;) :) if it is possible :)

oo yes, i have never use JBoss Cluster, can it increase performance ?
also i'm just trying to optimize my database table, but theres nothing to performance persist into table, for select there is so many performance tuning options (Query Hints, indexes, partitions and so on), but for insert i don't know haw i can increase performance on database level

17. Re: 10,000 Recorc Per Second (In EJB 3.0)

snau2005 Apr 23, 2007 5:19 AM (in response to grdzeli_kaci)

also i'm just trying to optimize my database table, but theres nothing to performance persist into table, for select there is so many performance tuning options (Query Hints, indexes, partitions and so on), but for insert i don't know haw i can increase performance on database level

indexes slowing insert statements. Also id which is generated by hibernate (@Id @GeneratedValue(strategy=GenerationType.TABLE) was much faster than with sequence (GenerationType.SEQUENCE)

18. Re: 10,000 Recorc Per Second (In EJB 3.0)

grdzeli_kaci Apr 23, 2007 8:24 AM (in response to grdzeli_kaci)

"snau2005" wrote:
also i'm just trying to optimize my database table, but theres nothing to performance persist into table, for select there is so many performance tuning options (Query Hints, indexes, partitions and so on), but for insert i don't know haw i can increase performance on database level

indexes slowing insert statements. Also id which is generated by hibernate (@Id @GeneratedValue(strategy=GenerationType.TABLE) was much faster than with sequence (GenerationType.SEQUENCE)

hi again snau2005,

ok, of course, i will try to use GenerationType.TABLE, maybe it will be increase performance a bit,

does anybody knows some database performance optimizations, which can help me a bit ?

Regards,
Paata

19. Re: 10,000 Recorc Per Second (In EJB 3.0)

oskar.carlstedt Apr 23, 2007 8:57 AM (in response to grdzeli_kaci)

Ok!!

Having indexes will slow down an insert statement. Why? The database has to write an extra entry into an index for each insert. Depending on indexed column, this may also slow down update statements.

Using transacttions is even worse. The database must write into a transaction log to be able to do a rollback if necessary.

Using sequences instead of an id generated by Java. Yes, a little bit of course. For each insert statement, Hibernate must query the database for the next sequence number, or hibernate must update the Java object with the new sequence number set by the database using an sql query.

In general all SQL databases are slow when it comes to disk io, i.e. taking care of the data. The data files are normally not optimized for either reading and/or writing. A relation database will force the disk to move the read head, which is a physical movement, serveral times on the disk for each data read/write operation. Indexes are normally read into memory, but the data has to be read/written. This is why realtion database are fast in finding items (in memory index lookup) but relatively slow when it comes to fetching/writing data.

Regards
Oskar

20. Re: 10,000 Recorc Per Second (In EJB 3.0)

fhh Apr 23, 2007 9:27 AM (in response to grdzeli_kaci)

Sorry, but this is ridicolous. If you want to tune duping 10.000 records into a table use a stored procedure and be done with it. That is what they are made for.

If this is too slow, you can still temporarily disable indexes, select a ptoper transaction isolation level etc. etc.

Regards

Felix

P.S.: And DON'T forget to use bind variables :-)

21. Re: 10,000 Recorc Per Second (In EJB 3.0)

snau2005 Apr 23, 2007 9:32 AM (in response to grdzeli_kaci)

grdzeli_kaci,

 int count = 0;
 for (int i = 0; i < usagesObjs.size(); i++) {
 UsageObject usageObject = usagesObjs.get(i);
 entityManager.persist(usageObject);
 if (i / 50 >= 0) {
 entityManager.flush();
 count = 0;
 }
 count++;

if (i / 50 >= 0) { - here perhaps is an error, shouldn't be count instead of i ?

i found my old test, run it again, and got 40 000 entries in 11-13 seconds, but my table have only 4 columns no indexes etc ...

22. Re: 10,000 Recorc Per Second (In EJB 3.0)

snau2005 Apr 23, 2007 10:00 AM (in response to grdzeli_kaci)

i found my old test, run it again, and got 40 000 entries in 11-13 seconds, but my table have only 4 columns no indexes etc ...

23. Re: 10,000 Recorc Per Second (In EJB 3.0)

oskar.carlstedt Apr 23, 2007 10:03 AM (in response to grdzeli_kaci)

Hi All!

The intention of my last post was to explain why sequences, indexes etc. make the solution less performant.

I still agree with Felix. Huge bulk operations shall be done by using the tools provided by the database, in this case a stroed procedure.

//Oskar

24. Re: 10,000 Recorc Per Second (In EJB 3.0)

snau2005 Apr 23, 2007 10:19 AM (in response to grdzeli_kaci)

i found my old test, run it again, and got 40 000 entries in 11-13 seconds, but my table have only 4 columns no indexes etc ...

also i found time to write oracle procedure to do the same, 40 000 entries one procedure insert per 6 seconds (5 -7 seconds). So about twice fast as ejb implementation, i think quite good results for ejb.

25. Re: 10,000 Recorc Per Second (In EJB 3.0)

fhh Apr 23, 2007 10:35 AM (in response to grdzeli_kaci)

Can you post the code of your stored procedure?

40.000 records in 5-6 seconds is too low. I would expect much better performance.

Regards

Felix

26. Re: 10,000 Recorc Per Second (In EJB 3.0)

snau2005 Apr 23, 2007 10:52 AM (in response to grdzeli_kaci)

"fhh" wrote:
Can you post the code of your stored procedure?

40.000 records in 5-6 seconds is too low. I would expect much better performance.

Regards

Felix

here it is the function. Please ignore parameters (i just took one function as example, as you see i generate id's by myself (was lazy to create sequence :)). I use the same table (TUser as in ejb implementation)

FUNCTION GETTUSER(traderCode varchar2, userName varchar2) RETURN varchar2 is
 rez varchar2(4000);
 aaa number(10);
 iii number(10);

 begin
 select max(id) into aaa from TUser;
 FOR iii in (aaa + 1) .. (aaa + 40001) LOOP
 insert into TUser (id, userid, username, userdesc) values (iii, 'useridproc', 'usernameproc', 'userdesc'||iii);
 END LOOP;
 commit;
 rez := 'hoho and the bottle of rum';
 return rez;
end;

27. Re: 10,000 Recorc Per Second (In EJB 3.0)

grdzeli_kaci Apr 23, 2007 10:56 AM (in response to grdzeli_kaci)

ok , i'll try to explain why i need working from persistence layer, and don't want bulk operations,

1.i wrote so large EJB Application, i have distributed transactions and distributed databases,
i have 4 ejb applications which are working each other, and also i have two database : 1 - oracle 10g database for billing operations and 2 - oracle timesten database for live operations, my collaborator developers wrote one application using Threads and Oracle Timesten and this application send me Messages by Tibco Rendezvous (This is Network Application for Changing messages between some application, wrote different languages),

my program must be catch all messages and persist them into oracle database (also write oracle timesten database)

1. my entity persist operation is not so dummy (there is some selects, then some business logic, some validations and after all persist them) , if i choose bulk insert using stored procedures, then stored procedure must be contain this business logic which i have into java ( This is not good - this is not simple business logic !!!!!! on this case i will have two application : one - into java and second into oracle);

2, The second problem is that i want write sonme data into multiple database (one of them is oracle and second one is timesten into one transaction) , how is it possible using stored procedures.....

and maybe my first request 10,000 per second maybe is so many for persistence layer, but 1000 insert into oracle database without any business logic (only dummy insert),

It is not possible ? it is not good, i don't thing that it is not impossible (maybe my database need some performance), i am trying to do this like

oskar.carlstedt tald me

The data files are normally not optimized for either reading and/or writing.

i hear about optimization for this operation into Operation system, i will talk with my system and database administrators.

Regards,
Paata

28. Re: 10,000 Recorc Per Second (In EJB 3.0)

oskar.carlstedt Apr 23, 2007 11:10 AM (in response to grdzeli_kaci)

Ok, I nunderstand. Some questions:

Do you have a constant flow of messages? Do you need synchronus responses? If not, it might be possible for you to que these requests?

Is is a requirement to have the data published in the database right after you have returned a response?

//Oskar

29. Re: 10,000 Recorc Per Second (In EJB 3.0)

grdzeli_kaci Apr 23, 2007 11:18 AM (in response to grdzeli_kaci)

"oskar.carlstedt" wrote:
Ok, I nunderstand. Some questions:

Do you have a constant flow of messages? Do you need synchronus responses? If not, it might be possible for you to que these requests?

Is is a requirement to have the data published in the database right after you have returned a response?

//Oskar

hi Oskar,
My Problem is That it (some other program) send me messages and wait to answer (of course i have synchronus responses )
yes i know about ques but i can't use it :(
first i must return value and second multiple database persist in one transaction (i think you understand multiple database insert in one transaction - i have transaction marker and i want to persist both database insert with this markers)

Regards,
Paata