Performance problems Jboss 4.2.3 & Seam 2.1, JPA, RichFaces & Primefaces

[asterisco69]

Everybody have a nice day, this is my first post. Our company provides software development services and we also host the applications that we develop. We have right now 2 web POS systems that are heavily used (like 200 sales per day per app registered) and another 2 applications with moderate use. Our server is a Centos 6.4 x64 with 4GB of RAM and Core i5. We have Jboss 4.2.3 and our applications are developed using Seam 2.1 with JPA, Richfaces mostly & Primefaces, but the applications are too slow almost always and i don't know where to start to find out what the problem is. For example, we have a table with hundreds of registers, using richfaces table, and i don't know if is because lack of memory or something like that but in some moment the pagination doesn't work, i press the pagination buttons but it does nothing or the filters in the header  columns of the table doesnt work, but if i restart the server it's like it releases memory and it works normal, but after a while it runs out of memory or something like that and i have the same problems. The applications are too slow, also sometimes when i wanna save an information or update it the buttons doesnt work.

I don't know if is an issue related with the use of JPA or because the server isn't releasing memory when objects are not in use or something like that, as i said before i don't know where to start to discover where the bottleneck (or bottlenecks) is, but it worries me a lot because we aren't able to host more applications if those applications will become slower, it could cause a bad UX. Please help me or guide me about how to find bottlenecks in our apps. If you need some file or something of my Jboss config or my seam apps config just tell me.

Regards.

[baddeley84]

If you are using the rich:dataTable, its probably slow because the default filters used in RF 3.x do not use database pagination so although you may only be looking at 10 items in the table, the backing bean will have actually loaded the entire table from the database and then presents you with a filtered view.

I got around this by developing a new 'paged list' class that allows pagination based on this blog entry http://backtothefront.net/blog/2010/08/27/richfacesseam-server-side-paging-datatabledatascroller/

Actually works quite well and improves performance for large tables

[asterisco69]

Thanks for the tip David, actually i have a couple of catalogs (for example product sales, accounting accounts, and so on) that normally have like 17000 rows and exactly as you said, it loads the entire list even when you only see the first X rows (normally 10). I gonna check and implement your solution in the catalogs that have more rows, i'm thinking about writing some abstract classes based on your ItemsBean class just to inherit the methods to the controllers that needs them.

[asterisco69]

Well, i tried it and it works excellent, the only problem is that column filters doesnt work, i think because when i enter a criteria search in the column filters, it searchs it in the current X rows that i'm seeing. It would be awesome if when i use the filters it could re-run the query, but in order to do that i would have to embed the filter parameters to the HQL query.

[baddeley84]

Yes unfortunately you cannot use named parameters in the 'where' clause with HQL, I normally end up have a few different query's for each filter

Glad it worked ok, btw. I think PrimeFaces has database pagination built in to their datatables?

[asterisco69]

Also i wanna talk you about some of the problems that our apps have, to see if you know about what the problem is or if you have a clue. The main problems are:

• When i have a rich data table, even tables with 100 rows or something like that, filters doesn't work nor pagination.
• When i press a button that does an action like update, insert or delete a register (in a detail form) it doesn't react. Maybe after a while, if i press it again, it works.
• In general, applications are slow

Our applications are at a certain point normal, i mean, it's just a web app built in seam 2 that inserts, deletes, updates and queries against a postgresql database, it doesn't do complex calculations or things that require extreme process, our database doesn't have stored procedures or jobs running. This happens after some hours of working, if i restart the server after restarting it applications works normally without problems, it's something like degenerating with the time. This is the output when i type command top on linux:

top - 13:36:36 up 2 days, 15:33,  1 user,  load average: 0.00, 0.03, 0.08

Tasks: 162 total,   1 running, 161 sleeping,   0 stopped,   0 zombie

Cpu(s):  0.1%us,  0.1%sy,  0.0%ni, 99.8%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st

Mem:   3833320k total,  3715328k used,   117992k free,     5760k buffers

Swap:  2097144k total,    31204k used,  2065940k free,   562880k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

2081 jboss     20   0 5714m 2.8g 5880 S  1.1 76.8 130:06.83 java

    1 root      20   0 19360  968  748 S  0.0  0.0   0:00.81 init

    2 root      20   0     0    0    0 S  0.0  0.0   0:00.00 kthreadd

    3 root      RT   0     0    0    0 S  0.0  0.0   0:00.80 migration/0

    4 root      20   0     0    0    0 S  0.0  0.0   0:00.38 ksoftirqd/0

    5 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/0

    6 root      RT   0     0    0    0 S  0.0  0.0   0:00.15 watchdog/0

    7 root      RT   0     0    0    0 S  0.0  0.0   0:00.02 migration/1

    8 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/1

    9 root      20   0     0    0    0 S  0.0  0.0   0:00.38 ksoftirqd/1

   10 root      RT   0     0    0    0 S  0.0  0.0   0:00.13 watchdog/1

   11 root      RT   0     0    0    0 S  0.0  0.0   0:00.01 migration/2

   12 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/2

   13 root      20   0     0    0    0 S  0.0  0.0   0:00.36 ksoftirqd/2

   14 root      RT   0     0    0    0 S  0.0  0.0   0:00.13 watchdog/2

   15 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/3

   16 root      RT   0     0    0    0 S  0.0  0.0   0:00.00 migration/3

   17 root      20   0     0    0    0 S  0.0  0.0   0:00.32 ksoftirqd/3

   18 root      RT   0     0    0    0 S  0.0  0.0   0:00.13 watchdog/3

...

And also recently i activated remote monitoring with visualVM, but honestly i don't know how to read or interpret the graphics. There are some screenshots.

I know that it may sound like, i don't know, like "solve my problem" but it isn't, i'm just worried about this because it worries me that our server doesn't support the load and we weren't able to host more applications without fearing that apps will become slower or non functional, and even when i know how to program in JAVA i don't have knowledge about how to monitor apps or determine a bottleneck in our apps or interpret heaps and things like that, so any help, guide or clue will be appreciated.

Regards

[baddeley84]

I think you will need to dig into the code to troubleshoot this, how is the application packaged? .ear or .war? how long does the server take to startup?

Personally it sounds like the database has outgrown the application and you will need to start looking at your database queries to see where the bottlenecks are, are you using Hibernate?

If you dont want to dig into the code there are tools you can use to see which SQL queries are causing problems (I have used NewRelic for this)

You may also want to check how much data is being stored in the HTTP session (Conversation / Session scoped components) as this will slow things down

David

[asterisco69]

Thanks for your answer David. The applications are EAR. Right now there are 5 applications hosted in the server, and the startup time is short, like 1 minute (i think that is a short startup time, but correct me if i'm wrong) . I'm using JPA with Hibernate. Also i'm going to check that tool New Relic. About data stored in the HTTP session, where i can see that? In the browser with some tool like Firebug in Firefox? Or is on the server side, using VisualVM or another tool?

About what you said of database outgrown, you talk about size? becuase the size of the biggest databases that we have is like 40mb - 60mb, which i consider very small, but i will start looking database queries, i gonna log them in postgresql log.

Regards.