1 Reply Latest reply on May 24, 2014 2:08 AM by toxicbytes

    JBoss AS 7.1.1 final - getting slow over time

    vidyasagar.atmakuri

      I am facing a strange is JBoss AS 7.1.1 Final. I have a spring + hibernate based web application. After deployment all transactions are quick. But in a few days any transaction that involves updates to DB seem to become extremely slow. What would take 4 seconds, takes 40 seconds.

       

      Selects (read-only transactions) seem to be not affected by this problem. Their performance remains largely consistent.

       

      This happens in our production server only. I have tried to recreate it in a lower environment, but it simply does not happen.

       

      Server restart brings performance back to normal. But in a few days it becomes slow again. I use a MySQL database in production as well as non-production. I do not suspect MySQL because, JBoss restart is resolving the slow down. I asked the production DBA, and they do not see any deadlocks or bottlenecks on the DB side. My JBoss connection pool never goes beyond 10 used connections.

       

      In production I have -Xms512m -Xmx2048m

      In my test server I only have -Xms64m -Xmx512m - and it still does not slow down. Did not see any slow down on consistent production like use for 10 days. Production normally gets slow by 3rd or 4th day.

       

      I have reviewed jmap heapdumps taken every night. There does not seem to be a leak, as the utilization comes back to normal very regularly. The time spent on GC is also very less. I have reviwed jstack thread dumps and they all seem normal. CPU utilization is always below 10% and Memory (RAM) utilization is always below 30%.

       

      The only thing that stands out is - UPDATES only are becoming slow.

       

      Any ideas how I can debug this? Is there any specific log message I can look for? In production my log level is ERROR and I do not want to change it for all classes.

       

      Thanks in advance for any pointer that can help me crack this issue.

       

      Best Regards,

      sagar

        • 1. Re: JBoss AS 7.1.1 final - getting slow over time
          toxicbytes

          Did you check the growth in the number of threads? Are the threads growing exponentially? You might have to synchronize cleanup of related classes. You can also open the JMX management interface for your application and view all that is happening using VisualVM or JConsole. If everything is transactional and stateless as you say, there should be no reason for the old generation to grow up to the ceiling of the old generation space; most of the GC (minor) should occur in the young generation space of the heap. You can use commercial (or, open source) Java profiler tools to identify which methods have performance issues.