5 Replies Latest reply on Jun 15, 2012 1:52 AM by wdfink

    JBoss 4 Cluster (5 nodes) sends duplicated requests to Oracle database (10g)

    sumulu

      I am a DBA working on an application using JBoss 4.2.1 GA clustering MQ:

       

      I observered several same requests from database side, all these sessions with exact the same query (select query) was running and with same BIND vars caught for a large reports. The total number of duplicated sessions from database side was increasing if no action (picking off the duplicated sessions), and CPU climbs higher and higher to 100% within one minutes, cause aggressive oralce latch contention and other wait.

       

      This did not happen often, 4 times last week and 2 times 2 months ago, could be application related.

       

      Just want to figure out when the duplicated threads requests can happen from the jBoss clustering.

       

      Our cluster:

      Jboss 4.2.1 GA, 5 nodes windows 2003 servers 32bits

      A, B and C are the same, D and E has better CPU and RAM.

       

      Anyone who experienced the similiar issue, please kindly leave some input about how to target the issue and how to handle this issue - change settings.

       

      Thanks a lot,

       

      - Hank

      BTW: how to make this topic to public??? thanks

        • 1. Re: JBoss 4 Cluster (5 nodes) sends duplicated requests to Oracle database (10g)
          wdfink

          You should check together with the application developer where the query came from.

          I suppose there is a scheduler or a manual action to start this.

          • 2. Re: JBoss 4 Cluster (5 nodes) sends duplicated requests to Oracle database (10g)
            sumulu

            Hi, WDF,

             

            Thanks a lot for you reply, appreciated very much.

             

            The problem in our team is that the developer would like admit the duplicated sessions from JBoss clustered servers , I got shocked. but I am curious about to find out more . it is good for our application.

             

            There are different types transactions in our application:

            1. short transaction: change/insert and update/delete record, amount adjustment.

            2. long transaction: reports and long DSS query to get summary information as reference for furthur change.

            3. an independant process synchronize several databases

             

            We othen have:

            1. memory can not be GCed in clustered nodes and total memory went higher over the certain limit, like 1000M on a windows 32 bits server.

            2. connection pool exceeded default min, and over max sometimes, this could be one symptom of memory leak?

             

            From what I observed during the issue. the duplicated requests was related to user behavior, when we tried to reproduce issue and we were not able to reproduce it off-peak time. Issue seems came up in special situation. we imitated user behavior - click a butten to generate report which was generating PDF report file (suppose there is one file), once the file generated and it will be shown in a pop-up window, in my testing, report pop-up very quick, less than one second, there is no duplciated sessions shown either from database side.

             

            In the day with issue, we get feed back from users, they said, she clicked the button to view the reports and there pop-up windows was there but there is no content for awhile, so the user went back to main windows and repeatedly click the button multiple times. them I noticed the dulicated sessions when I got CPU database alert, I caught several dulicated requests, at beginning, dups come up form one jboss node then spread to several other nodes, the query is the reporting query, they're running same query with same BIND vars. Obviousely, the jboss did not work normal at the moment, I am not sure if jboss encountered the memory leak and exhaused the memory for the thread. Any possible for cluster to spread the request over to other nodes?

             

            Even in normal time, I can see like 4 PDF duplicated files for one report, supposed to have one, the files names listed as report_abc(1).pdf, report_abc(2).pdf ... report_abc(8).pdf. the most were generated at the same time, a few are one minute later.

             

            The report could be long or short all depending on the date ranges users selected for the report, I do not know if the long transactions may use thread go to multiple nodes to process the request, from database side, in normal time, I seldom see the duplicated requests, sometimes, I noticed slightly duplicated requests but less than 4 duplicated requests could not cause CPU% climbing in our database, 8 dupliated will cause 65%~85%, an addtional one will lead 10% CPU increase, a few more will hit 100%, I used to see over 27 dulicated sessions, and put database in freeze.

             

            all sesssion I mentioned here is Oracle session.

             

            My questions:

            1. when jBoss nodes had memory issue (failed GC), is it possible for the cluster to spread the large requests over to second or serveral nodes to get data from databases?

            2. the connections pool increased (database sessions from jBoss servers) over min~max ranges, are these connection going to work normally or partially? I doubt it will as I noticed sometimes, quicte a few long session showing idle in database, but it still holds the locks on some database tables and cause blocking to other DMLs.

            3. adjust GC interval will reduce memory leak on jBoss server ? is this good choice for a busy, small but handing large transaction? is there a memory limit for each thread in jBoss?

            4. will jBoss 5 fix some memory bugs and offer functional enhancement?

             

            Thanks a lot,

             

            - Hank

            • 3. Re: JBoss 4 Cluster (5 nodes) sends duplicated requests to Oracle database (10g)
              wdfink

              1)

              not simple, because there is no loadbalancing policy that respect it. And I suppose it is not easy to write one it will be an overhaead to measure it and provide it to the client.

              2)

              I there are still locks I suppose it will be an application issue that the locks are not freed, it can be that a transaction is not correctly commited or rolledback

              3)

              Adjusting GC can not reduce memory leak, a leak is an issue of not freed reference to objects, the GC will not remove it from memory.

              But if you mean to optimize the GC, yes this is possible but will be alway a continuous work.

              4)

              In JBoss5 there are different bugfixes and enhancements, but there are many applications still running JBoss4 without any presure to move.

              I would recommend to go directly to AS7 because this will be the base for further development and it will be faster than AS4/5.

              But it will take a bit planing and have migrations issues.

              • 4. Re: JBoss 4 Cluster (5 nodes) sends duplicated requests to Oracle database (10g)
                sumulu

                Wolf-Dieter Fink,

                 

                Thanks a lot, here is one more question.

                 

                Our application type are quite different, the most of the transactions are small and quick, but there are some large ones, like large calculation and large output transaction. for example, some applications generate very large output for PDF report,

                my question: is there a limit (max) on memory per thread in 32 bits jBoss server? when the memeory request exceeded a certain amount, could this be out of control and also cause memory shortage for other threads?

                Is there a good tool can monitoring memory issue and failed session on jBoss server?

                 

                Thanks a lot,

                 

                - Hank

                • 5. Re: JBoss 4 Cluster (5 nodes) sends duplicated requests to Oracle database (10g)
                  wdfink

                  There is no limitation of memory per thread but it has an effect to other running threads if you have not enough heap space.

                   

                  One thing that you can do in your code is to remove references, mean that you use variables in a limited scope {} where you need it or set references = null. This is to tell the JVM you don't need the object and the GC can remove it.

                  To understand how the memory works the Tuning GC page will be good to start.

                   

                  To monitor the memory consumption I prefer jstat, it's provided by the java installation and add not to much overhead during runtime. Other GUIs might use JVM heap during runtime (I've seen some bad issues that such tool bring the server down).