8 Replies Latest reply on Aug 22, 2005 7:45 PM by genman

    Low CPU Utilization on Linux

    jizzoe

      I'm facing a problem where we are running JBoss 3.2.6 on Linux 9.0 in a clustered production environment that recieives very high traffic. Each box has 4 processors.

      We are noticing that each CPU never goes above 25% CPU Utilization. Even with high loads and after removing all but one box from the cluster.

      Has anyone experienced anything similar? We moved to Red Hat Advanced Server, but to no avail. I saw a BEA forum that says that if the thread count is too low (for WebLogic), the CPU will never reach 100% utilization. Sounds reasonable. Could the same be true for JBoss? If so, how do you set the thread count?

      Any help will be greatly appreciated.

      Thanks!

        • 1. Re: Low CPU Utilization on Linux

          jizzoe;

          Can you provide more information ?

          Is the application measurably slow ? If so, how do you know ? If the CPU utlil. is low, perhaps the load you are applying is simply insufficient to keep the boxes busy.

          If the app is slow, you need to figure out where the botleneck is. Last time I saw this particular symptom, it was a misconfigured NIC on the database server. This was resulting in very slow DB queries, but no visible contention on the app servers or the DB server.

          //Nicholas

          • 2. Re: Low CPU Utilization on Linux
            jizzoe

            Hi Nickman,

            Thanks for the response. Yes, we are seeing slowdown, measurably. The CPU goes to 100%, but it's not the expected way. We have 4 processors. Each processor gets up to 25% (never any higher): 100% = 25% X 4. Once this threshold is reached, JBoss crawls. This is observed in a production system for a highly public, well-known, and high-traffic application. This application is middle-ware business logic only (EJBs, POJOs, etc). There is no web server or JSP/servlet load on this box. And it's not very DB intensive.

            So, the load is sufficient and we have ruled out the DB as the bottle-neck. We have never seen this application (or any other of our production apps running on JBoss) produce a CPU utilization higher than 25% per processor.

            Any ideas? Has anyone ran into a similar situation? Does anyone have experience running JBoss on multi-CPU box running Linux 9.0? What do you think about the possibility that we may have our thread count too low, so that it could be maxing out our threads, but never fully utilizing our CPUs?

            As always, any help is greatly appreciated.

            --Joe

            • 3. Re: Low CPU Utilization on Linux
              thoennes

              How do you determine CPU utilization?

              Perhaps it is scaled by the number of processors, so 25% already means maximum usage?

              Cheers, Jörg

              • 4. Re: Low CPU Utilization on Linux

                I might be guessing but do you have a hyperthreading CPU or do you actually have 4 cpu's. Could this not explain what you observe?

                • 5. Re: Low CPU Utilization on Linux

                  jizzoe;

                  Just so we undestand what the CPU utilization is, can you post a sar -u or a top output ?

                  //Nicholas

                  • 6. Re: Low CPU Utilization on Linux
                    gozilla

                    jizzoe

                    What king of JVM are you running ?

                    Aren't you running it in green thread mode ?

                    Gozilla

                    • 7. Re: Low CPU Utilization on Linux
                      jizzoe

                      Hi Everyone,

                      Thanks for the responses. I will try to answer all questions to help me get to the bottom of this.

                      By CPU Utilization, I mean we are never seeing all 4 CPUs go above a combined amount of 120%. To illustrate, here are a few "top" printouts:

                      CPU0 states: 15.1% user, 3.3% system, 0.0% nice, 81.1% idle
                      CPU1 states: 15.2% user, 2.4% system, 0.0% nice, 81.2% idle
                      CPU2 states: 6.0% user, 1.2% system, 0.0% nice, 92.1% idle
                      CPU3 states: 64.3% user, 7.2% system, 0.0% nice, 27.3% idle


                      CPU0 states: 7.3% user, 3.2% system, 0.0% nice, 89.0% idle
                      CPU1 states: 6.1% user, 1.3% system, 0.0% nice, 91.4% idle
                      CPU2 states: 1.0% user, 0.4% system, 0.0% nice, 98.0% idle
                      CPU3 states: 83.2% user, 3.4% system, 0.0% nice, 12.2% idle



                      CPU0 states: 6.4% user, 1.1% system, 0.0% nice, 91.3% idle
                      CPU1 states: 12.1% user, 1.4% system, 0.0% nice, 86.0% idle
                      CPU2 states: 3.3% user, 1.3% system, 0.0% nice, 94.2% idle
                      CPU3 states: 76.1% user, 3.3% system, 0.0% nice, 19.4% idle


                      We have 4 CPUs, not 1 Hyperthreaded CPU. We are using the standard Linux JDK 1.4.2 from Sun with the standard VM args in JBoss's run.sh, so as far as I know, we're not using Green Threads.

                      What I'm looking for is something similar to this "top" output (NOTE: not a real output, just something that would be ideal)

                      CPU0 states: 85.4% user, 6.3% system, 0.0% nice, 7.1% idle
                      CPU1 states: 94.4% user, 4.0% system, 0.0% nice, 0.5% idle
                      CPU2 states: 84.3% user, 7.3% system, 0.0% nice, 7.2% idle
                      CPU3 states: 86.2% user, 7.0% system, 0.0% nice, 6.1% idle

                      Is there anything in the JDK or JBoss that consciously prevents us from using all available CPU? Any other ideas or tuning suggestions or light shed is greatly appreciated.

                      Thanks,

                      Joe

                      • 8. Re: Low CPU Utilization on Linux
                        genman


                        Some operations in JBoss or your O/S may not be very parralelizable, blocking the other CPUs. There are tools to find out where your application (or JBoss) may be blocking the other CPUs, such as JProbe or JProfiler.