1 2 Previous Next 29 Replies Latest reply on Dec 13, 2010 11:06 PM by jaikiran

    JBoss 6 100% CPU usage after scanning

    nick.x.newman

      Hi,

       

      My instance of JBoss is subjected to external scanning to check for vulnerabilities, and after these scans it often ends up in a state in which it seems to work fine, but uses just over 100% of a CPU (on a 4 CPU box) thereafter.  I'd like to avoid this unnecessary CPU load.

       

      The first thing I did to investigate was to use the JMX console to look at the thread CPU usage.  That showed that threads named pool-2-thread-2 through pool-2-thread-13 had very high CPU time (e.g. 2917560 ms and rising), so it's clear that these are the problems.  (But pool-2-thread-1 had only 1490 ms used, for some reason.)

       

      Oddly, when I ask the same MBean to display a thread dump, these threads seem not to be there.  So I obtained a thread dump using kill -3 uid (it's on a Linux box) .  To avoid cluttering things up I have included that dump as an attachment rather than pasting it all here.  However, here is a sample of one of these threads.  It seems to be typical of this pool-2-thread-x group except that thread-4 is "runnable" rather than "waiting on condition".

       

      "pool-2-thread-13" prio=10 tid=0x082e4000 nid=0x58dc waiting on condition [0xbed9e000]
         java.lang.Thread.State: WAITING (parking)
              at sun.misc.Unsafe.park(Native Method)
              - parking to wait for  <0xd8a9ce60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)
              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2061)
              at org.jboss.threads.QueueExecutor.takeTask(QueueExecutor.java:725)
              at org.jboss.threads.QueueExecutor.access$400(QueueExecutor.java:45)
              at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:814)
              at java.lang.Thread.run(Thread.java:619)
              at org.jboss.threads.JBossThread.run(JBossThread.java:122)

      "pool-2-thread-13" prio=10 tid=0x082e4000 nid=0x58dc waiting on condition [0xbed9e000]

         java.lang.Thread.State: WAITING (parking)

              at sun.misc.Unsafe.park(Native Method)

              - parking to wait for  <0xd8a9ce60> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

              at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)

              at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)

              at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:778)

              at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2061)

              at org.jboss.threads.QueueExecutor.takeTask(QueueExecutor.java:725)

              at org.jboss.threads.QueueExecutor.access$400(QueueExecutor.java:45)

              at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:814)

              at java.lang.Thread.run(Thread.java:619)

              at org.jboss.threads.JBossThread.run(JBossThread.java:122)

       

      But having got this far I'm not sure how to proceed.  If anyone has any ideas (other than telling my company to quit being so paranoid and to lay off the scanning) I'd love to hear them.

       

      Thanks,

      Nick

        • 1. Re: JBoss 6 100% CPU usage after scanning
          genman

          I've seen something similar as well in some situations. I can't explain what triggers this, however. I'll post a thread dump if I find it.

          • 2. Re: JBoss 6 100% CPU usage after scanning
            nick.x.newman

            I've seen several reports of this type of problem on the web, but not much in the way of positive solutions.  I'm glad it's not just me.  (Misery loves company!)

             

            Perhaps if several people have this same problem it should be raised as a bug rather than just a discussion.

            • 3. Re: JBoss 6 100% CPU usage after scanning
              jaikiran

              Can you attach the entire thread dumps (take around two or three dumps at around 5 seconds interval) when you see the 100% CPU usage?

              • 4. Re: JBoss 6 100% CPU usage after scanning
                dimitris

                I am curious what do you expect to find by scanning JBoss (or any Java application for that matter) for vulnerabilities?

                 

                Does the 100% cpu persist for a long time after the scanning?

                • 5. Re: JBoss 6 100% CPU usage after scanning
                  nick.x.newman

                  Hi Dimitris,

                   

                  The vulnerabilities don't have to be in JBoss itself, and don't have to present ways to hack through to the OS.  For instance, perhaps I left the JBoss JMX and Admin consoles wide open. Or perhaps I deployed a badly coded web application that contained a SQL injection flaw.  That sort of thing.

                   

                  Regarding how long it stays at 100%, well, it's still there, so I assume it's a permanent condition.  (Luckily aside from the CPU drain things still seem to work, so I left it in case I had requests for extra information.)

                   

                  Nick

                  • 6. Re: JBoss 6 100% CPU usage after scanning
                    dimitris

                     

                    The vulnerabilities don't have to be in JBoss itself, and don't have to present ways to hack through to the OS.  For instance, perhaps I left the JBoss JMX and Admin consoles wide open. Or perhaps I deployed a badly coded web application that contained a SQL injection flaw.  That sort of thing.

                     

                    Do scanner actually catch that sort of thing?

                     

                    Could you try something? Exclude from the scaners the directories watched for hot deployment and see if the still raising the CPU.

                    • 7. Re: JBoss 6 100% CPU usage after scanning
                      nick.x.newman

                      Hi Dimitris,

                       

                      Well, it depends on the scanner of course.  But certainly some scanners are clever enough to search for that sort of thing.  I see it trying all sorts of SQL-inject types of command (e.g. constructs with mismatched single quotes) and also malformed XML of various types, etc.

                       

                      I can't help you with the second part.  I do not control what gets scanned or when.  But I believe that the scanning is only for the I/O ports, not the disk.

                       

                      Nick

                      • 8. Re: JBoss 6 100% CPU usage after scanning
                        jaikiran

                        By the way, is that scanning tool named Nessus? I have seen some other users report, in these forums, similar issues with that tool against JBoss AS.

                        • 9. Re: JBoss 6 100% CPU usage after scanning
                          dimitris

                          All right, you are talking about port scanning, I missunderstood that. Then examining the stack trace is really the way to go.

                          • 10. Re: JBoss 6 100% CPU usage after scanning
                            nick.x.newman

                            Hi Jaikiran,

                             

                            Yes, I think that Nessus is indeed the tool.

                             

                            I'm sorry that it has taken me a while to respond, but I had disconnected from the terminal session and that makes it very tricky to get the thread dumps that you requested.  (Especially since I couldn't restart JBoss without losing the weird state it is in).

                             

                            In the end I decided that I had to write a servlet to get what you wanted and then deploy it into the running system.  So to make sure that you know exactly what it is that I'm giving you I have attached the code that generates the output as well as the output itself.

                             

                            If I need to make a change to the code that produced the dumps please let me know.

                             

                            Thank you for helping with this!

                            Nick

                            • 11. Re: JBoss 6 100% CPU usage after scanning
                              nick.x.newman

                              I think this may be the other report that you mentioned.  http://community.jboss.org/message/540415

                              • 12. Re: JBoss 6 100% CPU usage after scanning
                                nick.x.newman

                                Hi Jaikiran,

                                 

                                I think I have some more input on this that may be useful.

                                 

                                The affected JBoss servers had the AJP port open during these scans, and various exceptions were thrown during the scan (Invalid message, EOFException, etc).

                                 

                                Another JBoss instance (presumably subjected to the same scans) had the AJP port listener removed and it was NOT affected by this High CPU usage.

                                 

                                I removed the AJP listeners from the affected servers, restarted them, and so far they have been fine.

                                 

                                So it seems likely that the problem is triggered by various sorts of rubbish being thrown at the AJP port by the Nessus scanner., and removing the listener for that port (assuming that you have no need of it) solves the problem.


                                Does that throw any light on a potential solution?  (For me the problem is resolved, but for those that need the AJP there may still be problems)

                                 

                                Thanks,

                                Nick

                                 

                                Message was edited by: Nick Newman.  See entry below.

                                • 13. Re: JBoss 6 100% CPU usage after scanning
                                  jaikiran

                                  I haven't actually been able to look into the details of this yet. Sorry about that. Once I'm done with some of my pending work, I'll take a look at this and see what's going on. And thank you for the details about the AJP ports - that's going to help narrow down the investigation.

                                  • 14. Re: JBoss 6 100% CPU usage after scanning
                                    nick.x.newman

                                    Hi Jaikiran,

                                     

                                    I'm sure you're very busy.  The delay is not a problem for me.  I just appreciate your help!

                                     

                                    Unfortunately it seems that I was wrong about the AJP port.  The servers with the AJP listeners removed did in fact go back to 100% CPU.  I think I just hadn't waited long enough for them to get scanned.

                                     

                                    As for the server that is not affected - I have indirect evidence that it is getting scanned (I see exceptions very like those thrown during a scan) but it never seems to be affected in the same way. The main difference I can think of (other than the applications being served) is that the unaffected JBoss is on a virtual machine, so perhaps it has different performance characteristics that might affect timing issues.

                                     

                                    So I am back to having no ideas about this.  But I do have some more input.  I had configured access logs to be kept on the affected JBoss so that we have a trace of what Nessus tried.  I also have the server logs indicating the exceptions that were thrown.

                                     

                                    It seems odd that the first exceptions are thrown before the first access attempt is recorded, but perhaps Nessus tries some ill-formed requests first just to see what ports are open.

                                     

                                    I'm not sure if these logs will be of much use, but they are attached.

                                     

                                    Thanks,

                                    Nick

                                    1 2 Previous Next