8 Replies Latest reply on Oct 2, 2015 2:48 PM by here4learn

    Widlfly ejb thread pool minitoring

    here4learn

      Hi,

       

      I have configured thread pool size of 500 in Widlfy standalone-full.xml, My web application(tomcat) communicates with Widlfy using remote JNDI EJB lookup. I have a question here, I tried to run a CLI Command to monitor the EJB thread pool, and below is the output of the same, If you see the "current-thread-count" is showing as 500, Is it supposed be like that.. or Have I messed up something in configuration.

      {

          "outcome" => "success",

          "result" => {

              "active-count" => 4,

              "completed-task-count" => 4795185L,

              "current-thread-count" => 500,

              "keepalive-time" => {

                  "time" => 5000L,

                  "unit" => "MILLISECONDS"

              },

              "largest-thread-count" => 500,

             "max-threads" => 500,

              "name" => "default",

              "queue-size" => 0,

              "rejected-count" => 0,

              "task-count" => 4795190L,

              "thread-factory" => undefined

          }

        • 1. Re: Widlfly ejb thread pool minitoring
          jaysensharma

          Hello,

           

          "active-count" =>  "The approximate number of threads that are actively executing tasks."  (In the current snapshot you have 4 active threads working on some tasks)

           

          "largest-thread-count" => "The largest number of threads that have ever simultaneously been in the pool."   (in your case we see that at some point of time there were 500 threads simultaneously been in the pool)


          "current-thread-count" => "The current number of threads in the pool.", Or we can say its the number of threads in the executor pool, Some of those might be executing some tasks and some of them might Not be executing any tasks rather sitting idle.

           

          NOTE:  Once threads are created and added to the pool then this count is incremented and once the threads are removed from the pool then the count should be reduced ideally. So in your case it looks like in peak load the thread count increased to 500 (which is your "max-threads size limit) and those threads remained in the pool as idle threads where as currently you have only 4 active threads as we see in the "active-count".

          • 2. Re: Widlfly ejb thread pool minitoring
            here4learn

            Hi Jay,

             

            Thanks for replying, I have one doubt, when you mentioned about "peak load the thread count increased to 500", Actually there was no load on the server, Infact one user was logged in, also even during the load test, I have never seen his value going up or down, It always remain constant, Please correct me.

             

            --

            Thanks

            • 3. Re: Widlfly ejb thread pool minitoring
              jaysensharma

              Hello Rocks,

               

                    You mentioned that "here was no load on the server, Infact one user was logged in" which looks little strange.

               

                    But in general, Looking at the stats you posted it looks like the your server was running for quite a long time As it has [ "task-count" => 4795190L ]    Out of which ["completed-task-count" => 4795185L,]

               

                      So looks like the  server was performing lots of tasks.  Also the "largest-thread-count" => 500, indicates the number of threads that have ever simultaneously been in the pool. So it looks like without your notice there was a scenario on the server when 500 simultaneously threads were created and present in the pool (usually in case of heavy task execution).   

                      

                


                   

              • 4. Re: Widlfly ejb thread pool minitoring
                here4learn

                Hello Jay,

                 

                Yes you were right initial it is zero. And one of our load runner server has executed some good number of transactions before I took the snapshot.

                 

                "active-count" => 0,
                   "completed-task-count" => 0L,
                   "current-thread-count" => 0,
                   "keepalive-time" => {
                       "time" => 5000L,
                       "unit" => "MILLISECONDS"
                   },
                   "largest-thread-count" => 0,
                   "max-threads" => 500,
                   "name" => "default",
                   "queue-size" => 0,
                   "rejected-count" => 0,
                   "task-count" => 0L,
                   "thread-factory" => undefined
                }

                 

                 

                But When I had run the CLI command for EJB thread pool, that time I was sure that it had only one user, Is there a way I can force unused threads to go back into the pool, Do you see any concern of the output pattern which I posted in my original post.

                Reason why I am asking is that my application communicates with wildfly EJB using remote JNDI look ups and performance has gone worse, Its putting threads into BLOCKED state(before I increased those to 500). Please see below exception stack trace, Could you please suggest.

                 

                Thread 15391: (state = BLOCKED)

                - java.lang.Object.wait(long) @bci=0

                - java.lang.Object.wait() @bci=2, line=502

                - org.jboss.ejb.client.EJBClientInvocationContext.awaitResponse() @bci=135, line=413 (Compiled frame)

                - org.jboss.ejb.client.EJBInvocationHandler.doInvoke(org.jboss.ejb.client.EJBInvocationHandler, boolean, java.lang.Object, java.lang.reflect.Method, java.lang.Object[], org.jboss.ejb.client.EJBClientContext) @bci=28, line=202 (Compiled frame)

                - org.jboss.ejb.client.EJBInvocationHandler.doInvoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=86, line=181 (Compiled frame)

                - org.jboss.ejb.client.EJBInvocationHandler.invoke(java.lang.Object, java.lang.reflect.Method, java.lang.Object[]) @bci=14, line=144 (Compiled frame)

                • 5. Re: Widlfly ejb thread pool minitoring
                  here4learn

                  Hi Jay,

                   

                  Looks like there is something wrong with our EJB pool, the active thread count is not at all reducing. I also checked one thing, when I made the first request via our app to widlfly, 70% of the threads count soar up, and in 5 mins all "current-thread-count" equals to 500, and active count was jumping between 0,1,0,1,2,1,0, I never see it going above 5, And when I took the thread dump, All "EJB Default 1" to "500" were waiting on condition, Please let me know what to fix here. I tried many options but no luck. Keep alive also not releasing these threads.. time is 5000 ms in config.

                   

                   

                  /subsystem=ejb3/thread-pool=default/:read-resource(include-runtime=true)                                                                   {                              

                      "outcome" => "success",

                      "result" => {

                          "active-count" => 0,

                          "completed-task-count" => 98057L,

                          "current-thread-count" => 500,

                          "keepalive-time" => {

                              "time" => 2000L,

                              "unit" => "MILLISECONDS"

                          },

                          "largest-thread-count" => 500,

                          "max-threads" => 500,

                          "name" => "default",

                          "queue-size" => 0,

                          "rejected-count" => 0,

                          "task-count" => 98057L,

                          "thread-factory" => undefined

                      }

                  • 6. Re: Widlfly ejb thread pool minitoring
                    here4learn

                    Hi Jay,

                     

                    Could you please put some light on my query, I don't see the active thread count going down even if application is idle.. "keepAliveTime" not working as expected.

                    • 7. Re: Widlfly ejb thread pool minitoring
                      jaysensharma

                      Hello Rocks,

                         This is an expected behavior. Please see the "$WILDFLY_HOME/docs/schema/jboss-as-ejb3_2_0.xsd"

                       

                         A thread pool executor with an unbounded queue.  Such a thread pool has a core size and a queue with no upper bound.  When a task is submitted, if the number of running threads is less than the core size, a new thread is created.  Otherwise, the task is placed in queue.

                       

                         In general, default ejb pool uses an unbounded queue executor, Which has a core thread count, queue, and max threads. Threads beyond the core thread count are created only if the queue is full. The core threads never time out.

                       

                         You may have a question then what is the timeout usage?

                             >>> The threads which are created between the core thread and max thread limit only those gets timeout timed out.

                       

                         A bounded queue never fills so the unbounded queue executor essentially has core threads equal to max threads. Since all threads are core threads, none of them time out.

                      • 8. Re: Widlfly ejb thread pool minitoring
                        here4learn

                        Hi Jay,

                         

                        I see, That makes sense, One final question, is there any recommended default value for max thread size.