4 Replies Latest reply on Feb 6, 2017 10:53 PM by jbertram

    500 ActiveMQ Global Threads on Java Clients

    pinkushn

      I am upgrading my client/server app from old JBoss to WildFly 10 and running into some memory problems that may be related to the shift from much-beloved HornetQ to ActiveMQ Artemis.  My client application has a JMS-based subscription to a topic on the WildFly 10 server, and this happily functions fine.  The problem arises over time as more messages get consumed.  It appears that with more incoming topic messages, more ActiveMQ 'global threads' spawn and persist until the point that there are 500 such threads costing about 1/3 meg per thread.  This is new territory for me, as this does not occur in my old HornetQ-based configuration.

       

      I would like to limit the number of ActiveMQ threads that are allowed to spawn.  I'm supposing there is a pool with a 500 count limit?  How can I dramatically limit this?

       

      In case this helps, here is the relevant portion of my standalone.xml and also the jndi properties I use to look up the RemoteConnectionFactory.  I've used bold to show one failed attempt to control threads via properties in the factory lookup on the client (based on information in the ActiveMQ docs).

       

      standalone.xml

       

      <subsystem xmlns="urn:jboss:domain:messaging-activemq:1.0">

                  <server name="default">

                      <security enabled="false"/>

                      <management notification-address="jms.queue.notificationsQueue"/>

                      <journal file-size="102400"/>

                      <transaction timeout="900000"/>

                      <shared-store-master/>

                      <security-setting name="#">

                          <role name="guest" delete-non-durable-queue="true" create-non-durable-queue="true" consume="true" send="true"/>

                      </security-setting>

                      <address-setting name="#" message-counter-history-day-limit="10" page-size-bytes="2097152" max-size-bytes="10485760" expiry-address="jms.queue.ExpiryQueue" dead-letter-address="jms.queue.DLQ"/>

                      <http-connector name="http-connector" endpoint="http-acceptor" socket-binding="http"/>

                      <http-connector name="http-connector-throughput" endpoint="http-acceptor-throughput" socket-binding="http">

                          <param name="batch-delay" value="50"/>

                      </http-connector>

                      <in-vm-connector name="in-vm" server-id="0"/>

                      <http-acceptor name="http-acceptor" http-listener="default"/>

                      <http-acceptor name="http-acceptor-throughput" http-listener="default">

                          <param name="batch-delay" value="50"/>

                          <param name="direct-deliver" value="false"/>

                      </http-acceptor>

                      <in-vm-acceptor name="in-vm" server-id="0"/>

                      <jms-queue name="ExpiryQueue" entries="java:/jms/queue/ExpiryQueue"/>

                      <jms-queue name="DLQ" entries="java:/jms/queue/DLQ"/>

                      <jms-queue name="TeacherToServer" entries="java:jboss/exported/jms/queue/TeacherToServer"/>

                      <jms-queue name="StudentToServer" entries="java:jboss/exported/jms/queue/StudentToServer"/>

                      <jms-queue name="notificationsQueue" entries="java:jboss/exported/jms/queue/notificationsQueue"/>

                      <jms-topic name="ToTeachers" entries="java:jboss/exported/jms/topic/ToTeachers"/>

                      <jms-topic name="ToStudents" entries="java:jboss/exported/jms/topic/ToStudents"/>

                      <jms-topic name="ApproveDenyEnrollment" entries="java:jboss/exported/jms/topic/ApproveDenyEnrollment"/>

                      <connection-factory name="InVmConnectionFactory" entries="java:/ConnectionFactory" connectors="in-vm"/>

                      <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector"/>

                      <pooled-connection-factory name="hornetq-ra" transaction="xa" entries="java:/JmsXA java:jboss/DefaultJMSConnectionFactory" connectors="in-vm"/>

                  </server>

              </subsystem>

       

      client side setup

       

      ...

       

        Properties props = new Properties();

        props.put(Context.INITIAL_CONTEXT_FACTORY, "org.jboss.naming.remote.client.InitialContextFactory");

        props.put(Context.PROVIDER_URL,"http-remoting://************:8080");

        props.put(Context.SECURITY_PRINCIPAL, "*******");

        props.put(Context.SECURITY_CREDENTIALS, "*******");

        props.put("jboss.naming.client.ejb.context", true);

      //Below is a failed attempt at limiting client side threads based on ActiveMQ docs

      //props.put("connection.ConnectionFactory.useGlobalPools", false);

      //props.put("connection.ConnectionFactory.scheduledThreadPoolMaxSize", 10);

      //props.put("connection.ConnectionFactory.threadPoolMaxSize", 50);

       

       

        InitialContext ctx = new InitialContext(jndiProps);

        ConnectionFactory factory = (ConnectionFactory) ctx.lookup("jms/RemoteConnectionFactory");

       

         //following field is of type javax.jms.Connection

        jmsConnection = factory.createConnection();

       

      //This is the Topic that appears to spawn the pile of threads

        Session session = jmsConnection.createSession(false, TopicSession.AUTO_ACKNOWLEDGE);

        Topic topic = (Topic) ctx.lookup("jms/topic/ToTeachers");

        MessageConsumer subscriber = session.createConsumer(topic, MessageFactory.getSelector(teacher));

        subscriber.setMessageListener(listener);

       

        teacherToServerSession = jmsConnection.createSession(false, QueueSession.AUTO_ACKNOWLEDGE);

        teacherToServerQueue = (Queue) ctx.lookup("jms/queue/TeacherToServer");

        teacherToServerProducer = teacherToServerSession.createProducer(teacherToServerQueue);

       

        teacherToStudentsSession = jmsConnection.createSession(false, Session.AUTO_ACKNOWLEDGE);

        topic = (Topic) ctx.lookup("jms/topic/ToStudents");

        teacherToStudentsProducer = teacherToStudentsSession.createProducer(topic);

       

        jmsConnection.setExceptionListener(new ExceptionListener() {

             ...

        });

       

        jmsConnection.start();

       

      Thanks in advance for any insights!

        • 1. Re: 500 ActiveMQ Global Threads on Java Clients
          jbertram

          You were on the right track with the changes you made in your initial context properties.  These properties need to be set connection factory you're actually looking up in JNDI, e.g.:

           

          <connection-factory name="RemoteConnectionFactory" entries="java:jboss/exported/jms/RemoteConnectionFactory" connectors="http-connector" use-global-pools="false" thread-pool-max-size="50" scheduled-thread-pool-max-size="10" />

           

          The Artemis documentation references the initial context properties because it uses a client-side only JNDI implementation which means things need to be configured in the context itself.  However, Wildfly uses a server-side JNDI implementation which means things are configured on the server.

           

          Try that out and let me know how it goes.

          • 2. Re: 500 ActiveMQ Global Threads on Java Clients
            pinkushn

            Thanks, Justin.  Exactly what I needed.

            • 3. Re: 500 ActiveMQ Global Threads on Java Clients
              pinkushn

              Hello, Justin.  I'm chiming in again with two hopefully simple follow-up questions.

               

              #1) Does each AMQ client side thread result in a separate message-consuming potentiality?  In other words, if I set thread-pool-max-size to 50, does that mean 50 threads end up concurrently checking for messages to consume?

               

              I ask because I'm noting that on unexpected AMQ connection failure on the server-side, I see a huge stack of AMQ 'connection failure has been detected' all from the same address but different ports.  This appears to be when ONE client terminates abruptly.

               

              #2) If I reduce the thread pool size on the client, is there a way to probe on the client side to see if it is failing to keep up with available messages on the server side?

               

              Thanks.  My little company (Get More Math) deeply appreciates WildFly and it's starting to look we'll be able to cross to EAP soon!

              • 4. Re: 500 ActiveMQ Global Threads on Java Clients
                jbertram

                #1) Does each AMQ client side thread result in a separate message-consuming potentiality? In other words, if I set thread-pool-max-size to 50, does that mean 50 threads end up concurrently checking for messages to consume?

                No.

                 

                #2) If I reduce the thread pool size on the client, is there a way to probe on the client side to see if it is failing to keep up with available messages on the server side?

                You can simply look at the queue where the consumer is attached and keep track of the message count.  If the message count rises steadily then that indicates the consumer isn't keeping up with the load.