7 Replies Latest reply on Mar 26, 2008 8:59 AM by fretzlaff_fretzlaff

    Problem on load test with FUSE MB

    fretzlaff_fretzlaff

      Hi,

       

      I have the same problem as described in AMQ-1611 using FUSE MB 5.0.0.9 and FUSE SMX 3.3.0.8.

      I'm using SMX with JCAFlow for failover, I have about 70 activationSpecs and try the send 100 files to each Bindind components (25 at all) that starts the flow .

      When the first lot of messages has processed I send more 100 messages and then a get the error bellow.

      Anybory knows what I can do to solve this problem?

       

      Thank's

       

      [ERROR] [17/Mar/2008 20:01:43:123] Failed to checkpoint a message store: java.util.concurrent.ExecutionException: java.io.IOException: Already started.
      h3java.util.concurrent.ExecutionException: java.io.IOException: Already started.
              at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:222)
              at java.util.concurrent.FutureTask.get(FutureTask.java:83)
              at org.apache.activemq.store.journal.JournalPersistenceAdapter.doCheckpoint(JournalPersistenceAdapter.java:398)
              at org.apache.activemq.store.journal.JournalPersistenceAdapter$1.iterate(JournalPersistenceAdapter.java:119)
              at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:118)
              at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:42)
              at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:885)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
              at java.lang.Thread.run(Thread.java:619)
      Caused by: java.io.IOException: Already started.
              at org.apache.activemq.store.jdbc.TransactionContext.begin(TransactionContext.java:148)
              at org.apache.activemq.store.jdbc.JDBCPersistenceAdapter.beginTransaction(JDBCPersistenceAdapter.java:356)
              at org.apache.activemq.store.journal.JournalPersistenceAdapter.beginTransaction(JournalPersistenceAdapter.java:193)
              at org.apache.activemq.util.TransactionTemplate.run(TransactionTemplate.java:41)
              at org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:258)
              at org.apache.activemq.store.journal.JournalMessageStore.checkpoint(JournalMessageStore.java:233)
              at org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:368)
              at org.apache.activemq.store.journal.JournalPersistenceAdapter$4.call(JournalPersistenceAdapter.java:367)
              at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
              at java.util.concurrent.FutureTask.run(FutureTask.java:138)
              ... 3 more
      

       

      My config:

       

              <sm:flows>
                      &lt;sm:jcaFlow jmsURL="vm://localhost" connectionManager="#connectionManager" /&gt;
              &lt;/sm:flows&gt;
      
      
              &lt;jencks:connectionManager id="connectionManager"
                      containerManagedSecurity="false" transaction="xa"
                      transactionManager="#transactionManager"
                      poolingSupport="#poolingSupport"
                      connectionTracker="#connectionTracker" /&gt;
      
              &lt;jencks:transactionManager id="transactionManager"
                      transactionLogDir="./data/txlog"
                      defaultTransactionTimeoutSeconds="600" /&gt;
      
              &lt;jencks:connectionTracker id="connectionTracker"
                      geronimoTransactionManager="#transactionManager" /&gt;
      
              &lt;jencks:poolingSupport id="poolingSupport"
                      connectionMaxWaitMilliseconds="30000" connectionMaxIdleMinutes="1"
                      poolMinSize="10" poolMaxSize="64" /&gt;
      
              &lt;amq:broker useJmx="true" name="broker" brokerName="localhost"
                      dataDirectory="data/amq"&gt;
      
                      &lt;amq:persistenceAdapter&gt;
                              &lt;amq:journaledJDBC useDatabaseLock="false"
                                      dataDirectory="data/amq" journalLogFiles="5" dataSource="#sqlserver-ds" /&gt;
                      &lt;/amq:persistenceAdapter&gt;
      
                      &lt;amq:transportConnectors&gt;
                              &lt;amq:transportConnector uri="vm://localhost" /&gt;
                      &lt;/amq:transportConnectors&gt;
      
                      &lt;amq:systemUsage&gt;
                              &lt;amq:systemUsage&gt;
                                      &lt;amq:memoryUsage&gt;
                                              &lt;amq:memoryUsage limit="64 mb" percentUsageMinDelta="20" /&gt;
                                      &lt;/amq:memoryUsage&gt;
                                      &lt;amq:tempUsage&gt;
                                              &lt;amq:tempUsage limit="100 mb" /&gt;
                                      &lt;/amq:tempUsage&gt;
                                      &lt;amq:storeUsage&gt;
                                              &lt;amq:storeUsage limit="1 gb" name="foo" /&gt;
                                      &lt;/amq:storeUsage&gt;
                              &lt;/amq:systemUsage&gt;
                      &lt;/amq:systemUsage&gt;
              &lt;/amq:broker&gt;
      
              &lt;bean id="sqlserver-ds" class="org.apache.commons.dbcp.BasicDataSource" destroy-method="close"&gt;
                      &lt;property name="driverClassName" value="net.sourceforge.jtds.jdbc.Driver" /&gt;
                      &lt;property name="url" value="jdbc:jtds:sqlserver://server:1433/test_db;tds=8.0" /&gt;
                      &lt;property name="username" value="user" /&gt;
                      &lt;property name="password" value="pass" /&gt;
                      &lt;property name="initialSize" value="10" /&gt;
                      &lt;property name="maxActive" value="64" /&gt;
                      &lt;property name="maxIdle" value="2" /&gt;
                      &lt;property name="maxWait" value="15000" /&gt;
              &lt;/bean&gt;
      

       

      I tried to put the stack trace and XML config, but it does not appear on the post. How I can do it?

      Edited by: fretzlaff on Mar 17, 2008 7:58 PM

       

      Edited by: fretzlaff on Mar 17, 2008 8:00 PM

        • 1. Re: Problem on load test with FUSE MB
          bsnyder

          This appears to be a known issue as it's noted in both AMQ-1063 and AMQ-1611.

           

          Have you tried using straight JDBC message persistence to work around the issue temporarily?

           

          Bruce

          • 2. Re: Problem on load test with FUSE MB
            fretzlaff_fretzlaff

            I tried and it works fine on the first 5.000 messages, but after sent 10.000 messages it stay slow to process.

            • 3. Re: Problem on load test with FUSE MB
              bsnyder

              It sounds like messages might be getting backed up on the broker. Are you using topics or queues? I know you're using message persistence (and straight JDBC persistence is much slower than journaled JDBC), but are you using durable topics? Also, how many consumers are you pointing at the destinations?

               

              Bruce

              • 4. Re: Problem on load test with FUSE MB
                fretzlaff_fretzlaff

                I don't use any explicity configuration of JMS, I had configured the SMX to use JCA flow and I don't have any JMS component. I just let the SMX make the failover for me.

                I have some components that extends PojoSupport and I need to send messages using the sendSync method, this the ony thing I did different from the normal use of ServiceMix:

                 

                     RobustInOnly rio = this.createRobustInOnlyExchange(exchange);
                     QName service = new QName(NAMESPACEURI, c.getTarget());
                     rio.setService(service);
                     RobustInOnlyImpl impl = (RobustInOnlyImpl) rio;
                     impl.setPersistent(Boolean.FALSE);
                     rio.setProperty(JbiConstants.FLOW_PROPERTY_NAME, "seda");                    
                     NormalizedMessage nm = rio.createMessage();
                     IntegratorMarshaller.toNMS(imsg, nm);
                     rio.setInMessage(nm);
                     boolean sent = this.sendSync(rio);
                

                 

                I change the properties of de exchange to use the Seda flow because the JCA flow does not support sendSync.

                • 5. Re: Problem on load test with FUSE MB
                  fretzlaff_fretzlaff

                  I was looking the messages on the database and notice that the ACTIVEMQ_MSGS has a lot of messages.

                  When I restart ServiceMix all the messages is reprocessed.

                  I tried to change Fuse MB to version 4.1.2.5 and this problem not occurs, but in the other hand when I send more than 1.000 messages the system hangs after process some messages.

                  With version 5.0.0.9 and amqPersistence the problem also not occurs but I take this error: AMQ-1544

                   

                  What I can do?

                  Which version is stable so I can use in production?

                  • 6. Re: Problem on load test with FUSE MB
                    bsnyder

                     

                    With version 5.0.0.9 and amqPersistence the problem also not occurs but I take this error: AMQ-1544

                     

                     

                    So are you seeing the error displayed in AMQ-1544? It looks like it's fixed in ActiveMQ 5.1. There is a snapshot of the FUSE Message Broker 5.1 available but it has not yet been fully released. You could try to drop in the FUSE MB 5.1 libraries as a temporary work around.

                     

                    Bruce

                    • 7. Re: Problem on load test with FUSE MB
                      fretzlaff_fretzlaff

                      Do you know when 5.1 will be released?