1 2 Previous Next 15 Replies Latest reply on Jan 24, 2011 1:50 AM by fcorneli

    AS Deployers versus HornetQ deployers

    clebert.suconic

      Say you have a service that is depending on a destination, and you want that destination deployed as soon as the HornetQ server is up...

       

      With the application server deployers, there's no way to guarantee the destinations will be deployed before the server is started. For destinations deployed by the AS deployers, services would need to declare their dependencies for the Micro Container to handle them.

       

       

      If we want certain destinations to be deployed before the server is up, we would need to have them on the main configuration file (Under deploy/hornetq/*xml), and have the HornetQ deployers or anything on the server doing it at least once.

       

       

      But it happens that during the latest changes on the HQ deployers, this doesn't work any more. If you run the EE/MDB examples, you will see that the MDBs are deployed before the destinations are deployed.

       

       

      We could fix this by adding the depencies on the MDBs, what would fix the case for AS6 and EAP5. But what about other cases where we don't have the AS deployers in place?

        • 1. Re: AS Deployers versus HornetQ deployers
          ataylor

          Ok, so what changed were made so that this doesnt work any more?

           

          also, adding them to MDB's is a no no.

          • 2. Re: AS Deployers versus HornetQ deployers
            clebert.suconic

            I believe this was changed by Jeff on the HornetQ deployers, trying to avoid destinations being deployed twice.

             

            The HornetQ deployers should deploy the destinations at least once during startup. They don't need to keep scanning for changes when integrated on the AS, but they should at least do it before the server is started.

            • 3. Re: AS Deployers versus HornetQ deployers
              jmesnil

              I don't follow: to me, the idea was that when we're integrated in the AS, the AS deployers were in charge to deploy the queues.

              It seems ugly to have core queues deployed once by HornetQ QueueDeployer and then again by the AS RealDeployer or whatever (I thought there was a lot of mumbo jumbo to have it work with managed objects too but I digress)

               

              I does not make sense to deploy queues from the AS deployers before the server is started (as there is nothing to deploy on). So the AS deployers needs to make sure the HornetQ server is started before deploying anything.

               

              Looking at deployers/hornetq-deployers-jboss-beans.xml, it seems we're missing some dependencies here:

              => the deployers should <demand> HornetQServer (so that the server is started before they deploy)

              => HornetQDestinationCreator should <demand> the deployers (to be sure the resources are deployed before the destinations are passed to the MDB)

               

              Does that make sense?

              • 4. Re: AS Deployers versus HornetQ deployers
                ataylor

                I think that the deployers do wait for HornetQ to start before deploying the queues, either  intentionally or not. The issue is that the hornetq AS deployers should deploy before the MDB deployers.

                • 5. Re: AS Deployers versus HornetQ deployers
                  ataylor

                  actually, they are started in the correct order, if you look at the logs below u can see this. There must be a bug somewhere in the deployers, its funny how the second time u run it it works because th equeue is loaded from storage instead?

                   

                  [exec] 17:37:31,635 INFO  [HornetQJMSRealDeployer] Deploying unit AbstractVFSDeploymentContext@25077087{vfsfile:/home/andy/projects/JBPAPP_5_1_hornetq-int/build/output/jboss-5.1.0.Branch/server/mdb-bmt-example-profile/deploy/hornetq/hornetq-jms.xml} with config org.hornetq.jms.server.config.impl.JMSConfigurationImpl@1065b63

                       [exec] 17:37:32,168 SEVERE [HornetQActivation] Unable to start activation

                       [exec] HornetQException[errorCode=100 message=Queue jms.queue.testQueue does not exist]

                       [exec]     at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:287)

                       [exec]     at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateConsumer(ClientSessionImpl.java:1556)

                       [exec]     at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:447)

                       [exec]     at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:392)

                       [exec]     at org.hornetq.core.client.impl.DelegatingSession.createConsumer(DelegatingSession.java:201)

                       [exec]     at org.hornetq.ra.inflow.HornetQMessageHandler.setup(HornetQMessageHandler.java:168)

                       [exec]     at org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:287)

                       [exec]     at org.hornetq.ra.inflow.HornetQActivation$SetupActivation.run(HornetQActivation.java:502)

                       [exec]     at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)

                       [exec]     at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)

                       [exec]     at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)

                       [exec]     at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)

                       [exec]     at java.lang.Thread.run(Thread.java:619)

                       [exec] 17:37:34,035 INFO  [ProfileServiceBootstrap] Loading profile: ProfileKey@39aa42[domain=default, server=default, name=mdb-bmt-example-profile]

                  • 6. Re: AS Deployers versus HornetQ deployers
                    ataylor

                    Ok, heres a summary of whats actually happening and how we caan solve it.

                     

                    firstly the issue:

                     

                    The AS has a number of deployers 2 of which are the HornetQ JMS deployers that deploy destinations etc and the MDB deployers. The ordering of these deployers can easily be controlled via dependency tags in configs. Each deployer will create a deployment unit that has a specific lifecycle, i.e. start and stop. In our case when started this will create a JMS resourse in the MDB deployers case an dMDB activation. These are created and handed to the container to be started at a later point in time, here lies the issue. Its not possible to control in what order these deployment units are started, actually it looks like its alphabetical as if u name the mdb ear to zzz something it gets called later.

                     

                    Anyway, having chatted with the AS/eap guys they are of the opinion that this is not an issue. They are also of the opinion that an MDB being activated should never be dependant on a JMS Server being available (I agree on this point if the server is remote).

                     

                    This being the case all we can really do is to retry activating the MDB on failure to activate, basically waiting for the JMS resource to be recreated. The same issue actually exists in AS 5 using JBM and this is how the generic RA deals with it.

                    • 7. Re: AS Deployers versus HornetQ deployers
                      jmesnil

                      After a talk with AS guys about http://community.jboss.org/thread/152189 on IRC, their proposed solution is that it is up to the RA to handle the case where the MDB is activated before the JMS resources are deployed.

                       

                      Taking JmsActivation as an example, I updated HornetQActivation to try to setup in multiple attempts in case of "failures" (connection issues, queues not there, roles not deployed, etc.)

                       

                      With this "fix" when the AS is started, we have a warning displayed because the MDB is deployed before HornetQ resources are deployed and the RA is successfully setup:

                       

                       

                           [exec] 12:22:31,141 INFO  [EJBContainer] STARTED EJB: org.hornetq.javaee.example.server.MDB_BMTExample ejbName: MDB_BMTExample
                           [exec] 12:22:31,627 INFO  [TomcatDeployment] deploy, ctxPath=/hornetq
                           [exec] 12:22:31,669 WARNING [HornetQActivation] Failure in HornetQ activation org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@743ef044 destination=queue/testQueue destinationType=javax.jms.Queue ack=Dups-ok-acknowledge durable=false clientID=null user=null maxSession=15)
                           [exec] HornetQException[errorCode=100 message=Queue jms.queue.testQueue does not exist]
                           [exec]      at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBlocking(ChannelImpl.java:287)
                           [exec]      at org.hornetq.core.client.impl.ClientSessionImpl.internalCreateConsumer(ClientSessionImpl.java:1556)
                           [exec]      at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:447)
                           [exec]      at org.hornetq.core.client.impl.ClientSessionImpl.createConsumer(ClientSessionImpl.java:392)
                           [exec]      at org.hornetq.core.client.impl.DelegatingSession.createConsumer(DelegatingSession.java:201)
                           [exec]      at org.hornetq.ra.inflow.HornetQMessageHandler.setup(HornetQMessageHandler.java:166)
                           [exec]      at org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:294)
                           [exec]      at org.hornetq.ra.inflow.HornetQActivation$SetupActivation.run(HornetQActivation.java:560)
                           [exec]      at org.jboss.resource.work.WorkWrapper.execute(WorkWrapper.java:205)
                           [exec]      at org.jboss.util.threadpool.BasicTaskWrapper.run(BasicTaskWrapper.java:260)
                           [exec]      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
                           [exec]      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
                           [exec]      at java.lang.Thread.run(Thread.java:637)
                           [exec] 12:22:31,722 WARNING [config] Unable to process deployment descriptor for context '/hornetq'
                           [exec] 12:22:31,723 INFO  [config] Initializing Mojarra (1.2_13-b01-FCS) for context '/hornetq'
                           [exec] 12:22:31,918 ERROR [HornetQQueueDeployment] >>>>>> Deploying Queue testQueue
                           [exec] 12:22:32,150 INFO  [ProfileServiceBootstrap] Loading profile: ProfileKey@48ca42a9[domain=default, server=default, name=mdb-bmt-example-profile]
                           [exec] 12:22:32,170 INFO  [Http11Protocol] D???marrage de Coyote HTTP/1.1 sur http-127.0.0.1-8080
                           [exec] 12:22:32,221 INFO  [AjpProtocol] Starting Coyote AJP/1.3 on ajp-127.0.0.1-8009
                           [exec] 12:22:32,235 INFO  [ServerImpl] JBoss (Microcontainer) [5.1.0.Branch (build: SVNTag=JBPAPP_5_1_0_Branch date=201005181423)] Started in 1m:2s:332ms
                           [exec] 12:22:41,673 INFO  [HornetQActivation] Attempting to reconnect org.hornetq.ra.inflow.HornetQActivationSpec(ra=org.hornetq.ra.HornetQResourceAdapter@743ef044 destination=queue/testQueue destinationType=javax.jms.Queue ack=Dups-ok-acknowledge durable=false clientID=null user=null maxSession=15)
                           [exec] 12:22:41,772 INFO  [HornetQActivation] Reconnected with HornetQ

                       

                       

                       

                      The example works fine but this is ugly: a stack trace is displayed and it seems like HornetQ is not working properly. It reconnects after the server is started... greaaaattttt....

                       

                      I used the same defaults than the AS 5 generic JMS RA:

                      - 5 attempts to setup

                      - 10 seconds before retrying to setup

                      I am not sure if it is a better idea to reuse our reconnect properties for that (by default for the RA, they mean infinite reconnection and no retry interval defined)

                       

                      To sum things up, this reconnection thingamagic is just a prettier way to say "sleep until AS decide to deploy Hornetq resources" with stack traces and log warning for better effect... I still think this is not the correct way to fix this issue. When HornetQ is integrated into the AS, it should be possible to say "let's deploy  HornetQ resources before the AS deploy user resources" but the AS team disagrees, so be it!

                       

                      I've attached the patch if someone want to have a 2nd look...

                      • 8. Re: AS Deployers versus HornetQ deployers
                        ataylor

                        maybe we could do a separate check for the queue using a queue query and back off

                        • 9. Re: AS Deployers versus HornetQ deployers
                          timfox

                          Andy Taylor wrote:

                           

                          maybe we could do a separate check for the queue using a queue query and back off

                          That won't work for a topic though

                          • 10. Re: AS Deployers versus HornetQ deployers
                            ataylor

                            dont we deploy a dummy queue for a topic?

                            • 11. Re: AS Deployers versus HornetQ deployers
                              fcorneli

                              Seems like this issue remains in JBoss AS 6.0.0.Final. And the "fix" only works if the application server deploys the application EARs fast enough before the HornetQ gives up. So on slow servers, or with heavy applications that take a long time to deploy, the JMS setup could fail. Why wasn't such a critical issue fixed before 6.0.0.Final came out?

                              • 12. Re: AS Deployers versus HornetQ deployers
                                jaikiran

                                Reading this thread, it appears that it's just a WARN message that gets logged and later the HornetQActivation does reconnect to the queue/topic when it is created.

                                 

                                Frank Cornelis wrote:

                                 

                                So on slow servers, or with heavy applications that take a long time to deploy, the JMS setup could fail.

                                I don't know how this is implemented, but I guess the number of times (and/or the time interval) to check for the JMS destination is probably configurable? Someone from HornetQ team might be able to confirm.

                                 

                                Frank Cornelis wrote:

                                 

                                Why wasn't such a critical issue fixed before 6.0.0.Final came out?

                                Is adding an explicit dependency on the queue/topic, an option for you? If yes, then we can discuss how that needs to be done.

                                • 13. AS Deployers versus HornetQ deployers
                                  fcorneli

                                  Heh... on a fast machine it works. On a slow machine I get

                                   

                                  08:25:11,714 INFO  [org.hornetq.ra.inflow.HornetQActivation] awaiting topic/queue creation queue/polling/task-queue

                                   

                                  and finally

                                   

                                  DEPLOYMENTS IN ERROR:

                                    Deployment "queue/polling/task-queue" is in error due to the following reason(s): ** NOT FOUND Depends on 'queue/polling/task-queue' **

                                   

                                  How do I declare an explicit dependency on the queue?

                                  • 14. AS Deployers versus HornetQ deployers
                                    jaikiran

                                    Frank Cornelis wrote:

                                     


                                     

                                    How do I declare an explicit dependency on the queue?

                                    http://community.jboss.org/message/563234#563234

                                    1 2 Previous Next