That definitely should not happen. If you have an app that you can share to reproduce this behavior, that would be great. Reference bindings are started first, so it may be a timing issue with initialization of the hornetq subsystem and our deployments during startup. Tomo - can you have a look?
hmm, I tried a couple of times but it didn't happen... however it's possible the SwitchYard application is deployed before the HornetQ ResourceAdapter is initialized.
Phillip, could you try adding jboss-deployment-structure.xml into META-INF directory of your application jar?
<?xml version="1.0" encoding="UTF-8"?> <jboss-deployment-structure> <deployment> <dependencies> <module name="org.hornetq.ra" /> </dependencies> </deployment> </jboss-deployment-structure>
I am not sure I understand how adding org.hornetq.ra module to my application should solve it? My server configuration has org.jboss.as.messaging as an extension which has a dependency on that module. I shouldn't need define a dependency for my application, correct?
I didn't see your replies until late, but I did try restarting my services several times without creating the issue again myself. I did this from my home pc. I've only had this issue at work. I have it fairly frequently there. My ubuntu vm at work runs slower than one I have at home. I do think this is a runtime race issue. I'll try recreating this at work. I'll ask if I can share my projects.
Awhile back I did trace through your code... I recall it was something to do with binding references to hornetq queues. It was looking up hornetq resource but if its not available it would give error I mentioned above. I checked into to type it was which was some jboss service that documentation said you shouldn't block in. Sorry I am going off of what I remember. I am checking out your code now to find it now (actually it is eclipse import is what takes too long) but I need to put my kid to sleep now. I'll try to find it again at work in the morning.
I after a few attempts at work I was able to cause the exception again. I'm not sure if its related but service failing to start only happened when I had messages already persisted to my queues.
Here is a better stack trace:
09:45:52,799 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-10) MSC000001: Failed to start service jboss.deployment.subunit."backend-integration-services.ear"."backend-action.jar".SwitchYardService: org.jboss.msc.service.StartException in service jboss.deployment.subunit."backend-integration-services.ear"."backend-action.jar".SwitchYardService: java.lang.IllegalArgumentException: Unique key for ResourceAdapter 'hornetq-ra.rar' couldn't be found. at org.switchyard.as7.extension.services.SwitchYardService.start(SwitchYardService.java:80) at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25] at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25] Caused by: java.lang.IllegalArgumentException: Unique key for ResourceAdapter 'hornetq-ra.rar' couldn't be found. at org.switchyard.component.jca.deploy.JCAActivator.handleReferenceBinding(JCAActivator.java:204) at org.switchyard.component.jca.deploy.JCAActivator.activateBinding(JCAActivator.java:85) at org.switchyard.deploy.internal.Deployment.deployReferenceBindings(Deployment.java:299) at org.switchyard.deploy.internal.Deployment.start(Deployment.java:141) at org.switchyard.as7.extension.deployment.SwitchYardDeployment.start(SwitchYardDeployment.java:101) at org.switchyard.as7.extension.services.SwitchYardService.start(SwitchYardService.java:73) ... 5 more
Now that I can recreate it I will try again with your suggested fix Tomo.
Unfortunately adding a hornetq-ra dependency to all my deployments didn't fix the problem. Having data in the queues isn't issue either as all data was processed when I restarted the last time.
I'll ask my management if I can share my projects but let me describe my projects to see if that helps.
I have 4 ear deployments. All them have switchyard services in one jar.
Incidentally I have the deployments organized all in their own ear as I would have issues if two switchyard jars had references to the same queue in an ear. When calling the method on reference interface it would block until the transaction timed out. I also had to duplicate interfaces with the @OperationTypes which I was using to refer to same queue because if those interfaces were not in the same jar as the service no transformations would occur. My use case here was I had a dead letter queue I wanted to route messages to if I wasn't able process them. I digress but it would be nice if I didn't need to put different SY composite service jars in different ears and have to duplicate interfaces in each jar.
First deployment polls messages using JPA from an AS400 database then use camel route to place messages in appropriate queue. The second deployment reads that initial message and enriches it with other data from the AS400 then routes it on to the third deployment to do the business processing. The fourth deployment reads error messages of a queue and routes those via JPA back to AS400. Each deployment uses JCA component to route messages via queues on to the next component. All messages are xml bound to JAXB classes. I'm using 1.0.0.Final of Switchyard with EAP 6.1.0.
I'm guessing the problem is that HornetQ ResourceAdapter is deployed internally so we need to look for a way to hook that internal event. Anyway, I'll try reproduce it but if you could provide your applications that would be great.
OK, debugger helps me to reproduce this - just capture the PooledConnectionFactoryService to delay the HornetQ RAR deployment process. Created a JIRA to track this issue.