Intermittent Services Startup Error

[prower]

I often get intermittent error where some of my Switchyard  services will fail to start.  Here is the stack trace I get:

15:25:00,329 INFO  [org.jboss.as.controller] (Controller Boot Thread) JBAS014774: Service status report

JBAS014777:   Services which failed to start:      service jboss.deployment.subunit."backend-integration-services.ear"."backend-action.jar".SwitchYardService: org.jboss.msc.service.StartException in service jboss.deployment.subunit."backend-integration-services.ear"."backend-action.jar".SwitchYardService: java.lang.IllegalArgumentException: Unique key for ResourceAdapter 'hornetq-ra.rar' couldn't be found.

It's really not special to this particular service.  I have others that will also get this same error when starting the container.  I can work around it by restarting the container.  There seems to be some kind of race condition occurring.  Often all my services will start up correctly but almost just as often one or more will get the above error.

The switchyard binding they all have in common is the outbound jca binding.  All but one of my four services also use an inbound jca binding.  I've seen this error occur on each of my services at some point.  Above example is only services which doesn't have an inbound jca binding so that makes me believe this possibly is an issue with the outbound jca binding.  I'm using SY 1.0.0.Final with EAP 6.1.0.  I couldn't find an existing issue that seamed related to problem I'm having.  Is this a reported bug?  Any solution better than my just keep restarting until all your services start exist?

Thanks.

[kcbabo]

That definitely should not happen.  If you have an app that you can share to reproduce this behavior, that would be great.  Reference bindings are started first, so it may be a timing issue with initialization of the hornetq subsystem and our deployments during startup.  Tomo - can you have a look?

[igarashitm]

hmm, I tried a couple of times but it didn't happen... however it's possible the SwitchYard application is deployed before the HornetQ ResourceAdapter is initialized.

Phillip, could you try adding jboss-deployment-structure.xml into META-INF directory of your application jar?

• application.jar!/META-INF/jboss-deployment-structure.xml

<?xml version="1.0" encoding="UTF-8"?>
<jboss-deployment-structure>
  <deployment>
    <dependencies>
      <module name="org.hornetq.ra" />
    </dependencies>
  </deployment>
</jboss-deployment-structure>

Thanks,

Tomo

[prower]

I am not sure I understand how adding org.hornetq.ra module to my application should solve it?  My server configuration has org.jboss.as.messaging as an extension which has a dependency on that module.  I shouldn't need define a dependency for my application, correct?

I didn't see your replies until late, but I did try restarting my services several times without creating the issue again myself.  I did this from my home pc.  I've only had this issue at work.  I have it fairly frequently there.  My ubuntu vm at work runs slower than one I have at home.  I do think this is a runtime race issue.  I'll try recreating this at work.  I'll ask if I can share my projects. 

Awhile back I did trace through your code... I recall it was something to do with binding references to hornetq queues.  It was looking up hornetq resource but if its not available it would give error I mentioned above.  I checked into to type it was which was some jboss service that documentation said you shouldn't block in.  Sorry I am going off of what I remember.  I am checking out your code now to find it now (actually it is eclipse import is what takes too long) but I need to put my kid to sleep now.  I'll try to find it again at work in the morning.

[prower]

I after a few attempts at work I was able to cause the exception again.  I'm not sure if its related but service failing to start only happened when I had messages already persisted to my queues.

Here is a better stack trace:

09:45:52,799 ERROR [org.jboss.msc.service.fail] (MSC service thread 1-10) MSC000001: Failed to start service jboss.deployment.subunit."backend-integration-services.ear"."backend-action.jar".SwitchYardService: org.jboss.msc.service.StartException in service jboss.deployment.subunit."backend-integration-services.ear"."backend-action.jar".SwitchYardService: java.lang.IllegalArgumentException: Unique key for ResourceAdapter 'hornetq-ra.rar' couldn't be found.
  at org.switchyard.as7.extension.services.SwitchYardService.start(SwitchYardService.java:80)
  at org.jboss.msc.service.ServiceControllerImpl$StartTask.startService(ServiceControllerImpl.java:1811) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
  at org.jboss.msc.service.ServiceControllerImpl$StartTask.run(ServiceControllerImpl.java:1746) [jboss-msc-1.0.4.GA-redhat-1.jar:1.0.4.GA-redhat-1]
  at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [rt.jar:1.7.0_25]
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [rt.jar:1.7.0_25]
  at java.lang.Thread.run(Thread.java:724) [rt.jar:1.7.0_25]
Caused by: java.lang.IllegalArgumentException: Unique key for ResourceAdapter 'hornetq-ra.rar' couldn't be found.
  at org.switchyard.component.jca.deploy.JCAActivator.handleReferenceBinding(JCAActivator.java:204)
  at org.switchyard.component.jca.deploy.JCAActivator.activateBinding(JCAActivator.java:85)
  at org.switchyard.deploy.internal.Deployment.deployReferenceBindings(Deployment.java:299)
  at org.switchyard.deploy.internal.Deployment.start(Deployment.java:141)
  at org.switchyard.as7.extension.deployment.SwitchYardDeployment.start(SwitchYardDeployment.java:101)
  at org.switchyard.as7.extension.services.SwitchYardService.start(SwitchYardService.java:73)
  ... 5 more

Now that I can recreate it I will try again with your suggested fix Tomo.

[prower]

Unfortunately adding a hornetq-ra dependency to all my deployments didn't fix the problem.  Having data in the queues isn't issue either as all data was processed when I restarted the last time. 

I'll ask my management if I can share my projects but let me describe my projects to see if that helps.

I have 4 ear deployments.  All them have switchyard services in one jar. 

Incidentally I have the deployments organized all in their own ear as I would have issues if two switchyard jars had references to the same queue in an ear.  When calling the method on reference interface it would block until the transaction timed out. I also had to duplicate interfaces with the @OperationTypes which I was using to refer to same queue because if those interfaces were not in the same jar as the service no transformations would occur.  My use case here was I had a dead letter queue I wanted to route messages to if I wasn't able process them.  I digress but it would be nice if I didn't need to put different SY composite service jars in different ears and have to duplicate interfaces in each jar.

First deployment polls messages using JPA from an AS400 database then use camel route to place messages in appropriate queue.  The second deployment reads that initial message and enriches it with other data from the AS400 then routes it on to the third deployment to do the business processing.  The fourth deployment reads error messages of a queue and routes those via JPA back to AS400.  Each deployment uses JCA component to route messages via queues on to the next component.  All messages are xml bound to JAXB classes.  I'm using 1.0.0.Final of Switchyard with EAP 6.1.0.

[igarashitm]

I'm guessing the problem is that HornetQ ResourceAdapter is deployed internally so we need to look for a way to hook that internal event. Anyway, I'll try reproduce it but if you could provide your applications that would be great.

[igarashitm]

OK, debugger helps me to reproduce this - just capture the PooledConnectionFactoryService to delay the HornetQ RAR deployment process. Created a JIRA to track this issue.

[SWITCHYARD-1789] JCAActivator should wait for the HornetQ RAR deployment - JBoss Issue Tracker