ActiveMQ starts to fail in Fuse Jboss when a feature is deployed

[chubutin]

Im having a problem when I deploy a feature. The feature contains three bundles, and Karaf deploys well these bundles, but when they are deployed ActiveMQ starts to having problems.

The deployed bundles are simples. The "complicated" is a camel route who expose a CXF endpoint and call a endpoint mock. I just attached to this threar the .kar, the zip of that kar and my fuse log. The service is running, but the problem with activeMQ happend always

The .kar is in https://dl.dropboxusercontent.com/u/225304/PruebaFeature-1.0-SNAPSHOT.kar

The .zip of the .kar is in https://dl.dropboxusercontent.com/u/225304/prueba.zip

The error is always the same:

2013-05-14 15:19:48,046 | INFO  | veMQ Broker: amq | ActiveMQServiceFactory$$anon$1   | ?                                   ? | 106 - org.springframework.context - 3.1.3.RELEASE | Refreshing org.fusesource.mq.fabric.ActiveMQServiceFactory$$anon$1@33c91e: startup date ; root of context hierarchy

2013-05-14 15:19:48,048 | INFO  | veMQ Broker: amq | XBeanXmlBeanDefinitionReader     | ?                                   ? | 105 - org.springframework.beans - 3.1.3.RELEASE | Loading XML bean definitions from file /home/ramiro/tecPlata/jboss-fuse-6.0.0.redhat-024/etc/activemq.xml

2013-05-14 15:19:48,095 | INFO  | veMQ Broker: amq | DefaultListableBeanFactory       | ?                                   ? | 105 - org.springframework.beans - 3.1.3.RELEASE | Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@1885c3a: defining beans ; root of factory hierarchy

2013-05-14 15:19:48,159 | INFO  | veMQ Broker: amq | PListStoreImpl                   | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | PListStore:[/home/ramiro/tecPlata/jboss-fuse-6.0.0.redhat-024/data/amq/amq/tmp_storage] started

2013-05-14 15:19:48,163 | ERROR | veMQ Broker: amq | BrokerService                    | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Failed to start Apache ActiveMQ (amq, null). Reason: javax.management.InstanceAlreadyExistsException: org.apache.activemq:type=Broker,brokerName=amq

javax.management.InstanceAlreadyExistsException: org.apache.activemq:type=Broker,brokerName=amq

          at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)[:1.6.0_30]

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)[:1.6.0_30]

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)[:1.6.0_30]

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)[:1.6.0_30]

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)[:1.6.0_30]

          at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)[:1.6.0_30]

          at org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)[114:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]

          at org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)[114:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]

          at org.apache.activemq.broker.BrokerService.startManagementContext(BrokerService.java:2337)[114:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]

          at org.apache.activemq.broker.BrokerService.start(BrokerService.java:543)[114:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]

          at org.fusesource.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration$$anon$3.run(ActiveMQServiceFactory.scala:307)[128:org.jboss.amq.mq-fabric:6.0.0.redhat-024]

2013-05-14 15:19:48,164 | INFO  | veMQ Broker: amq | BrokerService                    | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Apache ActiveMQ 5.8.0.redhat-60024 (amq, null) is shutting down

2013-05-14 15:19:48,168 | INFO  | veMQ Broker: amq | TransportConnector               | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Connector openwire Stopped

2013-05-14 15:19:48,169 | INFO  | veMQ Broker: amq | PListStoreImpl                   | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | PListStore:[/home/ramiro/tecPlata/jboss-fuse-6.0.0.redhat-024/data/amq/amq/tmp_storage] stopped

2013-05-14 15:19:48,169 | INFO  | veMQ Broker: amq | KahaDBStore                      | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Stopping async queue tasks

2013-05-14 15:19:48,169 | INFO  | veMQ Broker: amq | KahaDBStore                      | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Stopping async topic tasks

2013-05-14 15:19:48,169 | INFO  | veMQ Broker: amq | KahaDBStore                      | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Stopped KahaDB

2013-05-14 15:19:48,169 | INFO  | veMQ Broker: amq | BrokerService                    | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Apache ActiveMQ 5.8.0.redhat-60024 (amq, null) uptime 0.010 seconds

2013-05-14 15:19:48,169 | INFO  | veMQ Broker: amq | BrokerService                    | ?                                   ? | 114 - org.apache.activemq.activemq-osgi - 5.8.0.redhat-60024 | Apache ActiveMQ 5.8.0.redhat-60024 (amq, null) is shutdown

2013-05-14 15:19:48,169 | INFO  | veMQ Broker: amq | ActiveMQServiceFactory           | ?                                   ? | 128 - org.jboss.amq.mq-fabric - 6.0.0.redhat-024 | Broker amq failed to start.  Will try again in 10 seconds

2013-05-14 15:19:48,169 | INFO  | veMQ Broker: amq | ActiveMQServiceFactory           | ?                                   ? | 128 - org.jboss.amq.mq-fabric - 6.0.0.redhat-024 | Exception on start: javax.management.InstanceAlreadyExistsException: org.apache.activemq:type=Broker,brokerName=amq

javax.management.InstanceAlreadyExistsException: org.apache.activemq:type=Broker,brokerName=amq

          at com.sun.jmx.mbeanserver.Repository.addMBean(Repository.java:453)[:1.6.0_30]

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.internal_addObject(DefaultMBeanServerInterceptor.java:1484)[:1.6.0_30]

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerDynamicMBean(DefaultMBeanServerInterceptor.java:963)[:1.6.0_30]

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerObject(DefaultMBeanServerInterceptor.java:917)[:1.6.0_30]

          at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.registerMBean(DefaultMBeanServerInterceptor.java:312)[:1.6.0_30]

          at com.sun.jmx.mbeanserver.JmxMBeanServer.registerMBean(JmxMBeanServer.java:482)[:1.6.0_30]

          at org.apache.activemq.broker.jmx.ManagementContext.registerMBean(ManagementContext.java:380)[114:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]

          at org.apache.activemq.broker.jmx.AnnotatedMBean.registerMBean(AnnotatedMBean.java:72)[114:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]

          at org.apache.activemq.broker.BrokerService.startManagementContext(BrokerService.java:2337)[114:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]

          at org.apache.activemq.broker.BrokerService.start(BrokerService.java:543)[114:org.apache.activemq.activemq-osgi:5.8.0.redhat-60024]

          at org.fusesource.mq.fabric.ActiveMQServiceFactory$ClusteredConfiguration$$anon$3.run(ActiveMQServiceFactory.scala:307)[128:org.jboss.amq.mq-fabric:6.0.0.redhat-024]

This example I used a clean Fuse. Any ideas of what is happening? i dont know if the problem is the configuration of ActiveMQ, or anything else.

This is what i recive when I list activemq in Karaf

This is when I list the broker in karaf before the deploy

Name = KahaDBPersistenceAdapter[/home/ramiro/tecPlata/jboss-fuse-6.0.0.redhat-024/data/amq/kahadb]

brokerName = amq

Transactions = []

Size = 12835

InstanceName = KahaDBPersistenceAdapter[/home/ramiro/tecPlata/jboss-fuse-6.0.0.redhat-024/data/amq/kahadb]

Data =

type = Broker

Service = PersistenceAdapter

brokerName = amq

service = Health

CurrentStatus = Good

type = Broker

brokerName = amq

connector = clientConnectors

type = Broker

StatisticsEnabled = true

connectorName = openwire

destinationName = ActiveMQ.Advisory.MasterBroker

MemoryUsageByteCount = 0

DequeueCount = 0

type = Broker

destinationType = Topic

Name = ActiveMQ.Advisory.MasterBroker

MinEnqueueTime = 0

MaxAuditDepth = 2048

AverageEnqueueTime = 0.0

InFlightCount = 0

MemoryLimit = 67108864

brokerName = amq

EnqueueCount = 1

MaxEnqueueTime = 0

MemoryUsagePortion = 1.0

ProducerCount = 0

UseCache = true

BlockedProducerWarningInterval = 30000

AlwaysRetroactive = false

Options =

MaxProducersToAudit = 64

PrioritizedMessages = false

ConsumerCount = 0

ProducerFlowControl = true

Subscriptions = []

QueueSize = 0

MaxPageSize = 200

DispatchCount = 0

MemoryPercentUsage = 0

ExpiredCount = 0

TopicSubscribers = []

TemporaryQueues = []

Uptime = 8 minutes

TemporaryTopicSubscribers = []

MemoryPercentUsage = 0

BrokerVersion = 5.8.0.redhat-60024

StatisticsEnabled = true

TotalDequeueCount = 0

TopicProducers = []

QueueSubscribers = []

Topics = org.apache.activemq:type=Broker,brokerName=amq,destinationType=Topic,destinationName=ActiveMQ.Advisory.MasterBroker

TotalMessageCount = 0

SslURL =

TemporaryQueueSubscribers = []

BrokerName = amq

DynamicDestinationProducers = []

Persistent = true

DataDirectory = /home/ramiro/tecPlata/jboss-fuse-6.0.0.redhat-024/data/amq

Queues = []

DurableTopicSubscribers = []

TotalConsumerCount = 0

InactiveDurableTopicSubscribers = []

JobSchedulerStoreLimit = 0

TempPercentUsage = 0

MemoryLimit = 67108864

VMURL = vm://amq

OpenWireURL = tcp://fluxit-ntb-43:61616?maximumConnections=1000

JobSchedulerStorePercentUsage = 0

TotalEnqueueCount = 1

TemporaryQueueProducers = []

StompSslURL =

TemporaryTopics = []

StompURL =

Slave = false

BrokerId = ID:fluxit-ntb-43-33261-1368560785626-0:1

TotalProducerCount = 0

StorePercentUsage = 0

brokerName = amq

StoreLimit = 107374182400

TransportConnectors = {openwire=tcp://fluxit-ntb-43:61616?maximumConnections=1000}

TemporaryTopicProducers = []

TempLimit = 53687091200

QueueProducers = []

type = Broker

and after the deploy

Name = KahaDBPersistenceAdapter[/home/ramiro/tecPlata/jboss-fuse-6.0.0.redhat-024/data/amq/kahadb]

brokerName = amq

Transactions = []

Size = 12979

InstanceName = KahaDBPersistenceAdapter[/home/ramiro/tecPlata/jboss-fuse-6.0.0.redhat-024/data/amq/kahadb]

Data =

type = Broker

Service = PersistenceAdapter

brokerName = amq

service = Health

CurrentStatus = Good

type = Broker

brokerName = amq

connector = clientConnectors

type = Broker

StatisticsEnabled = true

connectorName = openwire

destinationName = ActiveMQ.Advisory.MasterBroker

MemoryUsageByteCount = 0

DequeueCount = 0

type = Broker

destinationType = Topic

Name = ActiveMQ.Advisory.MasterBroker

MinEnqueueTime = 0

MaxAuditDepth = 2048

AverageEnqueueTime = 0.0

InFlightCount = 0

MemoryLimit = 67108864

brokerName = amq

EnqueueCount = 1

MaxEnqueueTime = 0

MemoryUsagePortion = 1.0

ProducerCount = 0

UseCache = true

BlockedProducerWarningInterval = 30000

AlwaysRetroactive = false

Options =

MaxProducersToAudit = 64

PrioritizedMessages = false

ConsumerCount = 0

ProducerFlowControl = true

Subscriptions = []

QueueSize = 0

MaxPageSize = 200

DispatchCount = 0

MemoryPercentUsage = 0

ExpiredCount = 0

TopicSubscribers = []

TemporaryQueues = []

Uptime = 33.592 seconds

TemporaryTopicSubscribers = []

MemoryPercentUsage = 0

BrokerVersion = 5.8.0.redhat-60024

StatisticsEnabled = true

TotalDequeueCount = 0

TopicProducers = []

QueueSubscribers = []

Topics = org.apache.activemq:type=Broker,brokerName=amq,destinationType=Topic,destinationName=ActiveMQ.Advisory.MasterBroker

TotalMessageCount = 0

SslURL =

TemporaryQueueSubscribers = []

BrokerName = amq

DynamicDestinationProducers = []

Persistent = true

DataDirectory = /home/ramiro/tecPlata/jboss-fuse-6.0.0.redhat-024/data/amq

Queues = []

DurableTopicSubscribers = []

TotalConsumerCount = 0

InactiveDurableTopicSubscribers = []

JobSchedulerStoreLimit = 0

TempPercentUsage = 0

MemoryLimit = 67108864

VMURL = vm://amq

OpenWireURL = tcp://fluxit-ntb-43:61616?maximumConnections=1000

JobSchedulerStorePercentUsage = 0

TotalEnqueueCount = 1

TemporaryQueueProducers = []

StompSslURL =

TemporaryTopics = []

StompURL =

Slave = false

BrokerId = ID:fluxit-ntb-43-33261-1368560785626-0:3

TotalProducerCount = 0

StorePercentUsage = 0

brokerName = amq

StoreLimit = 107374182400

TransportConnectors = {openwire=tcp://fluxit-ntb-43:61616?maximumConnections=1000}

TemporaryTopicProducers = []

TempLimit = 53687091200

QueueProducers = []

type = Broker

As you see, the broker ID changed.

I attached the .kar, the .zip of the kar and the log.

[davsclaus]

How do you deploy the KAR? By copying it to the deploy directory or something else?

I assume you use spring xml in your applications, as it seems spring-dm triggers spring bundles to refresh, and somehow also the etc/activemq.xml file which triggeres a reload of the embedded Broker. And causing that issue in the log.

What happens if you deploy your application as 3 individual bundles? eg maybe using the osgi:install command?

[davsclaus]

I logged a ticket about this bug

http://fusesource.com/issues/browse/ENTESB-726

[ffang]

Hi,

The features.xml in your kar is incorrect which cause this error. It has some bundle like

    <bundle>mvn:org.apache.felix/org.apache.felix.configadmin/1.2.4</bundle>
    <bundle>mvn:org.apache.aries/org.apache.aries.util/1.0.0</bundle>
    <bundle>mvn:org.apache.aries.proxy/org.apache.aries.proxy.api/1.0.0</bundle>
    <bundle>mvn:org.apache.aries.blueprint/org.apache.aries.blueprint/1.0.1.redhat-60024</bundle>

Those bundles are very fundamental for the  container and already get installed by container by default.

It shouldn't be in your features.xml, or if they're  there, you should have

resolver="(obr)" for feature and dependency="true" for those bundle so that OBR resolver can kick in to prevent install redundant bundles.

Moreover, the

<bundle>mvn:org.apache.aries.blueprint/org.apache.aries.blueprint/1.0.1.redhat-60024</bundle>

is invalid for aries.blueprint 1.0.x, it should be

<bundle dependency="true" start-level="20">mvn:org.apache.aries.blueprint/org.apache.aries.blueprint.api/1.0.1.redhat-60024</bundle>
        <bundle dependency="true" start-level="20">mvn:org.apache.aries.blueprint/org.apache.aries.blueprint.core/1.0.1.redhat-60024</bundle>
        <bundle dependency="true" start-level="20">mvn:org.apache.aries.blueprint/org.apache.aries.blueprint.cm/1.0.1.redhat-60024</bundle>

instead. Otherwise you will see errors like

ERROR: Bundle org.apache.aries.blueprint [251] EventDispatcher: Error during dispatch. (java.lang.ClassCastException: org.apache.aries.blueprint.ext.impl.ExtNamespaceHandler cannot be cast to org.apache.aries.blueprint.NamespaceHandler)
java.lang.ClassCastException: org.apache.aries.blueprint.ext.impl.ExtNamespaceHandler cannot be cast to org.apache.aries.blueprint.NamespaceHandler

This means you have two conflict aries.blueprint bundle installed in your container which messed up almost everything.

In a summary, change your features.xml in your kar like

<?xml version="1.0" encoding="UTF-8"?>
<features>
  <feature name='tosMock' version='1.0.0-SNAPSHOT'>
    <bundle>mvn:com.tecplata.esb.services/tosMock/1.0.0-SNAPSHOT</bundle>
  </feature>
  <feature name='esb-entities' version='1.0.0-SNAPSHOT'>
    <bundle>mvn:com.tecplata.esb/esb-entities/1.0.0-SNAPSHOT</bundle>
  </feature>
  <feature name='vesselsService-sei' version='1.0.0-SNAPSHOT'>
    <feature version='1.0.0-SNAPSHOT'>esb-entities</feature>
    <bundle>mvn:com.tecplata.esb.services.sei/vesselsService-sei/1.0.0-SNAPSHOT</bundle>
  </feature>
  <feature name='vesselsVisitorService' version='1.0.0-SNAPSHOT'>
    <bundle>mvn:org.apache.camel/camel-core/2.10.0.redhat-60024</bundle>
    <feature version='1.0.0-SNAPSHOT'>vesselsService-sei</feature>
    <bundle>mvn:com.tecplata.esb.services/vesselsVisitorService/1.0.0-SNAPSHOT</bundle>
  </feature>
</features>

can make it work.

Freeman

Edited by: ffang on May 15, 2013 7:42 AM

Edited by: ffang on May 15, 2013 7:43 AM

[chubutin]

I deployed the Kar copying in the /deploy directory.

When I install the 3 bundles individually the deploy is ok.

Now I tried the Freeman's solution (remove the libs from the feature.xml) and it works great! Thank you Claus

[chubutin]

Thank you Freeman. I removed the aries, camel and karaf bundles from the feature.xml, generated de .kar again and it works great!

Im a little confuse for this, because I created the feature.xml with the features-maven-plugin and I put as  all the camel, karaf and spring dependencies, but the plugin still add that dependencies in the feature.xml.

Edited by: chubutin on May 15, 2013 3:19 PM

[chubutin]

I removed some dependencies from my pom and the plugin works great! Thank you very much!