Endlessly Discarded Message

[logan.hauenstein]

I have a network of brokers in an ActiveMQ 5.5.1 system that consists of sensor brokers (which generally produce data), two server brokers (which consume sensor data and produce higher-level data), and several UI-oriented brokers (which consume sensor and server data and produce a few low-bandwidth messages).  We use non-durable topics for all messages, we have a fairly conservative fixed-size message discard policy (set to ~30 messages right now), and we're using the discardingDLQBrokerPlugin to immediately discard old messages.

We recently encountered an issue where it looks like a broker (usually one of the servers, but occasionally we see it on a sensor) falls behind on a topic, discards messages, and appears to get stuck in an endless loop.  The attached log sample shows ~100 lines of many many MB of identical-looking logs -- the messageId never changes, the BrokerPath is always "SenType2100BigScreenN" repeated over and over.  The afflicted broker becomes effectively non-functional and seems to cause other message delivery problems across the network.

Has anyone seen this kind of issue before?  Are there any kind of configuration settings we can play around with to resolve this issue?  Is there any more information that I can provide that might help diagnose the issue?

More generally, I'd like to learn more about best practices for designing networks of brokers.  In particular, I'd like to get a better understanding for good/bad patterns to follow and how messages are routed through a system for both topics and queues.  Are there any resources that anyone can recommend?

Sample line of log:

2012-04-24 12:40:57,344 activemq.broker.region.TopicSubscription: - Discarding message ActiveMQObjectMessage {commandId = 3265, responseRequired = false, messageId = ID:large-2877-1335198336694-3:9662:1:1:7, originalDestination = null, originalTransactionId = null, producerId = ID:SENUNIT105-1029-1335282277437-2:1:1:1, destination = topic://sensor.sen.status.52, transactionId = null, expiration = 1335283503796, timestamp = 1335283473796, arrival = 0, brokerInTime = 1335279783213, brokerOutTime = 1335283474203, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@8ed8e2, marshalledProperties = org.apache.activemq.util.ByteSequence@40dfc4, dataStructure = null, redeliveryCounter = 0, size = 2348, properties = {sourceSensorID=52, isRawDataTrigger=false, breadcrumbId=ID-large-2874-1335198335678-0-295451, BrokerPath=SenType2100BigScreenN,SenType2100BigScreenN,SenType2100BigScreenN,SenType2100BigScreenN}, readOnlyProperties = false, readOnlyBody = false, droppable = false}

Edited by: logan.hauenstein on Apr 25, 2012 2:22 PM

[garytully]

It would be good to see the xml configuration of the brokers.

The message property:

BrokerPath=SenType2100BigScreenN,SenType2100BigScreenN,SenType2100BigScreenN,SenType2100BigScreenN

seems odd, as if the brokers in the network are all called the same name.

This would could be problematic. A network bridge depends on the brokerPath (an append only string property of all of the brokers that a message visits) to stop messages looping between brokers, so the brokerIds need to be unique.

In terms of best practice, it is often best to use separate network connectors for queues and topics, mostly because conduitSubscriptions="false" is a must for topics to avoid duplicates but it can be a performance impediment to queues.