Thanks for the reply, but first a quick update:
So over the weekend I had the issue again. Actually I am working on it right now.
Just one queue filling up even though all queues get the same amount of messages and doing the same amount of processing and all others (51 for this Topic) are working fine.
dump via jboss-cli:
6dc9d19f-34b9-45e8-a784-22c4ddccdfea : 11513
consumer-count : "result" => 1
delivering-count : "result" => 1
filter : "result" => "message_type = 'event_property_change' AND ( resource_url LIKE 'services/%/blah/something')"
dead-letter-address : "result" => undefined,
durable : "result" => false
expiry-address : "result" => undefined,
id : "result" => 119L
messages-added : "result" => 61469L
paused : "result" => false
queue-address : "result" => "jms.topic.MyChannel"
scheduled-count : "result" => 0L
temporary : "result" => true
I enabled logging for the MDB via jboss-cli and see no logs being produced which indicates to me that the MDB stopped handling of messages.
I cross-checked with the amount of subscribers on the Topic an the number is correct.
The MDB has the @MessageDriven annotation and also has
@TransactionManagement(value = TransactionManagementType.CONTAINER)
@TransactionAttribute(value = TransactionAttributeType.NOT_SUPPORTED)
@Mirek:
Is this what you meant with transactions? I also have transactions disabled at the producer side, which is a Singleton/Startup that creates the connection & session during creation.
I also have the MDB set to "useLocalTx", propertyValue = "true". Dunno if that can conflict with the above entries.
I also left everything at BLOCK and such as I want to find the culprit and reason for the failures.
Actually I think the best resolution would be to add the TTL so that the older messages get dumped, seeing that they will be replaced soon anyway.
Mind you, until it is actually full, I do not get the "blahblah is blocked" message! At least I am not getting it now.
Or if I totally missed what you are asking, pls elaborate so I can provided the needed information.
Thanks!
I have also thought about the GC being a culprit. Since the current instance is flooding again, I will restart and add in GC logging.
Could also be our infamous NIC driver. Had some issues before. Hope this is not a Heisenbug.
Update:
The deployment claims delivery is active for this MDB:
"BD.CBMEL" => {
"component-class-name" => "BD.CBMEL",
"declared-roles" => [],
"delivery-active" => true,
"execution-time" => 0L,
"invocations" => 0L,
"methods" => {},
"peak-concurrent-invocations" => 0L,
"pool-available-count" => 30,
"pool-create-count" => 1,
"pool-current-size" => 1,
"pool-max-size" => 30,
"pool-name" => "mdb-strict-max-pool",
"pool-remove-count" => 0,
"run-as-role" => undefined,
"security-domain" => "other",
"timers" => [],
"wait-time" => 0L,
"service" => undefined
}
A small dump of this weekends results:
No lines were removed and only the topic name changed to protect the innocent. Sadly we cannot see who really is being filled or exceeding max delivery attempts (but in the latter both are among those queues for this topic)
2016-07-14 13:45:45,923 WARN [org.hornetq.core.client] (Thread-1765) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-14 17:05:54,093 WARN [org.hornetq.core.client] (Thread-1759) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-15 15:48:07,943 ERROR [org.hornetq.ra] (Thread-21288 (HornetQ-client-global-threads-17694520)) HQ154004: Failed to deliver message: HornetQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=HQ119014: Timed out waiting for response when sending packet 43]
2016-07-15 15:48:47,475 WARN [org.hornetq.core.server] (Thread-3276 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=1c5caa8d-4913-11e6-8fb8-fdbc4bf27b50-11387154-12030799)) HQ222150: Message has exceeded max delivery attempts. No Dead Letter Address configured for queue 935d5575-936e-4b5b-9cca-5ff2059f0184 so dropping it
2016-07-15 16:18:06,844 ERROR [org.hornetq.ra] (Thread-21472 (HornetQ-client-global-threads-17694520)) HQ154004: Failed to deliver message: HornetQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=HQ119014: Timed out waiting for response when sending packet 43]
2016-07-15 16:18:21,456 WARN [org.hornetq.core.server] (Thread-3331 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=1c5caa8d-4913-11e6-8fb8-fdbc4bf27b50-11387154-12030799)) HQ222150: Message has exceeded max delivery attempts. No Dead Letter Address configured for queue 93d2f890-45f4-4769-9daa-48d84607214e so dropping it
2016-07-15 17:56:05,926 WARN [com.barco.bcm.propertystore.notification.NotificationEmitterImpl] (Thread-22171 (HornetQ-client-global-threads-17694520)) Execption @ sendNotification : javax.jms.JMSException: HQ119014: Timed out waiting for response when sending packet 30
2016-07-16 04:17:48,212 WARN [org.hornetq.core.client] (Thread-1809) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-16 19:28:48,267 WARN [org.hornetq.core.client] (Thread-1788) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-17 06:58:10,900 ERROR [org.hornetq.ra] (Thread-36848 (HornetQ-client-global-threads-17694520)) HQ154004: Failed to deliver message: HornetQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=HQ119014: Timed out waiting for response when sending packet 43]
2016-07-17 06:58:24,115 WARN [org.hornetq.core.server] (Thread-5864 (HornetQ-remoting-threads-HornetQServerImpl::serverUUID=1c5caa8d-4913-11e6-8fb8-fdbc4bf27b50-11387154-12030799)) HQ222150: Message has exceeded max delivery attempts. No Dead Letter Address configured for queue 93d2f890-45f4-4769-9daa-48d84607214e so dropping it
2016-07-17 08:15:49,870 WARN [org.hornetq.core.client] (Thread-1759) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-17 10:06:08,381 ERROR [org.hornetq.ra] (Thread-38074 (HornetQ-client-global-threads-17694520)) HQ154004: Failed to deliver message: HornetQConnectionTimedOutException[errorType=CONNECTION_TIMEDOUT message=HQ119014: Timed out waiting for response when sending packet 43]
2016-07-17 10:06:38,399 WARN [org.hornetq.ra] (Thread-38074 (HornetQ-client-global-threads-17694520)) HQ152002: Unable to roll local transaction back
2016-07-17 14:27:46,153 WARN [org.hornetq.core.client] (Thread-1906) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-17 14:27:56,171 WARN [org.hornetq.core.client] (Thread-1906) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-17 14:28:06,185 WARN [org.hornetq.core.client] (Thread-1906) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-17 14:28:16,193 WARN [org.hornetq.core.client] (Thread-1906) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-17 14:28:26,205 WARN [org.hornetq.core.client] (Thread-1906) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-17 14:28:36,213 WARN [org.hornetq.core.client] (Thread-1906) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-17 14:28:46,226 WARN [org.hornetq.core.client] (Thread-1906) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.
2016-07-18 04:38:48,311 WARN [org.hornetq.core.client] (Thread-2004) HQ212054: Destination address=jms.topic.MyChannel is blocked. If the system is configured to block make sure you consume messages on this configuration.