-
1. Re: 46.6. Avoiding Anti-Patterns
clebert.suconic Dec 30, 2010 11:54 AM (in response to rick.dong)If you have lots of consumers on a queue: you will be doing lots of scans on a single queue.
If you have lots of queues on an address: you will be distributing the message towards multiple queues. But you will be creating several addresses per queue.
Since you have very unique requirements, I would recommend you test careful to validate the best option for your architecture. Maybe you should test the new paging on trunk (which is based on files).
-
2. Re: 46.6. Avoiding Anti-Patterns
rick.dong Jan 12, 2011 11:20 PM (in response to clebert.suconic)Thanks for the reply clebert. I found that neither the approach works well in supporting lots of consumers using the filter. My goal is to have a many consumers each is only interested in the designated message (e.g. a message with a receiver ID). Having created many queues each with a filter on the receiver ID means that a message won't be routed until the matching queue is found. Same for creating one queue with many filtered consumers as the one and only consumer would have to be found. I've written some test case for both and the performance is rather poor with either of the approache. However, there is a third approach that is quite fast which is by creating non-filter queue that binds to an address which represents the the unique consumer. But this can lead to OOM when the message is to be sent to many addresses (e.g.: happened to me with 1 million). Any suggestions?
Stack traces with OOM
Exception in thread "pool-6-thread-1" java.lang.OutOfMemoryError: GC overhead limit exceeded
at org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:47)
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:39)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:139)
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:73)
at org.jboss.netty.buffer.DynamicChannelBuffer.<init>(DynamicChannelBuffer.java:64)
at org.jboss.netty.buffer.DynamicChannelBuffer.copy(DynamicChannelBuffer.java:273)
at org.hornetq.core.buffers.impl.ChannelBufferWrapper.copy(ChannelBufferWrapper.java:222)
at org.hornetq.core.message.impl.MessageImpl.forceCopy(MessageImpl.java:939)
at org.hornetq.core.message.impl.MessageImpl.encodeToBuffer(MessageImpl.java:872)
at org.hornetq.core.message.impl.MessageImpl.getEncodedBuffer(MessageImpl.java:470)
at org.hornetq.core.protocol.core.impl.wireformat.SessionSendMessage.encode(SessionSendMessage.java:70)
at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:163)
at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:147)
at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:289)
at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:139)
at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:144)
----- or the follow trace ----------------
SEVERE: Caught unexpected exception
java.lang.OutOfMemoryError: GC overhead limit exceeded
at java.util.ArrayList.<init>(ArrayList.java:112)
at org.hornetq.core.postoffice.impl.DuplicateIDCacheImpl.<init>(DuplicateIDCacheImpl.java:73)
at org.hornetq.core.postoffice.impl.PostOfficeImpl.getDuplicateIDCache(PostOfficeImpl.java:743)
at org.hornetq.core.paging.impl.PagingStoreImpl.<init>(PagingStoreImpl.java:196)
at org.hornetq.core.paging.impl.PagingStoreFactoryNIO.newStore(PagingStoreFactoryNIO.java:93)
at org.hornetq.core.paging.impl.PagingManagerImpl.newStore(PagingManagerImpl.java:230)
at org.hornetq.core.paging.impl.PagingManagerImpl.createPageStore(PagingManagerImpl.java:104)
at org.hornetq.core.paging.impl.PagingManagerImpl.getPageStore(PagingManagerImpl.java:130)
at org.hornetq.core.server.impl.ServerSessionImpl.requestProducerCredits(ServerSessionImpl.java:1033)
at org.hornetq.core.protocol.core.ServerSessionPacketHandler.handlePacket(ServerSessionPacketHandler.java:494)
at org.hornetq.core.protocol.core.impl.ChannelImpl.handlePacket(ChannelImpl.java:471)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.doBufferReceived(RemotingConnectionImpl.java:451)
at org.hornetq.core.protocol.core.impl.RemotingConnectionImpl.bufferReceived(RemotingConnectionImpl.java:412)
at org.hornetq.core.remoting.server.impl.RemotingServiceImpl$DelegatingBufferHandler.bufferReceived(RemotingServiceImpl.java:459)
at org.hornetq.core.remoting.impl.invm.InVMConnection$1.run(InVMConnection.java:135)
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:100)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
-
3. 46.6. Avoiding Anti-Patterns
rickxu Feb 11, 2011 4:44 AM (in response to rick.dong)I meet the same problom today. I just simply recieve messages with a listener:
Server side:
ServerLocator serverLocator = HornetQClient.createServerLocatorWithoutHA(new TransportConfiguration(InVMConnectorFactory.class.getName()));
ClientSessionFactory CSF = serverLocator.createSessionFactory();
ClientSession coreSession = CSF.createSession(true, true);
ClientSession listenerSession = CSF.createSession();
ClientConsumer messageConsumer = listenerSession.createConsumer(QUEUE_NAME);
messageConsumer.setMessageHandler(ToyCarRMIServer.this);
listenerSession.start();
public void onMessage(final ClientMessage message) {
System.out.println("recieve a messge")
}
Client side:
ServerLocator serverLocator = HornetQClient.createServerLocatorWithoutHA(new TransportConfiguration(NettyConnectorFactory.class.getName()));
ClientSessionFactory CSF = serverLocator.createSessionFactory();
...
ClientProducer producer = session.createProducer(QUEUE_NAME);
...
producer.send(message_request);
and quickly OOM in server process:
Exception in thread "pool-1-thread-2" java.lang.OutOfMemoryError: Java heap space
at org.jboss.netty.buffer.HeapChannelBuffer.<init>(HeapChannelBuffer.java:47)
at org.jboss.netty.buffer.BigEndianHeapChannelBuffer.<init>(BigEndianHeapChannelBuffer.java:39)
at org.jboss.netty.buffer.ChannelBuffers.buffer(ChannelBuffers.java:139)
at org.jboss.netty.buffer.HeapChannelBufferFactory.getBuffer(HeapChannelBufferFactory.java:73)
at org.jboss.netty.buffer.DynamicChannelBuffer.<init>(DynamicChannelBuffer.java:64)
at org.jboss.netty.buffer.DynamicChannelBuffer.copy(DynamicChannelBuffer.java:273)
at org.hornetq.core.buffers.impl.ChannelBufferWrapper.copy(ChannelBufferWrapper.java:222)
at org.hornetq.core.remoting.impl.invm.InVMConnection.write(InVMConnection.java:121)
at org.hornetq.core.protocol.core.impl.ChannelImpl.send(ChannelImpl.java:199)
at org.hornetq.core.protocol.core.impl.ChannelImpl.sendBatched(ChannelImpl.java:147)
at org.hornetq.core.client.impl.ClientProducerImpl.doSend(ClientProducerImpl.java:291)
at org.hornetq.core.client.impl.ClientProducerImpl.send(ClientProducerImpl.java:135)
at com.toycar.rmi.server.ToyCarRMIServer$5.run(ToyCarRMIServer.java:236)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Exception in thread "Thread-12 (group:HornetQ-server-threads9690924-3109534)" java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Unknown Source)
at java.lang.AbstractStringBuilder.expandCapacity(Unknown Source)
at java.lang.AbstractStringBuilder.append(Unknown Source)
at java.lang.StringBuffer.append(Unknown Source)
at java.io.StringWriter.write(Unknown Source)
at java.io.PrintWriter.write(Unknown Source)
at java.io.PrintWriter.write(Unknown Source)
at java.io.PrintWriter.print(Unknown Source)
at java.io.PrintWriter.println(Unknown Source)
at java.lang.Throwable.printStackTrace(Throwable.java:512)
at java.util.logging.SimpleFormatter.format(Unknown Source)
at java.util.logging.StreamHandler.publish(Unknown Source)
at java.util.logging.ConsoleHandler.publish(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at java.util.logging.Logger.doLog(Unknown Source)
at java.util.logging.Logger.log(Unknown Source)
at org.hornetq.core.logging.impl.JULLogDelegate.error(JULLogDelegate.java:68)
at org.hornetq.core.logging.Logger.error(Logger.java:169)
at org.hornetq.utils.OrderedExecutorFactory$OrderedExecutor$1.run(OrderedExecutorFactory.java:104)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)