-
3. Re: Memory leak with clustered JMS Server
Thanks Mirek, for quick response.
System goes OOM even if I leave my deployment idle. Here is messaging subsystem which we are using on Machine 3 where MDBs are deployed. It is same on Machine 1 and 2. Additionally. machine 1 and 2 have topics and queue deployed in messaging subsystem.
<subsystem xmlns="urn:jboss:domain:messaging:1.2">
<hornetq-server>
<clustered>true</clustered>
<persistence-enabled>false</persistence-enabled>
<security-enabled>false</security-enabled>
<journal-file-size>102400</journal-file-size>
<journal-min-files>2</journal-min-files>
<connectors>
<netty-connector name="netty" socket-binding="messaging"/>
<netty-connector name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
</netty-connector>
<in-vm-connector name="in-vm" server-id="0"/>
</connectors>
<acceptors>
<netty-acceptor name="netty" socket-binding="messaging">
<param key="use-nio" value="true"/>
</netty-acceptor>
<netty-acceptor name="netty-throughput" socket-binding="messaging-throughput">
<param key="batch-delay" value="50"/>
<param key="direct-deliver" value="false"/>
<param key="use-nio" value="true"/>
</netty-acceptor>
<in-vm-acceptor name="in-vm" server-id="0"/>
</acceptors>
<broadcast-groups>
<broadcast-group name="bg-group1">
<socket-binding>messaging-group</socket-binding>
<broadcast-period>5000</broadcast-period>
<connector-ref>netty</connector-ref>
</broadcast-group>
</broadcast-groups>
<discovery-groups>
<discovery-group name="dg-group1">
<socket-binding>messaging-group</socket-binding>
<refresh-timeout>10000</refresh-timeout>
</discovery-group>
</discovery-groups>
<cluster-connections>
<cluster-connection name="my-cluster">
<address>jms</address>
<connector-ref>netty</connector-ref>
<discovery-group-ref discovery-group-name="dg-group1"/>
</cluster-connection>
</cluster-connections>
<security-settings>
<security-setting match="#">
<permission type="send" roles="guest"/>
<permission type="consume" roles="guest"/>
<permission type="createNonDurableQueue" roles="guest"/>
<permission type="deleteNonDurableQueue" roles="guest"/>
</security-setting>
</security-settings>
<address-settings>
<!--default for catch all-->
<address-setting match="#">
<dead-letter-address>jms.queue.DLQ</dead-letter-address>
<expiry-address>jms.queue.ExpiryQueue</expiry-address>
<redelivery-delay>0</redelivery-delay>
<redistribution-delay>1000</redistribution-delay>
<max-size-bytes>10485760</max-size-bytes>
<address-full-policy>BLOCK</address-full-policy>
<message-counter-history-day-limit>10</message-counter-history-day-limit>
</address-setting>
</address-settings>
<jms-connection-factories>
<connection-factory name="InVmConnectionFactory">
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/ConnectionFactory"/>
</entries>
</connection-factory>
<connection-factory name="RemoteConnectionFactory">
<connectors>
<connector-ref connector-name="netty"/>
</connectors>
<entries>
<entry name="java:jboss/exported/jms/RemoteConnectionFactory"/>
</entries>
</connection-factory>
<pooled-connection-factory name="hornetq-ra">
<transaction mode="xa"/>
<connectors>
<connector-ref connector-name="in-vm"/>
</connectors>
<entries>
<entry name="java:/JmsXA"/>
</entries>
</pooled-connection-factory>
</jms-connection-factories>
</hornetq-server>
</subsystem>
-
4. Re: Memory leak with clustered JMS Server
Hi Ravi,
I can't find anything what could cause OOM. Only what comes to my mind is standalone.conf file and whether you modified setting memory for JVM. By default there should be something like:
if [ "x$JAVA_OPTS" = "x" ]; then
JAVA_OPTS="-Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true"
JAVA_OPTS="$JAVA_OPTS -Djboss.modules.system.pkgs=$JBOSS_MODULES_SYSTEM_PKGS -Djava.awt.headless=true"
else
echo "JAVA_OPTS already set in environment; overriding default settings with values: $JAVA_OPTS"
fi
During start of the server you can check what is set in JAVA_OPTS variable. I can see:
=========================================================================
JBoss Bootstrap Environment
JBOSS_HOME: /home/mnovak/tmp/hornetq_eap6_dev/internal/eap6.1.0.ER8/server1/jboss-eap-6.1
JAVA: /home/mnovak/Downloads/jdk1.7.0_15/bin/java
JAVA_OPTS: -server -XX:+UseCompressedOops -Xms1303m -Xmx1303m -XX:MaxPermSize=256m -Djava.net.preferIPv4Stack=true -Djboss.modules.system.pkgs=org.jboss.byteman -Djava.awt.headless=true
=========================================================================
In our test we set memory for AS 7 server to:
-Xms64m -Xmx788m -XX:MaxPermSize=256m
Are you using virtual machines? Is there enough memory for them?
Cheers,
Mirek
-
5. Re: Memory leak with clustered JMS Server
Yes, we are using VMs and there is enough memory for servers. Following is the output at the beginning of JBoss server log on Application Server:
"JAVA_OPTS already set in environment; overriding default settings with values: -server -Xrs -Xms512m -Xmx512m -XX:+DisableExplicitGC -Xss192K -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=C:\logs\ -DAPP_SERVER_NAME=TestServer -DAPP_SERVER_PORT=4447 -Dservertype=cs -Djboss.connection.params=host=JMSServer1;port=5445,host=JMSServer2;port=5445 -Djboss.connection.classname=org.hornetq.core.remoting.impl.netty.NettyConnectorFactory,org.hornetq.core.remoting.impl.netty.NettyConnectorFactory -Dapp.name=TestServer"
===============================================================================
JBoss Bootstrap Environment
JBOSS_HOME: C:\jboss-as-7.1.2.Final
JAVA: C:\Program Files\Java\jdk1.7.0_04\bin\java
JAVA_OPTS: -Dprogram.name=standalone.bat -server -Xrs -Xms512m -Xmx512m -XX:+DisableExplicitGC -Xss192K -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=C:\logs\ -DAPP_SERVER_NAME=TestServer -DAPP_SERVER_PORT=4447 -Dservertype=cs -Djboss.connection.params=host=JMSServer1;port=5445,host=JMSServer2;port=5445 -Djboss.connection.classname=org.hornetq.core.remoting.impl.netty.NettyConnectorFactory,org.hornetq.core.remoting.impl.netty.NettyConnectorFactory -Dapp.name=TestServer
===============================================================================
In current test -Xms and -Xmx are set to 512MB but it goes OOM for 1024m also.
-
6. Re: Memory leak with clustered JMS Server
I tried by passing missing JAVA_OPTS but same result. I have uploaded heapdump at this location if it helps:
https://docs.google.com/file/d/0BzcNdKDAu5f4Ujd0d3V0NXNGVGs/edit?usp=sharing
-
7. Re: Memory leak with clustered JMS Server
1 of 1 people found this helpfulAnother thought...could you try to set reconnectAttempts and setupAttempts to 10 in activation-config property.
-
8. Re: Memory leak with clustered JMS Server
It seems that changing reconnectAttempts and setupAttempts to 10 has fixed this problem. But we want MDBs to keep on retrying in case of network disconnect.
-
9. Re: Memory leak with clustered JMS Server
Ok, it appears there is a leak in HornetQ resource apdapter when setupAttempts is set to -1 and it's indefinitely trying to create JMS connection to cluster. I've spent some time reproducing it with AS 7.2.0/EAP 6.1 but without success. Can you see warnings/errors in server.log, like connection to cluster is unsuccessful.
-
10. Re: Memory leak with clustered JMS Server
We are using JBoss As 7.1.2 and we are observing it only for MDBs which are for queues. I do not see message like "Connection to cluster is unsuccessful" but I do see following message getting logged after couple of hours starting the application:
06:36:33,054 INFO [org.jboss.as] (Controller Boot Thread) JBAS015874: JBoss AS 7.1.2.Final "Steropes" started in 65423ms - Started 2655 of 3202 services (546 services are passive or on-demand)
06:38:45,094 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /10.10.**.***:54108. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
06:54:09,517 WARN [org.hornetq.core.protocol.core.impl.RemotingConnectionImpl] (hornetq-failure-check-thread) Connection failure has been detected: Did not receive data from /10.10.**.***:60308. It is likely the client has exited or crashed without closing its connection, or the network between the server and client has failed. You also might have configured connection-ttl and client-failure-check-period incorrectly. Please check user manual for more information. The connection will now be closed. [code=3]
09:56:57,698 INFO [org.jboss.as.naming] (Remoting "****761l" task-3) JBAS011806: Channel end notification received, closing channel Channel ID 278d8f0f (inbound) of Remoting connection 12fe447e to /10.10.**.***:55638
09:57:19,633 ERROR [org.jboss.as.naming] (pool-5-thread-7) JBAS011809: Failed to send exception response to client: org.jboss.remoting3.NotOpenException: Writes closed
at org.jboss.remoting3.remote.RemoteConnectionChannel.openOutboundMessage(RemoteConnectionChannel.java:107) [jboss-remoting-3.2.7.GA.jar:3.2.7.GA]
at org.jboss.remoting3.remote.RemoteConnectionChannel.writeMessage(RemoteConnectionChannel.java:296) [jboss-remoting-3.2.7.GA.jar:3.2.7.GA]
at org.jboss.naming.remote.protocol.v1.WriteUtil.write(WriteUtil.java:59)
at org.jboss.naming.remote.protocol.v1.WriteUtil.writeExceptionResponse(WriteUtil.java:81)
at org.jboss.naming.remote.protocol.v1.RemoteNamingServerV1$MessageReciever$1.sendIOException(RemoteNamingServerV1.java:97)
at org.jboss.naming.remote.protocol.v1.RemoteNamingServerV1$MessageReciever$1.run(RemoteNamingServerV1.java:86)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) [rt.jar:1.7.0_04]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) [rt.jar:1.7.0_04]
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04]
-
11. Re: Memory leak with clustered JMS Server
I did some testing with AS 7.1.2 but still no success. Based on the WARN ...Connection failure has been detected: Did not receive... I suspect there could be network failure problem in environment or there is something in MDB what is causing the memory leak.
INFO log .. Channel end notification received, closing channel Channel ID 278d8f0f (inbound) of Remoting connection 12fe447e.. seems to be already resolved in [1]
Some other thoughts what could be done:
- upgrade to AS 7.2.0/EAP 6.1 and give it a try (there are bugfixes for HQ RA in it)
- if it does not help then provide a reproducer or better description/source what MDB is doing
- (try to debug the server when MDB is deployed and see why HorneQMessageHandler instances are not garbage collected)
-
12. Re: Memory leak with clustered JMS Server
Thanks Mirek, I'll try out with AS 7.2.0 and will let you know. I did debug the MDBs and MDBs are not doing anything at all when I leave my system idle.
-
13. Re: Memory leak with clustered JMS Server
Problem still exists with JBoss AS 7.2.0. I again checked heapdump and I could see that it is creating thousands of instances of MessageQHandler object for MDBs which are for queue. It is creating exactly 16 instances of MessageQHandler for MDBs which are for topic. It seems that it is not closing the instance properly in case of queue MDBs I can see following attributes for MessageQHandler instance:
Type |Name |Value
--------------------------------------------------------------------------------------
ref |cf |org.hornetq.core.client.impl.ClientSessionFactoryImpl @ 0xff80b298
ref |tm |com.arjuna.ats.jbossatx.jta.TransactionManagerDelegate @ 0xe146ab80
int |sessionNr |2
boolean|useXA |false
boolean|transacted|false
boolean|useLocalTx|false
ref |activation|org.hornetq.ra.inflow.HornetQActivation @ 0xe87c2170
ref |endpoint |null
ref |consumer |org.hornetq.core.client.impl.ClientConsumerImpl @ 0xff80ff90
ref |session |org.hornetq.core.client.impl.DelegatingSession @ 0xff80ddf0
--------------------------------------------------------------------------------------
Also, sessionNr is either 0,1 or 2 for all these instances while for topic MDBs these are 0-15. Something is wrong for queue MDBs. Please advise how I can enable extra logging in HornetQActivation class to check what it is doing there.
Thanks,
Ravi
-
14. Re: Memory leak with clustered JMS Server
Just replace logging subsystem by:
<subsystem xmlns="urn:jboss:domain:logging:1.2"> <console-handler name="CONSOLE"> <level name="DEBUG"/> <formatter> <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/> </formatter> </console-handler> <periodic-rotating-file-handler name="FILE"> <formatter> <pattern-formatter pattern="%d{HH:mm:ss,SSS} %-5p [%c] (%t) %s%E%n"/> </formatter> <file relative-to="jboss.server.log.dir" path="server.log"/> <suffix value=".yyyy-MM-dd"/> <append value="true"/> </periodic-rotating-file-handler> <logger category="com.arjuna"> <level name="INFO"/> </logger> <logger category="org.apache.tomcat.util.modeler"> <level name="WARN"/> </logger> <logger category="sun.rmi"> <level name="WARN"/> </logger> <logger category="jacorb"> <level name="WARN"/> </logger> <logger category="jacorb.config"> <level name="ERROR"/> </logger> <logger category="org.hornetq"> <level name="TRACE"/> </logger> <logger category="org.jboss.netty"> <level name="TRACE"/> </logger> <logger category="org.jboss"> <level name="INFO"/> </logger> <root-logger> <level name="TRACE"/> <handlers> <handler name="CONSOLE"/> <handler name="FILE"/> </handlers> </root-logger> </subsystem>This will provide TRACE logs for org.hornetq and org.jboss.netty packages.
-
15. Re: Memory leak with clustered JMS Server
Hi Mirek,
I am still struggling to find any clue from the debug logs but it seems that it continous logging that node is going up:
06:07:17,959 DEBUG [org.hornetq.core.client] (default-threads - 17) Reconnection successfull
06:07:17,960 DEBUG [org.hornetq.core.client] (default-threads - 17) NettyConnector [host=10.10.52.246, port=5445, httpEnabled=false, useServlet=false, servletPath=/messaging/HornetQServlet, sslEnabled=false, useNio=false] host 1: 10.10.52.115 ip address: 10.10.52.115 host 2: 10.10.52.246 ip address: 10.10.52.246
06:07:17,959 DEBUG [org.hornetq.core.client] (default-threads - 8) Reconnection successfull
06:07:17,959 DEBUG [org.hornetq.core.client] (Old I/O server worker (parentId: -724172059, [id: 0xd4d602e5, /10.10.52.246:5445])) Topology@c18c7e1[owner=ClusterConnectionImpl@1630353650[nodeUUID=0f746069-f52b-11e2-b02a-8bee9fa4e9b9, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-10-52-246, address=jms, server=HornetQServerImpl::serverUUID=0f746069-f52b-11e2-b02a-8bee9fa4e9b9]] is sending topology to Remote Proxy on channel 37389a2f
06:07:17,958 DEBUG [org.hornetq.core.client] (Thread-32 (HornetQ-client-global-threads-1266216150)) Node 776131b0-f422-11e2-9dfc-cfc70000de9f going up, connector = Pair[a=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-10-52-246, b=null], isLast=false csf created at
serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=2156ede6-f52b-11e2-b02a-8bee9fa4e9b9, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=JMSServer1, TransportConfiguration(name=2156ede7-f52b-11e2-b02a-8bee9fa4e9b9, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=JMSServer2], discoveryGroupConfiguration=null]: java.lang.Exception
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.<init>(ClientSessionFactoryImpl.java:197) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:808) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:311)
at org.hornetq.ra.inflow.HornetQActivation$SetupActivation.run(HornetQActivation.java:701)
at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:218)
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:806)
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:826)
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
06:07:17,960 DEBUG [org.hornetq.core.client] (Old I/O server worker (parentId: -724172059, [id: 0xd4d602e5, /10.10.52.246:5445])) Topology@c18c7e1[owner=ClusterConnectionImpl@1630353650[nodeUUID=0f746069-f52b-11e2-b02a-8bee9fa4e9b9, connector=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-10-52-246, address=jms, server=HornetQServerImpl::serverUUID=0f746069-f52b-11e2-b02a-8bee9fa4e9b9]] is sending topology to Remote Proxy on channel 77fe8305
06:07:17,960 DEBUG [org.hornetq.core.client] (default-threads - 8) NettyConnector [host=10.10.52.246, port=5445, httpEnabled=false, useServlet=false, servletPath=/messaging/HornetQServlet, sslEnabled=false, useNio=false] host 1: 10.10.52.115 ip address: 10.10.52.115 host 2: 10.10.52.246 ip address: 10.10.52.246
06:07:17,960 DEBUG [org.hornetq.core.client] (default-threads - 17) ClientSessionFactoryImpl received backup update for live/backup pair = TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-10-52-246 / null but it didn't belong to TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-10-52-246
06:07:17,961 DEBUG [org.hornetq.ra] (default-threads - 21) Using queue connection DelegatingSession [session=ClientSessionImpl [name=221ef91d-f52b-11e2-b02a-8bee9fa4e9b9, username=null, closed=false, factory = ClientSessionFactoryImpl [serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=215714f8-f52b-11e2-b02a-8bee9fa4e9b9, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=JMSServer1, TransportConfiguration(name=215714f9-f52b-11e2-b02a-8bee9fa4e9b9, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=JMSServer2], discoveryGroupConfiguration=null], connectorConfig=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-10-52-246, backupConfig=null], metaData=()]@7c484e61]
06:07:17,961 DEBUG [org.hornetq.core.client] (Thread-83 (HornetQ-client-global-threads-1266216150)) Node 0f746069-f52b-11e2-b02a-8bee9fa4e9b9 going up, connector = Pair[a=TransportConfiguration(name=netty, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=10-10-52-246, b=null], isLast=false csf created at
serverLocator=ServerLocatorImpl [initialConnectors=[TransportConfiguration(name=215455b2-f52b-11e2-b02a-8bee9fa4e9b9, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=JMSServer1, TransportConfiguration(name=215455b3-f52b-11e2-b02a-8bee9fa4e9b9, factory=org-hornetq-core-remoting-impl-netty-NettyConnectorFactory) ?port=5445&host=JMSServer2], discoveryGroupConfiguration=null]: java.lang.Exception
at org.hornetq.core.client.impl.ClientSessionFactoryImpl.<init>(ClientSessionFactoryImpl.java:197) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.core.client.impl.ServerLocatorImpl.createSessionFactory(ServerLocatorImpl.java:808) [hornetq-core-client-2.3.0.CR1.jar:]
at org.hornetq.ra.inflow.HornetQActivation.setup(HornetQActivation.java:311)
at org.hornetq.ra.inflow.HornetQActivation$SetupActivation.run(HornetQActivation.java:701)
at org.jboss.jca.core.workmanager.WorkWrapper.run(WorkWrapper.java:218)
at org.jboss.threads.SimpleDirectExecutor.execute(SimpleDirectExecutor.java:33)
at org.jboss.threads.QueueExecutor.runTask(QueueExecutor.java:806)
at org.jboss.threads.QueueExecutor.access$100(QueueExecutor.java:45)
at org.jboss.threads.QueueExecutor$Worker.run(QueueExecutor.java:826)
at java.lang.Thread.run(Thread.java:722) [rt.jar:1.7.0_04]
at org.jboss.threads.JBossThread.run(JBossThread.java:122)
Could you please share your test configuration where it is working? I'll check against my configuration.
Thanks,
Ravi
-
16. Re: Memory leak with clustered JMS Server
Hi Ravi,
sorry for late reply. I had a vacation. Here is my configuration.
Server 1:
<subsystem xmlns="urn:jboss:domain:messaging:1.3"> <hornetq-server> <persistence-enabled>true</persistence-enabled> <security-enabled>false</security-enabled> <cluster-password>${jboss.messaging.cluster.password:CHANGE ME!!}</cluster-password> <shared-store>true</shared-store> <journal-type>ASYNCIO</journal-type> <journal-file-size>10485760</journal-file-size> <journal-min-files>2</journal-min-files> <connectors> <netty-connector name="netty" socket-binding="messaging"/> <netty-connector name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50"/> </netty-connector> <in-vm-connector name="in-vm" server-id="0"/> </connectors> <acceptors> <netty-acceptor name="netty" socket-binding="messaging"/> <netty-acceptor name="netty-throughput" socket-binding="messaging-throughput"> <param key="batch-delay" value="50"/> <param key="direct-deliver" value="false"/> </netty-acceptor> <in-vm-acceptor name="in-vm" server-id="0"/> </acceptors> <broadcast-groups> <broadcast-group name="bg-group1"> <socket-binding>messaging-group</socket-binding> <broadcast-period>2000</broadcast-period> <connector-ref> netty </connector-ref> </broadcast-group> </broadcast-groups> <discovery-groups> <discovery-group name="dg-group1"> <socket-binding>messaging-group</socket-binding> <refresh-timeout>10000</refresh-timeout> </discovery-group> </discovery-groups> <cluster-connections> <cluster-connection name="my-cluster"> <address>jms</address> <connector-ref>netty</connector-ref> <retry-interval>1000</retry-interval> <use-duplicate-detection>true</use-duplicate-detection> <forward-when-no-consumers>false</forward-when-no-consumers> <max-hops>1</max-hops> <discovery-group-ref discovery-group-name="dg-group1"/> </cluster-connection> </cluster-connections> <security-settings> <security-setting match="#"> <permission type="send" roles="guest"/> <permission type="consume" roles="guest"/> <permission type="createNonDurableQueue" roles="guest"/> <permission type="deleteNonDurableQueue" roles="guest"/> </security-setting> </security-settings> <address-settings> <address-setting match="#"> <redelivery-delay>0</redelivery-delay> <max-size-bytes>52428800</max-size-bytes> <page-size-bytes>1048576</page-size-bytes> <address-full-policy>PAGE</address-full-policy> <redistribution-delay>0</redistribution-delay> </address-setting> </address-settings> <jms-connection-factories> <connection-factory name="InVmConnectionFactory"> <connectors> <connector-ref connector-name="in-vm"/> </connectors> <entries> <entry name="java:/ConnectionFactory"/> </entries> </connection-factory> <connection-factory name="RemoteConnectionFactory"> <connectors> <connector-ref connector-name="netty"/> </connectors> <entries> <entry name="java:jboss/exported/jms/RemoteConnectionFactory"/> </entries> </connection-factory> <pooled-connection-factory name="hornetq-ra"> <transaction mode="xa"/> <connectors> <connector-ref connector-name="in-vm"/> </connectors> <entries> <entry name="java:/JmsXA"/> </entries> </pooled-connection-factory> </jms-connection-factories> <jms-destinations> <jms-queue name="testQueue0"> <entry name="jms/queue/testQueue0"/> <entry name="java:jboss/exported/jms/queue/testQueue0"/> <durable>true</durable> </jms-queue> <jms-queue name="testQueue1"> <entry name="jms/queue/testQueue1"/> <entry name="java:jboss/exported/jms/queue/testQueue1"/> <durable>true</durable> </jms-queue> <jms-queue name="InQueue"> <entry name="jms/queue/InQueue"/> <entry name="java:jboss/exported/jms/queue/InQueue"/> <durable>true</durable> </jms-queue> <jms-queue name="OutQueue"> <entry name="jms/queue/OutQueue"/> <entry name="java:jboss/exported/jms/queue/OutQueue"/> <durable>true</durable> </jms-queue> </jms-destinations> </hornetq-server> </subsystem>Server2 config is the same as above.
Server 3 config with MDB. There is important only configuration of pooled-connection-factory. Also make sure that this server won't create HornetQ cluster with server1 and server2
<pooled-connection-factory name="hornetq-ra"> <transaction mode="xa"/> <connectors> <connector-ref connector-name="netty-remote"/> </connectors> <entries> <entry name="java:/JmsXA"/> </entries> </pooled-connection-factory> <connectors> ... <netty-connector name="netty-remote" socket-binding="messaging-remote"/> ... </connectors> <socket-binding-group name="standard-sockets" default-interface="public" port-offset="${jboss.socket.binding.port-offset:0}"> ... <outbound-socket-binding name="messaging-remote"> <remote-destination host="192.168.40.1" port="5445"/> </outbound-socket-binding> ... </socket-binding-group>Code of MDB:
package org.jboss.qa.hornetq.apps.mdb; import org.apache.log4j.Level; import org.apache.log4j.Logger; import javax.annotation.Resource; import javax.ejb.*; import javax.jms.*; import java.util.concurrent.atomic.AtomicInteger; /** * A MdbWithRemoteOutQueueToContaniner1 used in RemoteJcaTestCase. * This mdb reads messages from queue "InQueue" and sends to queue "OutQueue". */ @MessageDriven(name = "mdb1", activationConfig = { @ActivationConfigProperty(propertyName = "destinationType", propertyValue = "javax.jms.Queue"), @ActivationConfigProperty(propertyName = "destination", propertyValue = "jms/queue/InQueue")}) @TransactionManagement(value = TransactionManagementType.CONTAINER) @TransactionAttribute(value = TransactionAttributeType.REQUIRED) public class MdbWithRemoteOutQueueToContaniner1 implements MessageListener { private static final long serialVersionUID = 2770941392406343837L; private static final Logger log = Logger.getLogger(MdbWithRemoteOutQueueToContaniner1.class.getName()); private Queue queue = null; public static AtomicInteger numberOfProcessedMessages= new AtomicInteger(); @Resource(mappedName = "java:/JmsXA") private ConnectionFactory cf; @Resource private MessageDrivenContext context; @Override public void onMessage(Message message) { Connection con = null; Session session = null; try { long time = System.currentTimeMillis(); int counter = 0; try { counter = message.getIntProperty("count"); } catch (Exception e) { log.log(Level.ERROR, e.getMessage(), e); } String messageInfo = message.getJMSMessageID() + ", count:" + counter; log.debug(" Start of message:" + messageInfo); for (int i = 0; i < (5 + 5 * Math.random()); i++) { try { Thread.sleep((int) (10 + 10 * Math.random())); } catch (InterruptedException ex) { } } con = cf.createConnection(); session = con.createSession(false, Session.AUTO_ACKNOWLEDGE); if (queue == null) { queue = session.createQueue("OutQueue"); } con.start(); String text = message.getJMSMessageID() + " processed by: " + hashCode(); MessageProducer sender = session.createProducer(queue); TextMessage newMessage = session.createTextMessage(text); newMessage.setStringProperty("inMessageId", message.getJMSMessageID()); newMessage.setStringProperty("_HQ_DUPL_ID", message.getStringProperty("_HQ_DUPL_ID")); sender.send(newMessage); messageInfo = messageInfo + ". Sending new message with inMessageId: " + newMessage.getStringProperty("inMessageId") + " and messageId: " + newMessage.getJMSMessageID(); log.debug("End of " + messageInfo + " in " + (System.currentTimeMillis() - time) + " ms"); if (numberOfProcessedMessages.incrementAndGet() % 100 == 0) log.info(messageInfo + " in " + (System.currentTimeMillis() - time) + " ms"); } catch (Exception t) { log.error(t.getMessage(), t); this.context.setRollbackOnly(); } finally { if (con != null) { try { con.close(); } catch (JMSException e) { log.log(Level.FATAL, e.getMessage(), e); } } } } }Attribute reconnect-attempts is not set is by default -1 which means reconnect indefinitely. Attribute setup-attempts is not set and should have default value 10. There is no other activation-config property set for MDB.
Cheers,
Mirek
-
17. Re: Memory leak with clustered JMS Server
No problem Mirek, just curious in your configuration for netty-remote connector, you are pointing it to only one messaging server? Additionally in configuration of Server 1 and 2 Clustered = true is not set?
In my configuration, all 3 servers are forming cluster? Could that be a problem?