-
1. Re: JBoss AS 7.1.1 Final XA datasources memory leak
jesper.pedersen Sep 7, 2012 7:59 AM (in response to karl_zhang)1) Yes, add the recover-plugin element, and make sure that you have a connection checker defined
2) Use a nightly snapshot of AS, or build one of the branches for yourself. IronJacamar 1.1 is for EE7, so that won't work in AS7
3) Use 902 - or build latest Git master / REL_9_2_STABLE yourself - the key is to have a connection checker defined
-
2. Re: JBoss AS 7.1.1 Final XA datasources memory leak
karl_zhang Sep 7, 2012 8:03 AM (in response to jesper.pedersen)Dear Jesper,
Thank you for your detail explanation.
Best Regards,
Karl Gross
-
3. Re: JBoss AS 7.1.1 Final XA datasources memory leak
karl_zhang Sep 10, 2012 5:16 AM (in response to jesper.pedersen)Dear Jesper,
I did the following according to your instructions:
1. download a AS7 nightly build: https://ci.jboss.org/jenkins/job/JBoss-AS-7.x-latest/lastSuccessfulBuild/artifact/build/target/jboss-as-7.x.zip
2. update ironjacamar module to 1.1.0.Beta1, use postgres 9.1-902 JDBC 4
3. add the following elements in xa-datasource defnition:
<recovery>
<recover-plugin class-name="org.jboss.jca.core.recovery.ValidatingManagedConnectionFactoryRecoveryPlugin"/>
</recovery>
<validation>
<valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker"></valid-connection-checker>
<exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLExceptionSorter"></exception-sorter>
</validation>
* In JBTM-1183 the attribute no-recovery="false" is recommended also. But I could not use it, because JBoss gave an error during booting: Caused by: org.jboss.msc.service.StartException in anonymous service: JBAS010432: Unable to start the ds because it generated more than one cf.
Unfortunately the class com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule still used 74% of the total heap memory.
Could you please tell me, if I have done correctly? Or do you know any other issues related to this class? Thank you!
Best Regards,
Karl Gross
-
4. Re: JBoss AS 7.1.1 Final XA datasources memory leak
jesper.pedersen Sep 10, 2012 10:44 AM (in response to karl_zhang)2) You can't use IronJacamar 1.1 in AS7 - just use nightly build directly
3) We will look at the no-recovery issue
Please, provide an Eclipse MAT report that shows what is holding on to the instances.
-
5. Re: JBoss AS 7.1.1 Final XA datasources memory leak
karl_zhang Sep 10, 2012 11:26 AM (in response to jesper.pedersen)Dear Jesper,
thank you for your quick response. I will check it again using the nightly build directly.
A report from Eclipse MAT:
Class Name | Shallow Heap | Retained Heap | Percentage ------------------------------------------------------------------------------------------------------------------------------------------
.*com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule.* | | | com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule @ 0x6c89b628 | 48 | 673.829.016 | 73,88% |- java.util.Hashtable @ 0x6c9d7d38 | 40 | 673.828.208 | 73,88% | |- java.util.Hashtable$Entry[24575] @ 0x9014c2f8 | 98.312 | 673.828.136 | 73,88% | | |- java.util.Hashtable$Entry @ 0xa1e562c0 | 24 | 230.872 | 0,03% | | | |- java.util.Hashtable$Entry @ 0xa1604c78 | 24 | 192.696 | 0,02% | | | |- org.jboss.jca.core.tx.jbossts.XAResourceWrapperImpl @ 0xa1e59460 | 32 | 37.848 | 0,00% | | | |- com.arjuna.ats.internal.jta.recovery.arjunacore.RecoveryXids @ 0xa1e59480 | 32 | 304 | 0,00% | | | '- Total: 3 entries | | | | | |- java.util.Hashtable$Entry @ 0x90140630 | 24 | 230.864 | 0,03% | | |- java.util.Hashtable$Entry @ 0x9cd1e4b0 | 24 | 192.688 | 0,02% | | |- java.util.Hashtable$Entry @ 0x9db84dd8 | 24 | 192.688 | 0,02% | | |- java.util.Hashtable$Entry @ 0x9ec43e28 | 24 | 192.688 | 0,02% | | |- java.util.Hashtable$Entry @ 0x65fc69d8 | 24 | 192.680 | 0,02% | | |- java.util.Hashtable$Entry @ 0x9e38b9a0 | 24 | 192.680 | 0,02% | | |- java.util.Hashtable$Entry @ 0xa1e560e0 | 24 | 192.664 | 0,02% | | |- java.util.Hashtable$Entry @ 0x65fc8508 | 24 | 192.328 | 0,02% | | |- java.util.Hashtable$Entry @ 0x9db83f20 | 24 | 192.328 | 0,02% | | |- java.util.Hashtable$Entry @ 0x9c4a6ad0 | 24 | 192.320 | 0,02% | | |- java.util.Hashtable$Entry @ 0x9cd1e6d8 | 24 | 192.320 | 0,02% | | |- java.util.Hashtable$Entry @ 0xa05151f0 | 24 | 192.320 | 0,02% | | |- java.util.Hashtable$Entry @ 0x9a24d188 | 24 | 191.968 | 0,02% | | |- java.util.Hashtable$Entry @ 0xa0d8c3d8 | 24 | 191.968 | 0,02% | | |- java.util.Hashtable$Entry @ 0x999de8e0 | 24 | 191.960 | 0,02% | | |- java.util.Hashtable$Entry @ 0x9f483d40 | 24 | 154.168 | 0,02% | | |- java.util.Hashtable$Entry @ 0x65fc66c0 | 24 | 154.160 | 0,02% | | |- java.util.Hashtable$Entry @ 0x81ef5778 | 24 | 154.160 | 0,02% | | |- java.util.Hashtable$Entry @ 0x65fc9510 | 24 | 154.160 | 0,02% | | |- java.util.Hashtable$Entry @ 0x65fc9f18 | 24 | 154.160 | 0,02% | | |- java.util.Hashtable$Entry @ 0x65fc6678 | 24 | 154.160 | 0,02% | | |- java.util.Hashtable$Entry @ 0x90182ce0 | 24 | 154.160 | 0,02% | | |- java.util.Hashtable$Entry @ 0x96bd9488 | 24 | 154.160 | 0,02% | | |- java.util.Hashtable$Entry @ 0x97567ec0 | 24 | 154.160 | 0,02% | | '- Total: 25 of 17.746 entries; 17.721 more | | | | |- java.util.Collections$SynchronizedSet @ 0x6ca89178 | 16 | 32 | 0,00% | '- Total: 2 entries | | | |- java.util.LinkedList @ 0x6c9d7d08 | 24 | 504 | 0,00% |- java.lang.String @ 0x6bdff5f8 Local XARecoveryModule | 24 | 80 | 0,00% |- java.util.ArrayList @ 0x6c9d7d68 | 24 | 80 | 0,00% |- java.util.ArrayList @ 0x6c9d7d20 | 24 | 48 | 0,00% |- java.util.ArrayList @ 0x6c9d7cf0 | 24 | 40 | 0,00% |- com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryResourceManagerImple @ 0x6c9d7d60| 8 | 8 | 0,00% '- Total: 7 entries | | | class com.arjuna.ats.internal.jta.recovery.arjunacore.XARecoveryModule @ 0x6be237f8 | 0 | 0 | 0,00% Total: 2 entries (219.077 filtered) | | | ------------------------------------------------------------------------------------------------------------------------------------------
-
6. Re: JBoss AS 7.1.1 Final XA datasources memory leak
karl_zhang Sep 11, 2012 11:07 AM (in response to jesper.pedersen)Dear Jesper,
I started my JBoss instance yestoday, simply using a nightly build. Meanwhile the retained heap for XARecoveryModule was already about 50% of the total heap size. I have no ideas how to proceed with this. The configuration of my datasource looks like:
<xa-datasource jndi-name="java:/server.ds.core" pool-name="server.ds.core-pool" enabled="true" use-java-context="true">
<xa-datasource-property name="DatabaseName">
server_db
</xa-datasource-property>
<xa-datasource-property name="User">
user
</xa-datasource-property>
<xa-datasource-property name="ServerName">
server
</xa-datasource-property>
<xa-datasource-property name="PortNumber">
5432
</xa-datasource-property>
<xa-datasource-property name="Password">
xxx
</xa-datasource-property>
<xa-datasource-class>org.postgresql.xa.PGXADataSource</xa-datasource-class>
<driver>postgresql-jdbc4</driver>
<transaction-isolation>TRANSACTION_SERIALIZABLE</transaction-isolation>
<xa-pool>
<min-pool-size>1</min-pool-size>
<max-pool-size>75</max-pool-size>
<use-strict-min>true</use-strict-min>
</xa-pool>
<security>
<user-name>user</user-name>
<password>xxx</password>
</security>
<recovery>
<recover-plugin class-name="org.jboss.jca.core.recovery.ValidatingManagedConnectionFactoryRecoveryPlugin"/>
</recovery>
<validation>
<valid-connection-checker class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLValidConnectionChecker"></valid-connection-checker>
<exception-sorter class-name="org.jboss.jca.adapters.jdbc.extensions.postgres.PostgreSQLExceptionSorter"></exception-sorter>
<check-valid-connection-sql>select 1</check-valid-connection-sql>
</validation>
<timeout>
<idle-timeout-minutes>5</idle-timeout-minutes>
</timeout>
<statement>
<track-statements>TRUE</track-statements>
</statement>
</xa-datasource>
<drivers>
<driver name="postgresql-jdbc4" module="org.postgresql"/>
</drivers>
The version of ironjacamar in the night build JBoss is 1.0.12. I am using postgresql-9.1-902.jdbc4.jar. Could you please tell me, if I have done something wrong in the configuration? Thank you!
Best Regards,
Karl Gross
-
7. Re: JBoss AS 7.1.1 Final XA datasources memory leak
jesper.pedersen Sep 11, 2012 11:15 AM (in response to karl_zhang)You don't need the <check-valid-connection-sql> element, otherwise it looks good. See if you can upload a small heap dump (compressed) to this thread that shows the problem.
-
8. Re: JBoss AS 7.1.1 Final XA datasources memory leak
jesper.pedersen Sep 11, 2012 2:51 PM (in response to karl_zhang)Could you try and see if this completely untested jar makes a difference ?
-
ironjacamar-core-impl.jar 258.9 KB
-
-
9. Re: JBoss AS 7.1.1 Final XA datasources memory leak
karl_zhang Sep 14, 2012 5:29 AM (in response to jesper.pedersen)Dear Jesper,
thank you for creating the jar file. Unfortunately exchanging the jar file has made no difference. After JBoss was running not more than two days, the heap size for XARecoveryModule reached already 400MB.
Is this problem also related to the not implemented method Connection.isValid in postgresql-9.1-902?
Because I am testing a migrated project from JBoss 4, the memory dump is at least 600MB. It is difficult for me to provide a small dump file.
Best Regards,
Karl Gross
-
10. Re: JBoss AS 7.1.1 Final XA datasources memory leak
karl_zhang Sep 17, 2012 4:24 AM (in response to jesper.pedersen)Dear Jesper,
I have uploaded a compressed heap dump file with the size of 21MB. You can get it from http://www.2shared.com/file/jznfgHor/datasource-memory-leak-idp-dum.html. The dump is from our JBoss instance for single sign on.
Could you please help us to analyze this dump file and find a workaround also applicable for JBoss 7.1.1?
Because even if the workaround works for JBoss nightly build, it is not enough for users who want to migrate to JBoss 7 at the moment. Moreover no timeline for the next release can be found in the JBoss AS roadmap. This uncertainty will also make lot of users hesitate to change to JBoss7.
Thank you for your help!
Best Regards,
Karl Gross
-
11. Re: JBoss AS 7.1.1 Final XA datasources memory leak
jhalliday Sep 17, 2012 8:40 AM (in response to karl_zhang)ummm, oops. per my comment on JBTM-1183, the work in JBTM-924 should actually prevent this even where isValid is not working. Unfortunately that fix got scrambled by some later work, effectively creating a regression. We're on the case.
-
12. Re: JBoss AS 7.1.1 Final XA datasources memory leak
karl_zhang Sep 17, 2012 9:06 AM (in response to jhalliday)Dear Jonathan,
thank you for your reply! Because we are now almost at the end of migrating our application to JEE6, hopefully we could get a solution as soon as possible.
Best Regards,
Karl Gross
-
13. Re: JBoss AS 7.1.1 Final XA datasources memory leak
jhalliday Sep 17, 2012 9:17 AM (in response to karl_zhang)In that case reproduce it on your EAP server, log in to the customer portal and create a support ticket for it, referencing this thread. That way it's covered by the subscription SLA rather than the community 'when we get to it' model. We don't do community patch releases, so without a subscription you're waiting for a while.
-
14. Re: JBoss AS 7.1.1 Final XA datasources memory leak
didier2011 Sep 17, 2012 11:10 AM (in response to jhalliday)We're also experiencing this memory leak and need a fix for this, urgently, but we want a patch for 7.1.1 community version!
But as I understand this thread, there is no patch available at the moment? Isn't this a very heavy bug? We cannot run our application longer than three days without memory exception.