Security Exception| JBoss.org Content Archive (Read Only)

15. Re: Security Exception

timfox May 31, 2006 11:42 AM (in response to mwelss)

Great, thanks.

16. Re: Security Exception

timfox May 31, 2006 12:37 PM (in response to mwelss)

I haven't started looking at this yet, but just to get me started, you're saying that if you change the selector so it no longer has an "OR" expression, the leak disappears?

17. Re: Security Exception

mwelss May 31, 2006 2:40 PM (in response to mwelss)

Yes, exactly! The "OR" makes the difference.

18. Re: Security Exception

ovidiu.feodorov May 31, 2006 9:05 PM (in response to mwelss)

http://jira.jboss.org/jira/browse/JBMESSAGING-405

Clebert is taking a look at it.

19. Re: Security Exception

clebert.suconic Jun 2, 2006 6:45 PM (in response to mwelss)

I have added a testcase on jms, go to jboss-head/jms/test and call ant memory-leak-tests
(On windows only for now, using a JDK 5).

At the end of the test, I have these following unexpected objects. It doesn't seem a memory leak, just something that I didn't expect at this point.

[junit] <br> Class [Ljava.security.ProtectionDomain; had an increase of 10 instances represented by 400 bytes
[junit] <br> Class EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker had an increase of 10 instances represented by 160 bytes
[junit] <br> Class java.lang.ThreadLocal$ThreadLocalMap$Entry had an increase of 30 instances represented by 960 bytes
[junit] <br> Class java.util.HashMap$Entry had an increase of 10 instances represented by 240 bytes
[junit] <br> Class java.security.AccessControlContext had an increase of 10 instances represented by 240 bytes
[junit] <br> Class java.lang.Thread had an increase of 10 instances represented by 960 bytes
[junit] <br> Class [Ljava.lang.ThreadLocal$ThreadLocalMap$Entry; had an increase of 20 instances represented by 1600 bytes
[junit] <br> Class java.util.ArrayList had an increase of 10 instances represented by 240 bytes
[junit] <br> Class java.lang.ThreadLocal$ThreadLocalMap had an increase of 20 instances represented by 480 bytes

Anyway, another similar testcase, without using ObjectSerialization or expressions didn't have this.

This won't change if you produce 1000 or 100 messages.

Anyway, the testcase is working. Maybe we could work in top of that.

Clebert

20. Re: Security Exception

mwelss Jun 6, 2006 3:00 AM (in response to mwelss)

Clebert,

did you try my testcase? Can you reproduce the problem?

Best regards,

Martin

21. Re: Security Exception

timfox Jun 7, 2006 12:20 PM (in response to mwelss)

Hi Martin-

Hopefully we can investigate this soon.

Many of us are travelling this week and next week is JBoss World in Vegas so time (as usual) is a precious commodity right now.....

22. Re: Security Exception

clebert.suconic Jun 7, 2006 12:26 PM (in response to mwelss)

I tried to use the expression part only in a memory-leak testcase, and didn't have any luck. (it's not leaking)

I will expand to everything you have in your testcase. At this point I don't think the problem is the expressions.

take a look on MemoryLeakTest.java on jms project at jboss-head.

23. Re: Security Exception

mwelss Jun 8, 2006 3:51 AM (in response to mwelss)

I'am sorry I didn't put a readme in the zip-file, but it's very easy to reproduce the leak with my testcase, just take a jboss messaging configuration based on ejb3. Adjust the settings at the beginning of my build.xml to your needs and run ant (without parameters). The leak will be obvious within one or two minutes.

Enjoy JBoss World, I wish I could be there, too.

Martin

24. Re: Security Exception

timfox Jun 20, 2006 12:19 PM (in response to mwelss)

Hi Martin-

Sorry for taking so long to look at this, it's been a very busy few weeks.

I have run your test you sent to me using a fresh copy of JBoss 4.0.4GA, JDK5, and EJB3-RC8-FD.

I installed EJB3 fresh as per the EJB3 install instructions into the all configuration of the app server, and ran your test.

The only change I made was to make your test run against "all" rather than "messaging".

I let it run for just over 30 mins, during which time it sent and received almost 5000 messages (out of 20000).

During that time I didn't notice any increase in memory used.

Here is some snipped output from the beginning and the end of the run:

Near beginning:

2006-06-20 16:34:55,500 INFO [STDOUT] [Client] free: 64855KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:55,515 INFO [STDOUT] [Client] Message 276/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:55,531 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:55,531 INFO [STDOUT] [PlexMDB#2] message received: 3000
2006-06-20 16:34:55,875 INFO [STDOUT] [Client] free: 63907KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:55,890 INFO [STDOUT] [PlexMDB#3] message received: 3000
2006-06-20 16:34:55,890 INFO [STDOUT] [Client] Message 277/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:55,906 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:56,234 INFO [STDOUT] [Client] free: 63354KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:56,250 INFO [STDOUT] [Client] Message 278/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:56,265 INFO [STDOUT] [PlexMDB#4] message received: 3000
2006-06-20 16:34:56,296 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:56,656 INFO [STDOUT] [Client] free: 65304KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:56,687 INFO [STDOUT] [PlexMDB#0] message received: 3000
2006-06-20 16:34:56,687 INFO [STDOUT] [Client] Message 279/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:56,703 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:57,031 INFO [STDOUT] [Client] free: 64257KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:57,093 INFO [STDOUT] [PlexMDB#1] message received: 3000
2006-06-20 16:34:57,093 INFO [STDOUT] [Client] Message 280/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:57,265 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:57,609 INFO [STDOUT] [Client] free: 64499KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:57,609 INFO [STDOUT] [Client] Message 281/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:57,625 INFO [STDOUT] [PlexMDB#3] message received: 3000
2006-06-20 16:34:57,656 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:58,000 INFO [STDOUT] [Client] free: 64168KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:58,000 INFO [STDOUT] [Client] Message 282/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:58,015 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:58,390 INFO [STDOUT] [PlexMDB#2] message received: 3000
2006-06-20 16:34:58,421 INFO [STDOUT] [Client] free: 67122KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:58,437 INFO [STDOUT] [PlexMDB#4] message received: 3000
2006-06-20 16:34:58,437 INFO [STDOUT] [Client] Message 283/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:58,453 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:58,781 INFO [STDOUT] [Client] free: 69214KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:58,781 INFO [STDOUT] [Client] Message 284/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:58,796 INFO [STDOUT] [PlexMDB#1] message received: 3000
2006-06-20 16:34:58,828 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:59,156 INFO [STDOUT] [Client] free: 68480KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:59,171 INFO [STDOUT] [PlexMDB#3] message received: 3000
2006-06-20 16:34:59,187 INFO [STDOUT] [Client] Message 285/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:59,187 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:59,531 INFO [STDOUT] [Client] free: 67763KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:59,531 INFO [STDOUT] [Client] Message 286/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:59,546 INFO [STDOUT] [PlexMDB#0] message received: 3000
2006-06-20 16:34:59,578 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:34:59,937 INFO [STDOUT] [Client] free: 70032KB / total: 130240KB / max: 520256KB
2006-06-20 16:34:59,937 INFO [STDOUT] [Client] Message 287/20000 was successfully sent to the testQueue queue
2006-06-20 16:34:59,937 INFO [STDOUT] [PlexMDB#2] message received: 3000
2006-06-20 16:34:59,953 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:00,281 INFO [STDOUT] [Client] free: 69580KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:00,296 INFO [STDOUT] [Client] Message 288/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:00,312 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:00,640 INFO [STDOUT] [Client] free: 68893KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:00,656 INFO [STDOUT] [Client] Message 289/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:00,656 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:00,656 INFO [STDOUT] [PlexMDB#4] message received: 3000
2006-06-20 16:35:00,671 INFO [STDOUT] [PlexMDB#1] message received: 3000
2006-06-20 16:35:01,015 INFO [STDOUT] [Client] free: 67722KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:01,031 INFO [STDOUT] [PlexMDB#3] message received: 3000
2006-06-20 16:35:01,031 INFO [STDOUT] [Client] Message 290/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:01,062 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:01,421 INFO [STDOUT] [Client] free: 68607KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:01,437 INFO [STDOUT] [Client] Message 291/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:01,453 INFO [STDOUT] [PlexMDB#0] message received: 3000
2006-06-20 16:35:01,484 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:01,812 INFO [STDOUT] [Client] free: 67811KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:01,828 INFO [STDOUT] [Client] Message 292/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:01,843 INFO [STDOUT] [PlexMDB#2] message received: 3000
2006-06-20 16:35:02,125 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:02,500 INFO [STDOUT] [Client] free: 67067KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:02,531 INFO [STDOUT] [PlexMDB#1] message received: 3000
2006-06-20 16:35:02,546 INFO [STDOUT] [Client] Message 293/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:02,578 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:02,906 INFO [STDOUT] [Client] free: 65983KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:02,921 INFO [STDOUT] [Client] Message 294/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:02,937 INFO [STDOUT] [PlexMDB#3] message received: 3000
2006-06-20 16:35:02,968 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:03,328 INFO [STDOUT] [Client] free: 68501KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:03,343 INFO [STDOUT] [Client] Message 295/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:03,343 INFO [STDOUT] [PlexMDB#0] message received: 3000
2006-06-20 16:35:03,375 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:03,687 INFO [STDOUT] [Client] free: 67596KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:03,703 INFO [STDOUT] [PlexMDB#4] message received: 3000
2006-06-20 16:35:03,734 INFO [STDOUT] [Client] Message 296/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:03,734 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:04,062 INFO [STDOUT] [Client] free: 66903KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:04,062 INFO [STDOUT] [Client] Message 297/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:04,078 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:04,406 INFO [STDOUT] [PlexMDB#1] message received: 3000
2006-06-20 16:35:04,437 INFO [STDOUT] [Client] free: 64230KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:04,437 INFO [STDOUT] [Client] Message 298/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:04,453 INFO [STDOUT] [PlexMDB#2] message received: 3000
2006-06-20 16:35:04,468 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:04,828 INFO [STDOUT] [Client] free: 67827KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:04,843 INFO [STDOUT] [Client] Message 299/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:04,859 INFO [STDOUT] [PlexMDB#3] message received: 3000
2006-06-20 16:35:04,859 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:05,187 INFO [STDOUT] [Client] free: 67123KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:05,218 INFO [STDOUT] [Client] Message 300/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:05,218 INFO [STDOUT] [PlexMDB#0] message received: 3000
2006-06-20 16:35:05,250 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:05,578 INFO [STDOUT] [Client] free: 66389KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:05,578 INFO [STDOUT] [PlexMDB#4] message received: 3000
2006-06-20 16:35:05,593 INFO [STDOUT] [Client] Message 301/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:05,609 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:05,953 INFO [STDOUT] [Client] free: 65670KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:05,968 INFO [STDOUT] [Client] Message 302/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:05,968 INFO [STDOUT] [PlexMDB#1] message received: 3000
2006-06-20 16:35:06,000 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 16:35:06,359 INFO [STDOUT] [Client] free: 67324KB / total: 130240KB / max: 520256KB
2006-06-20 16:35:06,375 INFO [STDOUT] [Client] Message 303/20000 was successfully sent to the testQueue queue
2006-06-20 16:35:06,375 INFO [STDOUT] [PlexMDB#2] message received: 3000

And end of run:

2006-06-20 17:06:33,000 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:33,375 INFO [STDOUT] [Client] free: 64431KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:33,375 INFO [STDOUT] [PlexMDB#3] message received: 3000
2006-06-20 17:06:33,390 INFO [STDOUT] [Client] Message 4646/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:33,421 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:33,796 INFO [STDOUT] [Client] free: 71311KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:33,796 DEBUG [org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread] Shutdown
2006-06-20 17:06:33,796 INFO [STDOUT] [Client] Message 4647/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:33,812 INFO [STDOUT] [PlexMDB#0] message received: 3000
2006-06-20 17:06:33,828 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:34,171 INFO [STDOUT] [Client] free: 70747KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:34,187 INFO [STDOUT] [Client] Message 4648/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:34,187 INFO [STDOUT] [PlexMDB#4] message received: 3000
2006-06-20 17:06:34,234 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:34,578 INFO [STDOUT] [Client] free: 70096KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:34,578 INFO [STDOUT] [Client] Message 4649/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:34,593 INFO [STDOUT] [PlexMDB#1] message received: 3000
2006-06-20 17:06:34,609 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:34,953 INFO [STDOUT] [Client] free: 69290KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:34,968 INFO [STDOUT] [Client] Message 4650/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:34,968 INFO [STDOUT] [PlexMDB#2] message received: 3000
2006-06-20 17:06:35,000 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:35,453 INFO [STDOUT] [Client] free: 70486KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:35,468 INFO [STDOUT] [Client] Message 4651/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:35,468 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:35,484 INFO [STDOUT] [PlexMDB#3] message received: 3000
2006-06-20 17:06:35,843 INFO [STDOUT] [Client] free: 69793KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:35,859 INFO [STDOUT] [PlexMDB#0] message received: 3000
2006-06-20 17:06:35,859 INFO [STDOUT] [Client] Message 4652/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:35,890 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:36,250 INFO [STDOUT] [Client] free: 69193KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:36,265 INFO [STDOUT] [PlexMDB#4] message received: 3000
2006-06-20 17:06:36,281 INFO [STDOUT] [Client] Message 4653/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:36,296 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:36,640 INFO [STDOUT] [Client] free: 68521KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:36,656 INFO [STDOUT] [Client] Message 4654/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:36,656 INFO [STDOUT] [PlexMDB#1] message received: 3000
2006-06-20 17:06:36,687 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:37,062 INFO [org.apache.catalina.core.StandardWrapper] Waiting for 1 instance(s) to be deallocated
2006-06-20 17:06:37,062 INFO [STDOUT] [Client] free: 69636KB / total: 130240KB / max: 520256KB
2006-06-20 17:06:37,078 INFO [STDOUT] [PlexMDB#2] message received: 3000
2006-06-20 17:06:37,109 INFO [STDOUT] [Client] Message 4655/20000 was successfully sent to the testQueue queue
2006-06-20 17:06:37,125 INFO [STDOUT] [Client] Received messages, size: 4000
2006-06-20 17:06:37,468 INFO [STDOUT] [Client] free: 68802KB / total: 130240KB / max: 520256KB

As you can see I didn't observe any leaking of memory. It seems to stabilise around the 64-68K point on my system

Under what timescales did you observe a leak?

Do I need to run it for longer?

25. Re: Security Exception

smievoll Jun 21, 2006 3:48 AM (in response to mwelss)

We're also experiencing this leak (or at-least something related, and something we didn't see before upgrading from JBossMQ), but for us it takes a long time to become a real problem.

We run 4.0.4.GA with 1.0.1.CR2 on Sun JDK 1.5.0_07, with about 5 m/s. With 1.5GB max heap it takes about 4 - 6 days before we run out of memory, seeing a steady growth from heap monitoring.

We're browsing the queue with a LIKE expression before every put to the queue (optimistically and rather desperately rewritten from a OR exp when we saw this case, but it hasn't helped :-).

As for the missing properties files, we were experienced it to but fixed it by setting up a database backed JNDI realm, using the JMS_USER and JMS_ROLE tables just adding guest to them.

Apologize for not being able to give more concrete feedback, but hopefully we'll have more time to debug the issue in the coming weeks.

26. Re: Security Exception

timfox Jun 21, 2006 4:26 AM (in response to mwelss)

Ok, I am going to run Martin's test program on a different machine and let it run for days to see what happens.

bos - if you can reproduce your issue with a small test program that would be great too - do you only see your problem when you use selectors?

27. Re: Security Exception

mwelss Jun 21, 2006 4:55 AM (in response to mwelss)

Tim,

the memory leak becomes visible very quickly with my testcase, it will eat 1-2 MB every minute.
BUT: I didn't try your configuration yet, I can reproduce this reliably on jboss 4.0.4.GA-patch1 with ejb3 config and jboss-messaging 1.0.1CR2. I will try with EJB3-RC8 this afternoon and keep you informed :-)

Best regards,

Martin

28. Re: Security Exception

smievoll Jun 21, 2006 5:09 AM (in response to mwelss)

Sorry, have not been able to test without selectors yet. I'll try out some different senarios as soon as I can, and produce some test code. But unfortunately I won't be able to do anything before friday at the earliest.

29. Re: Security Exception

timfox Jun 21, 2006 5:15 AM (in response to mwelss)

"bos" wrote:
Sorry, have not been able to test without selectors yet. I'll try out some different senarios as soon as I can, and produce some test code. But unfortunately I won't be able to do anything before friday at the earliest.

Np, thanks for your help anyway.