JBossDeveloper

Hi,

My three week summer holiday just started, but I will try that as soon as I get back to work. I did already try switching the memory hog bundle to deployments but it still used up all the memory.

I'll get back to you on this on the first week of August.

-Hannu

Today, I tried to deploy all bundles as regular WFLY deployments. The same performance problems persist.

-Hannu

Would you like to try and find the exact runaway dependency chain or do you just want to take a look at how the exports bloom out of control in general? I don't think I can mock the full setup we have.

Here is a zip with the bundles I used when I recreated the effects in the first post. 2 standalone.xmls and spring-context versions. In the FIXED -version the circular dependency has been removed.

https://www.dropbox.com/s/uc5oyhnjp7ex4ay/slow_proto.zip?dl=0

Hi,

I tested this with our full setup by adding all our jars to the "deployments" -folder. This caused the Java process to fill memory and hang on startup. But it's promising that this fixed the issue on the demo setup. I tested the demo zip on my home computer today, added jars as regular deployments and got the same results as you.

I'm actually home on sick leave and can't test further with the full setup right now, but as soon as I get back to work I'll look more into this. Maybe I can chop the startup into few parts with CLI-scripts.

Thanks for testing it. This gives me new hope on solving the issue. I'll keep you updated.

-Hannu

Hi,

So I might have spoken too soon. After testing this multiple times here at work this morning, it seems this does sometimes work without issues. Other times it runs oom and the start fails. I used the deployments folder to deploy the jars.

How do you use the CLI script? I tried starting the server from standalone.bat first and then running "jboss-cli.bat --connect --file=spring.cli". That did work. Server started in under 5 seconds but it took over 30 seconds after that to deploy all the bundles with the CLI script.

It is still interesting that deploying the jars with deployment-scanner seems to work sometimes, whereas adding the jars as capabilities seemed to fail every time.

Here's a log file where I started the server as many times as necessary to not get OOM error. After the first two starts, I did not wait for the server to start fully, but instead, after the first error, killed it and started it again. Last start succeeded in under 8 seconds.

https://dl.dropboxusercontent.com/u/47204890/512mb-multiple-boots-last-one-successful.log

Another log file where I increased the -Xmx to 1024mb (to show it is not just at 512mb).

https://dl.dropboxusercontent.com/u/47204890/1024mb-3-boots-1-successful.log

Just to make sure I understand what you did - the FIXED spring-context had "orm.jpa.support" removed, right? And that's it?

Checked that now. Yes, that's correct.

On my machine this occurs around 50% of the time with this setup:

1. Unzip clean wildfly 10.

2. Unzip clean JBOSGi 2.5.2.

3. Unzip this zip to wildfly/standalone -folder: Dropbox - reproduce_bug.zip

4. Start from standalone.bat

Sometimes it seems to start normally(7-8 seconds) multiple times in a row, but it has never started 10 times in a row without OOME.

We were actually having another concurrency issue as well. Our normal setup was to have all OSGi bundles as capabilities and webapps(WARs) in the deployments folder. Webapps do not find some transitive dependencies and fail to start. If we manually set the amount of service threads to 1, webapps start without issues.

Is there something else I can do to help locate the race condition?

Sad to report, this has not fixed the issue for me. Do you mean you were never able to reproduce it or after applying the fix?

I followed the steps I gave in my last post but replaced 2.5.2 with 2.5.3 and removed javax.api from standalone.xml. I actually went trough 2 clean installations to make sure I did everything correctly. Both times I booted up 5+ times and more than half of those boot up attempts failed to an OOM error.

As seen in the logs, I'm using Windows 7 and my JRE version is 1.8.0_91-b14. You seem to be using Linux and a little never version of JRE. Can those cause a notable difference in the frequency that the OOME is occurring?