[arcivanov]

Ugh.
>> Do you mean you were never able to reproduce it or after applying the fix?

I mean your reproducible reproduce_bug.zip always resulted in an OOM for me, while the 2.5.3.Final-SNAPSHOT started within 7s every time with no OOM.

Here's my working theory for now. There is something going on with path resolution inside JBoss Modules when two separate modules export identical paths/packages. The problem is that the recursive module tracker (Module$Visitor) comprised of module name and different path filters constantly mutates while traversing paths and basically never detects an infinite loop. This doesn't seem to be anything that JBOSGI does per se (the fix in the SNAPSHOT adds one missing package that you tried to fill in by adding javax.api capability, which is embedded in the core).

[hannu1]

There's actually a cyclic dependency between javax.api and javax.sql.api which I also once thought could be the cause for the issue. I tried removing the dependency from module.xmls in both ends, but it didn't fix the issue, so I gave up on that.

Your theory sounds solid. When I was debugging, I also noticed that same dependencies were cycled through multiple times even though addExportedPaths() method has that "if (visited) return;" clause at the start.

Do you think this is an issue that should be raised separately on the JBoss Modules board? Do you have any contact with the core developers of JBoss Modules?

[arcivanov]

If the cyclic dependency between javax.sql.api and javax.api were a real WildFly wouldn't be functional - those are stock dependencies. API exports SQL API, but not the other way around.

I'll try to get back to a reproducible case and if I bump into something more substantial I'll file a bug with jboss modules core.

[arcivanov]

Hi Hannu,

Just to give you a status update - I think I pinpointed the problem with Lock mechanism in JBOSGI, which leads to the problems. I almost file a bug with modules, but as I was gathering supporting documentation for it I saw something peculiar in the logs related to framework logging.

[arcivanov]

Hannu, sorry, unfortunately while the locking problem is real, in theory, it does not affect this particular issue. The addExportedPath madness starts after OSGI completely resolves all modules and WildFly initiates module loading.
I'm filing a bug with Modules, to see what they'll say.

Meanwhile, the investigation so far produced: 

[MODULES-263] Blooming addExportedPaths explodes visited set - JBoss Issue Tracker

[JBOSGI-799] Blooming dependency resolution with Spring bundles - JBoss Issue Tracker

[JBOSGI-798] WildFly Framework Lock integration uses incorrect locking - JBoss Issue Tracker

[JBOSGI-797] Felix Resolver update to 1.8.0 - JBoss Issue Tracker

[JBOSGI-796] Integration desynce'd with javax.api packages - JBoss Issue Tracker

[hannu1]

All right. Thank you for taking the time to investigate this so thoroughly and keeping me updated. Let's hope Modules developers can track down the underlying issue.

[arcivanov]

Yep. Meanwhile, here are some other things I've found out from the reproducible case you've given me:

JBOSGI already includes Apache Aries - Blueprint. You deploy Eclipse Gemini Blueprint - Home as a dependency for other projects. Aries BP is a dependency for capabilities in Aries JNDI, JMX. I'm not entirely sure you want two BP providers available. If you could try disabling Aries BP capability (leaving other Aries intact) while also adding Eclipse Gemini BP as a capability deploying it into bundles, it might be a more "correct" deployment strategy. Alternatively, if you simply need OSGi BP without any proprietary Eclipse Gemini extensions, Aries should do just fine and you might not need Gemini at all.

[hannu1]

Hi,

We're dependent on Eclipse Gemini extensions. In the full setup I have left Aries Blueprint out, but it was done when all the bundles were deployed as capabilities. Eclipse Gemini Blueprint is dependent on Spring and Aop. If I were to add Gemini Blueprint as a capability, I would have to add Spring and Aop as capabilities too?

[arcivanov]

Understood. It was merely a suggestion as in my 16GB heap experiment they deployed side-by-side without apparent adverse effects (I obviously don't have an ability to test whether the application is actually working beyond simple resolution).

[arcivanov]

Hannu,

Well, unfortunately, this looks like the JBoss Modules problem judging by the latest comment in [MODULES-263] Blooming addExportedPaths explodes visited set - JBoss Issue Tracker.

Unfortunately, I'm not in a position to work on the core WildFly modules unless someone is actually interested in sponsoring the project. If this is something that you or your company might be interested in, please PM me.

[hannu1]

Hi Arcadiy,

Ok. Thank you for helping us track the cause of the issue.

I sent you an email about the sponsorship.