[alesj]

"jesper.pedersen" wrote:

The goal of Papaki is to provide a simple API for developers and thereby being able to deploy the library in any environment (like one of the current users on WebSphere).

Sure, but why re-invent the wheel?
* your Reflection vs. Javassist abstraction == JBoss Reflect
* your split between jar or directory == JBoss VFS

e.g. if you wanna add a restriction check (e.g. exclude any non-exported class by OSGi manifest),
you have to do it in 4 diff places:
* Reflect + jar
* Reflect + directory
* Javassist + jar
* Javassist + directory

With VFS this is a simple filter; recursion or plain resource.
Reflection abstraction is a single TypeInfoFactory change; via setter.

"jesper.pedersen" wrote:

If the current API and scope of the project doesn't fit into the MC project I welcome a fork of the code into a new project where we can share ideas across the projects.

Yup, that's what I think I'll do.

But I'm still confused by where your version of Papaki really can be used.
As it clearly lacks proper integration layer with existing environments,
be it MC, JBossAS, WebsFear, ...
While building repo, either from existing papaki.ser or runtime, you should be able to hook into existing restriction mechanisms.
Otherwise you can potentially load a bunch of code user restricted at runtime;
jboss-clasloading.xml, jboss-scanning.xml, ...
(and its matching metadata instances)

[alesj]

So - we are taking a small project, adding dependencies and making it more complex?

Why is adding dependecies making it more complex?
It's exactly the opposite - less code to maintain.

Adding dependencies is a bad idea. Projects should as a rule have the minimum number of runtime dependencies possible. Having lots of dependencies just makes it harder to refactor later. If you have a large web of dependencies between a set of projects, that's a big indicator that they ought to be one big project anyway, or at least split up differently.

Again nonsense.
Refactoring is extremely easy if you have an external thing that does things for you.
See my previous e.g.: "e.g. if you wanna add a restriction check"

In most cases those ext libs have stand the test of time,
so they are much better then the crap you quickly code yourself.
And the nicest things about all the pieces in my version of Papaki is that we control them all,
meaning it's easy to create any missing feature or fix/extend an existing one.

[alesj]

So we have to index all of them (unless there is some hint from the user that scanning is not necessary for a class or package or whatever)

Yes.
The restriction mechanism should be plugable,
and the one's I can think of right now are:
* OSGi manifest export header
* jboss-classloading.xml capabilities

and remember (sans-classloading) where annotations are located, and the relationships between classes, so that we can find all classes and subclasses with certain annotations without actually loading any classes.

This sounds interesting, but what's the easiest way to achieve this?
I guess Javassist knows how to handle this.
How hard would it be to add this to Reflect ...

But is this really necessary?
This should be done by tool anyway,
depending on explicitly provided dependencies (mvn, ant, ...),
so it should be easy to include all of the needed jars to make it run under UCL.

[dmlloyd]

"alesj" wrote:

So - we are taking a small project, adding dependencies and making it more complex?

Why is adding dependecies making it more complex?
It's exactly the opposite - less code to maintain.

Wrong. Because you end up contorting your code to match the use-case of the dependency, rather than the use case you're trying to achieve, causing ineffective code to become further entrenched, and making it even harder to fix later. VFS being one great example. Many projects relied on VFS because it is written for a particular use case. However, it just happened to be completely wrong. Now fixing it is a huge undertaking.

"alesj" wrote:

Adding dependencies is a bad idea. Projects should as a rule have the minimum number of runtime dependencies possible. Having lots of dependencies just makes it harder to refactor later. If you have a large web of dependencies between a set of projects, that's a big indicator that they ought to be one big project anyway, or at least split up differently.

Again nonsense.
Refactoring is extremely easy if you have an external thing that does things for you.

Refactoring code is easy. Refactoring semantics is really difficult (see VFS again).

"alesj" wrote:
In most cases those ext libs have stand the test of time,
so they are much better then the crap you quickly code yourself.
And the nicest things about all the pieces in my version of Papaki is that we control them all,
meaning it's easy to create any missing feature or fix/extend an existing one.

Standing the test of time is no test at all! What kind of argument is that. Just because someone didn't rewrite something for 4 years doesn't mean it's not crap. And there are many coders here whose "quickly coded crap" I would take over any multi-year-old library in a heartbeat. We're here to solve problems, not worship old ineffective solutions.

[dmlloyd]

"alesj" wrote:

So we have to index all of them (unless there is some hint from the user that scanning is not necessary for a class or package or whatever)

Yes.
The restriction mechanism should be plugable,
and the one's I can think of right now are:
* OSGi manifest export header
* jboss-classloading.xml capabilities

Far simpler than designing a plugin mechanism for restrictions is to simplify the API so that you just give it the input streams of classes you want to index. Simplfying the API is usually preferable to adding an abstraction.

"alesj" wrote:

and remember (sans-classloading) where annotations are located, and the relationships between classes, so that we can find all classes and subclasses with certain annotations without actually loading any classes.

This sounds interesting, but what's the easiest way to achieve this?
I guess Javassist knows how to handle this.
How hard would it be to add this to Reflect ...

But is this really necessary?
This should be done by tool anyway,
depending on explicitly provided dependencies (mvn, ant, ...),
so it should be easy to include all of the needed jars to make it run under UCL.

The specifications for many of our components (like EJB 3.1) tell us that we can't rely on this, or at least that's my understanding from the developer meeting when we decided we had to do it this way.

[alesj]

Wrong. Because you end up contorting your code to match the use-case of the dependency, rather than the use case you're trying to achieve, causing ineffective code to become further entrenched, and making it even harder to fix later.

A?
What's the point of frameworks then?

In my case all of the used libs were a perfect fit.
Dunno what kind of crap you use. ;-)

Many projects relied on VFS because it is written for a particular use case.

What particular use case?

Current VFS does exactly what it was meant for or what you expect from vfs.
The API is simple and usable, features are pretty rich.

Where we might argue about impl details,
the overall goals were achieved:
* single point of controlling resources lookup
** handling file locking
** file vs. jar vs. dir abstraction
** visitor pattern
* extension points
** assembled
** in-memory
* ...

But well, now that I see your take on ext libs,
I'm no longer wondering why VFS3, vs. trying to fix existing impl.

However, it just happened to be completely wrong. Now fixing it is a huge undertaking.

Wrong in what way?
Going back to AS4 behind an VFS api (aka VFS3) doesn't make it right. ;-)

Refactoring code is easy. Refactoring semantics is really difficult (see VFS again).

Specially when you rename packages for no reason, right? ;-)

Standing the test of time is no test at all! What kind of argument is that. Just because someone didn't rewrite something for 4 years doesn't mean it's not crap. And there are many coders here whose "quickly coded crap" I would take over any multi-year-old library in a heartbeat.

Again, what kind of crap are you using?

I'm not saying you're completely wrong.
But in this case - as again my use-case proves it -
I would never take that re-invented "reflection abstraction + resource lookup abstraction" over existing impls.

We're here to solve problems, not worship old ineffective solutions.

You don't sound convincing, specially with all the negative comments about ext code.
Looks like you're more into re-writing things than solving real problems.
The world is not perfect, the code is not perfect,
but at least the code is open sourced, hence you can fix it. ;-)

[alesj]

Far simpler than designing a plugin mechanism for restrictions is to simplify the API so that you just give it the input streams of classes you want to index. Simplfying the API is usually preferable to adding an abstraction.

A? What kind of nonsense is this?

And how do you plan on providing this streams?
Via abstraction!!!!

Or how else are you gonna know which one's should or shouldn't be scanned?
Should I list them by hand, all of the 1M classes?
Or what about if tomorrow Sun speeds up its modules impl,
and we all start doing module-info.java?

The specifications for many of our components (like EJB 3.1) tell us that we can't rely on this, or at least that's my understanding from the developer meeting when we decided we had to do it this way.

Then I would say this is their problem.
Who ever doesn't have automated build, has either just started programming or is moro....
The first one's don't need this feature yet, the 2nd one's don't deserve it. ;-)

[dmlloyd]

"alesj" wrote:

Wrong. Because you end up contorting your code to match the use-case of the dependency, rather than the use case you're trying to achieve, causing ineffective code to become further entrenched, and making it even harder to fix later.

A?
What's the point of frameworks then?

They have no point, in and of themselves, unless they solve a specific problem without introducing more problems. If the framework doesn't have a clearly defined contract, it is a problem. If the framework's contract cannot possibly be met (aka VFS2), it's a defect waiting to happen. If a framework doesn't have a strictly defined and limited scope it will eventually overlap and conflict with other frameworks.

"alesj" wrote:

But well, now that I see your take on ext libs,
I'm no longer wondering why VFS3, vs. trying to fix existing impl.

Ales, the implementation can't be fixed because it's the API that is broken. That's why the new solution is so much smaller. It has a clearly defined API, with contracts that can be met by the implementation. Yes there are implementation fixes as well, but it's fixing the API (and thus usage patterns) which gives us the performance increase and size decrease without compromising functionality.

"alesj" wrote:

Wrong in what way?
Going back to AS4 behind an VFS api (aka VFS3) doesn't make it right. ;-)

I don't see what you're getting at here.

"alesj" wrote:

Refactoring code is easy. Refactoring semantics is really difficult (see VFS again).

Specially when you rename packages for no reason, right? ;-)

Huh? Are you referring to VFS? Renaming the package is just a one-key code refactor, it has nothing to do with semantics.

"alesj" wrote:

I'm not saying you're completely wrong.
But in this case - as again my use-case proves it -
I would never take that re-invented "reflection abstraction + resource lookup abstraction" over existing impls.

What if there was a substantial speed increase and memory footprint decrease associated with doing so? That's the question I think we need to explore here.

"alesj" wrote:

We're here to solve problems, not worship old ineffective solutions.

You don't sound convincing, specially with all the negative comments about ext code.
Looks like you're more into re-writing things than solving real problems.
The world is not perfect, the code is not perfect,
but at least the code is open sourced, hence you can fix it. ;-)

And that's what I'm doing.

[alesj]

What if there was a substantial speed increase and memory footprint decrease associated with doing so? That's the question I think we need to explore here.

No.

As all the complex/slow processing should be done by the tool, pre-runtime.
Deserializing + reattaching annotation repository is O(1).

If you need this info runtime, I would bet there is little gain
whether you code it against ext libs or hack it yourself.

And I much more like to use ext libs, where I can complain if something doesn't work,
instead of everyone bothering me. ;-)
(welcome to VFS bothering .... )

And that's what I'm doing.

Sure, but that doesn't mean you should rewrite everything that you don't like. ;-)
btw: how is Remoting3 coming along ... :-)

[dmlloyd]

"alesj" wrote:

Far simpler than designing a plugin mechanism for restrictions is to simplify the API so that you just give it the input streams of classes you want to index. Simplfying the API is usually preferable to adding an abstraction.

A? What kind of nonsense is this?

And how do you plan on providing this streams?
Via abstraction!!!!

No, I'm saying rather than having the scanner pull in the classes, the API consumer pushes the classes into the scanner. This way you don't have to worry about missing use cases when you design the abstraction. This way, Papaki is less limited by usage patterns.

Case in point, say Papaki depends on VFS and uses that project's visitor system to scan classes. Now later on we want to scan classes that are not found on VFS (maybe they're being downloaded). In this case, tying to VFS was an arbitrary restriction that we pay for later since now all classes have to be wedged in there whether it makes sense or not. On the other hand, if we push classes in, then both use cases are easily accommodated.

"alesj" wrote:
[Or how else are you gonna know which one's should or shouldn't be scanned?
Should I list them by hand, all of the 1M classes?
Or what about if tomorrow Sun speeds up its modules impl,
and we all start doing module-info.java?

This logic would be something that belongs in the module-loading layer (which is currently spread between deployers and classloading as far as I can tell), not in the scanning layer. The scanning layer does not have semantic awareness of what should or should not be scanned, so there's no reason to push this functionality into it. Let it do one thing well.

"alesj" wrote:

The specifications for many of our components (like EJB 3.1) tell us that we can't rely on this, or at least that's my understanding from the developer meeting when we decided we had to do it this way.

Then I would say this is their problem.
Who ever doesn't have automated build, has either just started programming or is moro....
The first one's don't need this feature yet, the 2nd one's don't deserve it. ;-)

That's a valid opinion, but it doesn't help us meet the requirements that we have spelled out for us, which include (but are not limited to) performance improvement and complete support for these specs.

[dmlloyd]

"alesj" wrote:

What if there was a substantial speed increase and memory footprint decrease associated with doing so? That's the question I think we need to explore here.

No.

As all the complex/slow processing should be done by the tool, pre-runtime.
Deserializing + reattaching annotation repository is O(1).

Again that's your opinion, but spec compliance isn't optional and this is what we all agreed to accomplish. If you want to reopen the question, go for it.

"alesj" wrote:
Sure, but that doesn't mean you should rewrite everything that you don't like. ;-)
btw: how is Remoting3 coming along ... :-)

What else does it mean? I'm not rewriting things I don't like, I'm rewriting things that are not doing the job they're supposed to be doing, so that they do. I'm not arbitrarily picking on projects. I'm measurably improving things. Every single thing I've rewritten is measurably better than it was before. Logging is 10%+ faster and now supports JDK logging as well as Log4j, and is fully integrated with MC. Threads are now fully POJO-ized as well, and abstracted behind a clean and simple API which makes it very easy to plug in alternate implementations - without adding lots of extra abstraction. VFS is next because it's the next-most-significant blip on the radar in terms of performance problems which is something we are all supposed to be working towards fixing.

Remoting is coming along quite well. The Marshalling subproject (another evil rewrite!) is now being used by Infinispan (another evil rewrite!) with some great performance numbers. The XNIO subproject is also successful and will be the core of network configuration on AS 6. Were it not for the VFS work I'm doing now, I'd have 3.1 released already.

Not all rewrites are bad. It is not reasonable to maintain a deathgrip on existing code out of a sense of personal attachment. One must always be ready to throw away what they have if it's defective in design. Knowing this makes one rethink design in terms of avoiding lock-in. That's why my projects all have so few dependencies and impose few or no dependencies on their consumers.

My experience shows that it's more cost-effective in the long run to get rid of problems early on, rather than continually applying small patches. I'm sorry if you disagree but I think my approach is fairly well proven at this point, and I can and have been able to back that up with numbers rather than just a stream of increasing problems and complaints, which is what I think the logical culmination of the "use existing codebases at all costs" approach.

[alesj]

Again that's your opinion, but spec compliance isn't optional and this is what we all agreed to accomplish. If you want to reopen the question, go for it.

Spec compliance?

[dimitris]

Rather than keep going with the dog-fight, why not putting down the exact requirements/usecase for this module (papaki), or is this already discussed elsewhere?

I admit I don't know the details, but my personal preference would be for something really simple and really fast with minimal dependencies. Then users of the api/service do what they want with the information.

And I wouldn't count too much on pre-propressing steps, that would be a nice option, but IMHO it has to be fast without pre-indexing.

Now if there is overlap with existing functionality or there are things we can re-use, that's fine, but again the first thing is to be clear on what we want to achieve, rather than trying to fit the usecase into existing APIs that maybe not designed for that.

[emuckenhuber]

Rather than keep going with the dog-fight, why not putting down the exact requirements/usecase for this module (papaki), or is this already discussed elsewhere?

I haven't found a discussion on the goals of papaki - so i'll just add a few thoughts. Not all related to papaki, but just describing some of the problems i see with the annotation scanning. Maybe that helps that we can define clearer requirements what papaki should provide and what has to be done elsewhere.

* unified API, independent of the scanning strategy (precompiled or runtime) - as when writing deployers integration code it should not matter where this information comes from. The precompiled index has to be optional, since the spec requires runtime scanning. So either a precompiled index exists or we have to generate it during deployment time.
So that we really have a reusable index, which should be used by all deployers and we only do scanning once (or use the index).

* jboss-metadata - for example jboss-metadata is just asking the AE to get a list of classes which have a given top-level annotation (e.g. @Stateful). Then loading the class and do a separate annotation processing with java reflect. This is also because AE does not have enough information about inheritance, which is needed for all the EE annotation processing.
Beside jboss-metadata there are not many users of AnnotationEnvironment, but we really should avoid having things like getClassloader().loadClass(metaData.getClassName()).getAnnotation(X) if possible.

* jboss-mdr integration - this index should be reused in the MDR as well, as mentioned earlier in this thread.

* inclusion/exclusion of deployments, .jars, classes - we need a way to exclude stuff from annotation scanning, as we are mostly just doing too much scanning. Basically we already have jboss-scanning.xml - which makes sense for our own deployments. Still we need a programmatic way of excluding stuff like:
- if the deployment is not an EE5/EE6 deployment we can just skip any annotation scanning,
- if isMetaDataComplete()
- afaik the new servlet spec requires to exclude .jars (web-fragments), where we should be able to skip scanning as well.

[alesj]

"emuckenhuber" wrote:

* jboss-metadata - for example jboss-metadata is just asking the AE to get a list of classes which have a given top-level annotation (e.g. @Stateful). Then loading the class and do a separate annotation processing with java reflect. This is also because AE does not have enough information about inheritance, which is needed for all the EE annotation processing.

But then you should not use java reflect, but JBoss Reflect. ;-)
Why?
Because this info is already present if we use McAnn, as McAnn already uses JBoss Reflect, and Reflect caches this info.

And this also shows that things go hand in hand, and that this inheritance check should not be part of annotation scanning.

Not to mention another very good example why re-using existing libs is good. ;-)

"emuckenhuber" wrote:

Beside jboss-metadata there are not many users of AnnotationEnvironment, but we really should avoid having things like getClassloader().loadClass(metaData.getClassName()).getAnnotation(X) if possible.

Sure, leave it to JBoss Reflect. It's already there.
And with Reflect running on top of Javassist it's gonna be lazy as possible.

"emuckenhuber" wrote:

* jboss-mdr integration - this index should be reused in the MDR as well, as mentioned earlier in this thread.

Yes.
But this is just a matter of properly populating MDR's MetaDataLoaders.
e.g. sort of an MetaDataLoaderToAEAdapter

"emuckenhuber" wrote:

* inclusion/exclusion of deployments, .jars, classes - we need a way to exclude stuff from annotation scanning, as we are mostly just doing too much scanning. Basically we already have jboss-scanning.xml - which makes sense for our own deployments. Still we need a programmatic way of excluding stuff like:
- if the deployment is not an EE5/EE6 deployment we can just skip any annotation scanning,
- if isMetaDataComplete()
- afaik the new servlet spec requires to exclude .jars (web-fragments), where we should be able to skip scanning as well.

This is all already there, in Deployers.
Either as part of ScanningMetaData or AnnotationScanner's DeploymentUnit filters.
* e.g. http://anonsvn.jboss.org/repos/jbossas/branches/Branch_5_x/server/src/main/org/jboss/deployment/JBossMetaDataDeploymentUnitFilter.java

Web deployers should just properly push required info about fragments to this mechanisms.
e.g. using ScanningMetaDataBuilder (which you promised to write ;-)