Pure-CMP & FAQ Module revamp

[hxp]

Julien --

After reading the code for Jae's News module and reading his comments, it's becoming clear that you folks are pretty open to the pure-CMP approach that I've been advocating for the past few months -- and Jae is making good progress in the right direction. Nice!

I thought I would have to do some arm-twisting to get everyone to see the value of letting-go of the rest of the php/sql legacy, and moving to a pure-CMP backend architecture ..... and I had been waiting till after 1.1 to do that. So although I already have working code for 2 modules, before contributing it I was working on writing several documents/wikis to explain the why and how of pure-CMP.

I don't yet have the dump/load utility written, which I was planning to use to stuff the entities with initial data... which would be required to do a full replacement for the setup.xml/ddl/sql embedded intialization data. But I see that Jae's News module doesnt even attempt to stuff dummy data in at deploy time... so now I realize that you might be interested in what I already have running.

I have the backends of 2 modules -- FAQ and Quotes -- revamped to a pure-CMP approach. Builds and deploys are clean in my tests on HSQL, and operation is essentially the same as before (with no dummy data preloaded, of course). (Early tests on MySQL show a bug which can be fixed either of several ways; we can discuss later on the thread.) So I've successfully upgraded the Nukes FAQ module to pure-CMP so that it doesnt use sql/ddl anymore at all. I've recently gotten xdoclet, unknown-pk, and CMR to coexist without problems.

My original purpose with revamping FAQ & Quotes was to prove the concept so that pure-CMP can be used for all of Nukes2.0... provide some templates for how to do it... and inspire the team to move in that direction.

But now it looks like my work could also be useful for the 1.1 release. I can diff and submit patches (not my preference, with 5+ files that will need to be revved repeatedly), or submit whole modules, or with CVS RW, commit the changes to either HEAD or a pure R&D branch. If you want me to go the CVS route, I'm willing to become the maintainer of the FAQ module, and begin reworking the rest of it (some for 1.1; more for 1.2). My first priority would be to test & solidify the pure-CMP backend, and converge my approach with the approach that Jae is using in News. I feel good about that for 1.1.

I will be travelling from mid-May to early-July (and unavailable for coding), so I don't want to raise expectations too much.

From July onwards, I would foresee revamping & enhancing the entire FAQ module, consistent with JSR-168 and whatever frontend architecture is optimal for Nukes at that point.

I'd also be willing to coordinate some volunteers to rewrite many or all of the module EJBs to conform to the pure-CMP approach, and to be the conduit for reviewing and committing the changes to cvs.

The shift to pure-CMP means big simplifications in the whole "installer" area; and a new xml-based dump/load utility will allow far better migration, initialization, and bulk import/export. (I'll post wiki pages to describe what I have in mind in these areas... If some of this doesn't make sense yet, it's because I've got a pretty extensive architecture thought out for the Nukes backend, and really need to write a whitepaper or collection of wikis for the team to review and discuss.)

But maybe more important for now, I'm ready to switch into code-contribution mode.

--- Howard



FYI, this was the posting that I've changed my mind about:

"hxp" wrote:
Dennis --

"thepriz" wrote:
Someone was working on the faq module how is that comming?

Yeh, I said that I was happy that you picked Polls because I was making some changes to FAQ... but that's not because I was doing anything intended for 1.1 release.

If you get done with Polls and have time to do FAQ before 1.1, please do it.

My changes have zero to do with the front-end stuff (stripping out inline html, moving to templates, etc, or the jsr-168 impl) you guys are currently working on. My changes are 100% concerned with moving the backend to pure-CMP, with automatic pk generation, automatic table create/alter/delete, etc. After I successfully did this with Quotes (which has no CMR), I picked FAQ because it has the amount of CMR that I want to tackle next, and it seemed to be a "backwater module" that noone was paying any attention to (when I dont have CVS RW yet, I try to work where noone else is making frequent changes). But I've run into a tricky CMR-unknownpk-xdoclet interaction that I haven't had time to solve or to ask for help on; soon I'll write up this issue so the CMP/CMR/XDoclet gurus can tell me whether this is a bug in 3.2.3 ...or in xdoclet ...or I'm doing something stupid 8-).

So my FAQ changes are really just part of some early (proof-of-concept) legwork on what will become a pretty massive change in the whole nukes backend (not just because it will elimate all the database-specific setup.xml/DDL/SQL garbage, radically simplify installation, and ease schema evolution, ++)........ but it will be quite awhile before it's time to commit it into HEAD. So, if someone makes changes to FAQ for 1.1, or before I get to commit my stuff, that's fine, go for it -- I'll merge my work in later.

Anyway, I dont want to get you guys distracted with this stuff -- until after you've got 1.1 out ;)

-- Howard

[hxp]

Hey everyone,

I'm quite unfamiliar with JBoss, and I have a quick question. Is is possible to use symlinks in the deploy directory under linux? I would like users to be able to keep all their packages under their home directories and simply create symlinks from the deploy directory.

When I try to create a symlink to a war file (test.war -> /home/user/test.war), the following appears in the log file:

DEBUG [org.jboss.deployment.scanner.URLDeploymentScanner] Failed to deploy: ... file:/usr/jboss/jboss-4.0.0/server/default/deploy/test.war could not be opened, does it exist?

Is it a permissions problem, or simply not possible? Any help on this is greatly appreciated.

Dave.

[jae77]

at this point, i definately feel that pure cmp is the way to go (i stared at those ddls for quite some time debating whether or not i wanted to re-write them), but i don't think it's necessary for the 1.1 release.

i do think that if ppl are reworking a module for 1.1 and are comfortable w/ xdoclet, etc then by all means include it, but otherwise i think this should be set as a goal for the 2.0 release.

i don't have oracle, mysql, or postgres running at home, so while i haven't had any issues creating tables in hsqldb, i don't know if someone else may have an issue. (i know for hsqldb, i have to set the pk-constraint to false, but this may not be necessary for the other dbs).

btw: preloading data could probably be done w/ .bsh scripts that could be lumped into the deployed ear - they'd just need a check to see if the data is already there so it doesn't try to re-create it.

[hxp]

"hxp" wrote:


Jae --

"jae77" wrote:
at this point, i definately feel that pure cmp is the way to go

Great news, Jae -- thanks for your vote!

"jae77" wrote:
(i stared at those ddls for quite some time debating whether or not i wanted to re-write them),

Yeah, that was what held me back for a while. For a reminder, see #1-2-3 at
http://www.jboss.org/index.html?module=bb&op=viewtopic&t=46324.... The breakthru came when I realized that #2 & #3 were unnecessary -- at least for now, because 90%+ of the ddl code was loading dummy/demo data that could be dispensed with. The few entities which have to be preloaded are for the initial admin/admin user and associated group; and I came up with a cool idea for securely initializing these in a pure-CMP environment.

This realization also led to seeing that any/all of the non-core modules could be migrated to pure-CMP without waiting for the core to be migrated. So the problem reduced to just #1... and that on a module-by-module basis :-) ! Which made it feasible for me to do #1 by myself, and work out initial templates for other people to use to convert more modules.

So realizing we indeed can get rid of the ddl leads to all sorts of other good insights :-)

"jae77" wrote:
but i don't think it's necessary for the 1.1 release.

Agreed -- I hope nobody interprets this thread as saying the shift to pure-CMP must happen for 1.1! Instead, the intent is to show that a few non-core modules can go to pure-CMP for 1.1 . At this point, we are close to having 3 pure-CMP modules ready for 1.1 --- your News, and my Quotes and FAQ. Maybe Dennis will get inspired and tune Polls and Downloads in this direction (I dont think he has any init data; but probably still uses ddl for table creation; I haven't looked to see.)

"jae77" wrote:
i do think that if ppl are reworking a module for 1.1 and are comfortable w/ xdoclet, etc then by all means include it, but otherwise i think this should be set as a goal for the 2.0 release.

Yes, a great goal would be 100% pure-CMP (including the core) by 2.0; and a healthy number of non-core modules already moved to pure-CMP by 1.2.

"jae77" wrote:
i don't have oracle, mysql, or postgres running at home, so while i haven't had any issues creating tables in hsqldb, i don't know if someone else may have an issue. (i know for hsqldb, i have to set the pk-constraint to false, but this may not be necessary for the other dbs).

I have hsql on my home dev nukes inst, but have mysql on my rh9 linux box colo'd at the datacenter. Early tests show a bug with mysql and the nukes entity command; you'll see something more about this from me by the time you get back from Las Vegas.

"jae77" wrote:
btw: preloading data could probably be done w/ .bsh scripts that could be lumped into the deployed ear - they'd just need a check to see if the data is already there so it doesn't try to re-create it.

Ahhh.... we can do much better than that. I want to create a Lifecycle ("Meta-Admin") Module that would automte & manage all sorts of data initialization, bulk loading, dump&restore, schema-evolution + data-migration, fancy backup, etc, etc. Much of the good work that Noel and Sherman did for the installer would be absorbed into this much more general Lifecycle Module, because we're getting closer and closer to the realization that we don't need an "installer" at all.

One of the reasons that i made the
http://jboss.org/wiki/Wiki.jsp?page=InstallUninstallUpgrade
wiki is to pinpoint exactly how complex it really is to manage the lifecycle of a basic Nukes installation. One of my goals with the pure-CMP push and the Lifecycle Module will be to cut the number of steps in InstallUninstallUpgrade by 50% --- and then 50% again, until we have an ultra-simple way of getting-going and keeping up-to-date. Ultimately, just a single Ant task -- and everything other than that would be done within the secure single-sign-on Nukes environment.

(Have fun in Las Vegas -- just don't go broke, 'cause we need you to be able to afford to keep working on Nukes ;-))

-- Howard

[jae77]

"hxp" wrote:

(Have fun in Las Vegas -- just don't go broke, 'cause we need you to be able to afford to keep working on Nukes ;-))

hehe - shouldn't you be hoping that i win big so i can quit the day job and work on nukes full time? :)

i'm curious to know what the issue w/ the entity command is and how it relates to creating the tables themselves. i know hsqldb has an issue where you can't enable the pk-constraint when creating the tables. are you sure you're not missing something w/ the xdoclet/ant "config/setup?" i ask b/c we are generating successful deployments for other components and mysql.

[hxp]

"hxp" wrote:

"jae77" wrote:

"hxp" wrote:

(Have fun in Las Vegas -- just don't go broke, 'cause we need you to be able to afford to keep working on Nukes ;-))

hehe - shouldn't you be hoping that i win big so i can quit the day job and work on nukes full time? :)

Well... now that you put it that way... here's a hedge bet for you: the stakes are 2 large bowls of Pho, to be settled next time we're both within 25 miles of Ellicott City (House of Asia, 29@40); if you have a net loss in Las Vegas, I'll buy; otherwise you'll buy. With those stakes, you can be sure I'll be pulling for you to win really big this weekend. :-) You on?

[jae77]

hehe - sure - are you in the maryland area? i live in columbia (so i'm always w/in 25 miles of EC), but i'm headed back to nyc in the fall.

[hxp]

Jae --

"jae77" wrote:

are you sure you're not missing something w/ the xdoclet/ant "config/setup?"

No, I'm not sure. Please say more about this when you get a chance.

"jae77" wrote:

i ask b/c we are generating successful deployments for other components and mysql.

Careful, careful... those were not pure-CMP deployments... weren't relying on create-table....

I suspect we'll find something wrong with the interaction of create-table and nukes and mysql....

"jae77" wrote:
i'm curious to know what the issue w/ the entity command is and how it relates to creating the tables themselves.

Just a suspicion at this point; haven't had time to investigate to determine root cause.... will post more ~Sunday/Monday.

I have more to say on this that's relevant to News Module as well --- so I'm forking this thread, and continuing implem/debugging discussion at:

MySQL & Pure-CMP --- issues w News & FAQ modules

http://jboss.org/index.html?module=bb&op=viewtopic&t=49132

Bye for a few days.

-- Howard

[hxp]

Jae --

"jae77" wrote:
i am going to update mapping propery files (the ones used by the local.properties file) to have the pk-constraint settting automatically defined. this should handle that issue
...
i know i've accidently forgotten to change the mapping in the local.properties file and had these same types of errors.

OK, you're going to be setting pk-constraint default on a per-DB basis, sounds good. But what about properly switching jbosscmp-jdbc.xml settings?

From MySQL & Pure-CMP --- issues w News & FAQ modules
http://www.jboss.org/index.html?module=bb&op=posting&mode=quote&p=3833509

"sesques" wrote:

...You don't have to rely on database specific tags for the mapping, all the mappings are described in the standardjbosscmp-jdbc.xml file. For MySQL, the mapping for Strings is VARCHAR(250).

So, you just have to specify the datasource mapping type in jbosscmp-jdbc like LCB says ...

"LCB" wrote:

<jbosscmp-jdbc>...
 <defaults>
 <datasource>java:NukesDS</datasource>
 <datasource-mapping>mySQL</datasource-mapping>
 </defaults>

"hxp" wrote:
Hi Pascal, Hi LCB --

Thanks for your suggestions.

Your standardjbosscmp-jdbc.xml / jbosscmp-jdbc.xml analysis has the ring of truth to it;
we need to check into it to determine for sure.

Your analysis also explains why this is not a general problem for JBoss,
but is a general problem for Nukes....
all Nukes modules share the faulty configuration file(s).

These faulty configuration files would have been manually administered in a typical JBoss app environment,
but instead they are automatically generated or fixed in CVS
and automatically included in the Nukes build process.

Because Nukes modules have never used pure-CMP prior to now (in particular,
using the create-table xdoclet tag to generate the xml to get the container to create the tables),
this problem has never shown up before now.

The next question is where to put the fix(es) into Nukes --
* in the nukes installer?
* in the ant build process...
the build.xml? the local.properties file? the mapping propery files?
Jae's post begins to go into this question;
looks like we're hot on the trail of a good solution.

What is the best way -- in the Nukes context -- to implement what Pascal Sesques and LCB are recommending?

Will the user be able to control everything cleanly from local.properties? Will the installer work properly, even when a different datasource (other than the one specified in local.properties) is selected?

Most importantly, will this work across all nukes modules, and with all JBoss-supported databases, transparently without further user configuration?

-- Howard

[hxp]

Jae --

"jae77" wrote:
ok - i'm back from vegas, a little bit poorer (howard, you owe me some noodles) and married (YAY!!!)

Happy to keep my word; the noodles are on me.

You hadn't mentioned that you were about to be a family man... so now, not only to hedge your gambling losses in Vegas, but also to give a wedding gift, and thank you for your making the Nukes world (or at least local.properties) safe for pure-CMP (which you are about to do in the next few days, right? :)) .... I increase my offer: I'll buy a bowl of Pho for you and for every family member you bring with you.

Now, calculating the odds. The longer I wait to come within 25 miles of Ellicott City, the more kids you'll have and the more they'll eat... and the more relatives you'll be able to bring who are willing to listen to an intense conversation of java-jboss-nukes internals... ;-)

So better not wait. I'm buying my air tickets this week, and I'll be buying you Pho in June.

--Howard

[jae77]

hehe - i didn't really expect to come back a family man. we had talked about it, but i didn't actually think it would happen.

have no fear though, i don't plan on having kids any time soon (if at all), so our outting should be relatively inexpensive :)

----
getting back to the nukes discussion, i take it from your other post that everything is working ok. let me know if anything else comes up.

[julien1]

jae congrats !!!

[hxp]

Jae --

"jae77" wrote:

getting back to the nukes discussion, i take it from your other post that everything is working ok. let me know if anything else comes up.

Yes, we're making good progress.

OTOH, since the overall goal is creating a totally-clean pure-CMP template (that works transparently across all JBoss-supported DBs) that we can then use as a basis to redo all the modules and to base new modules on... and also making any changes ("global adjustments") necessary in the core, installer, &/or build process necessary to support proper operation of those pure-CMP templates/modules.... we've still got a ways to go.

1) There are architectural questions still open, mentioned on my past posts on this thread (please review, esp bottom half of May 4, 2004 01:27 AM post). (How these questions are answered will relate to how you do the "global adjustment" you mentioned; what changes will be made to the installer; etc).

2) Once you put the "global adjustment" in (and Julien gives me RW), I can immediately commit my updated FAQ & Quotes modules, and have confidence that they will work equivlently on MySQL, HSQL, etc, and be included in the 1.1 release. Next, you and I can converge our entity ejb xdoclet approaches, and produce the pure-CMP template that others can work from. After my east coast trip, I will redo the FAQ front end, and make other improvements for 1.2 as mentioned in the first message on this thread.

--- Howard

[jae77]

thx julien!!

howard, i'm not sure i understand what you're asking. editting the local.properties file to set the correct database

# Nukes target database, for now : hsqldb, mysql, postgresql are the possible values
nukes.database=hsqldb

automatically handles all the mapping issues for you. once you run the build, the jbosscmp-jdbc.xml file is automatically created w/ the correct mappings inside.

i don't think there is any way to automatically have the installer modify the datasource for us unless we have a way to write the file back out to the deploy directory (or just create it on the fly).

everything should be able to be controlled cleanly through modification of the local.properties file. we could probably extend the build script to prompt a user for some input and use that to get the databse to use so ppl don't have to worry about editting that file.

let me know if i'm off track on what you're looking for.

[jae77]

ok - the changes are checked in. update cvs to get them. be aware that when you re-run the build, if you have made any changes to the local.properties file in the build directory, it will be overwritten w/ the default one in build/etc.

in order to use the settings, just change your xdoclet configuration inside the entity beans to this:

@jboss.persistence
 create-table = "${nukes.create-table}"
 remove-table = "${nukes.remove-table}"
 pk-constraint = "${nukes.pk-constraint}"

create = true, remove = false, and pk-constraint = true for all db except hsqldb.

on a side note, the i remove the oracle8.properties file and instead made it work like the rest of the db settings. if you want to use oracle8, just specify that inside the local.properties file.