-
15. Re: Looking for a real life ESB transformation example
OK, Tom. Here's the challenge: The data (the most simple example I could devise in the environment where I typically work) looks like this:
ISA*00* *00* *ZZ*EXTERNAL_TP *ZZ*INTERNAL_TP *010806*1200*U*00401*000000003*0*T*:
GS*HS*EXTERNAL_AP*INTERNAL_AP*20010101*120000*001*X*004010X092
ST*270*0001
BHT*0022*13*10001234*19990501*1319
HL*1**20*1
NM1*PR*2*ABC COMPANY*****PI*842610001
HL*2*1*21*1
NM1*1P*1*JONES*MARCUS****SV*0202034
HL*3*2*22*0
TRN*1*93175-012547*9877281234
NM1*IL*1*SMITH*ROBERT*B***MI*11122333301
REF*1L*599119
DMG*D8*19430519*M
DTP*472*D8*19990501
EQ*98**FAM
SE*14*0001
GE*1*001
IEA*1*000000003
Pretty, huh? This is ANSI X12N, the required format for US healthcare these days (stay tuned, HL7 is being knocked about as the "new" standard upcoming -- it, at least, is XML). X12N format is based on "segments" (each line starting with a two- to three-character code). Each "element" of data is, at least in this example, separated by asterisks. For readability, I've left the segment terminator as a line feed.
This format is excruciatingly hierarchical. That is, a single transmission might contain multiple "interchanges" (that's the bit between the lines starting with "ISA" and ending with "IEA"). Each interchange may have one or more "groups" (bracketed by "GS" and "GE"). The target, if you will is the "transaction", located between "ST" and "SE".
I'm going to take a crack at this myself, but my suggestion would be to tackle this in stages. First, check the transmission for interchanges, validating the ISA and IEA segments according to rules (to be posted subsequently). Then, invoke one secondary action for each interchange found, to check for groups. And so on.
The output for the exercise is XML (the publisher who came up with the X12 has a version that I'm not too keen on) -- I'll be back shortly with my version, along with more descriptions for the input data elements. For now, I would recommend transforming this mess into an XML like<batch><ISA><ISA01>00</ISA01>... etc.</ISA></batch>
where ISA01 is the first element found in the ISA segment, etc. Where an empty value is found (see the HL segment), the example would be <HL02></HL02>.
I will be posting subsequent info via URLs so the forum can be spared the volume... :-)
Please advise if this makes little sense.
Dave De -
16. Re: Looking for a real life ESB transformation example
Nice one Dave !!!! You're not joking, it is ugly!!
Anyway, what doesn't kill you will make you stronger etc, so I'm looking forward to trying this out. To be honest, I was sorta thinking more in terms of SOAP to SOAP style examples, but this is great because I need to understand all the scenarios - especially the ugly ones ;-)Please advise if this makes little sense.
I've only had one read through it and it makes *some* sense :-) After a dozen or so reads I'm sure a clearer pisture will start to form.
First thought that came into my head after reading this was... would it be possible to write an X12N to SAX event parser? This would be a parser that could generate bog standard SAX events from an X12N stream, thereby allowing you to construct an XML DOM directly from the X12N stream - or alternatively, serializing the SAX event stream straight out to HL7. So, this would make the X12N stream *appear* as though it was an XML stream. Can't think why this wouldn't be possible, if X12N is hierarchical/predictable!!
On your XML'ified version of the X12N stream:<batch><ISA><ISA01>00</ISA01>... etc.</ISA></batch>
Why did you choose to represent each of the interchanges with new element tags i.e. ISA01, ISA02 etc?? Could it not be something like:<x12n><isa whatever="00">....</isa><isa whatever="13">....</isa> ... etc... </x12n>
... and use the position of each element to infer the 01 (ala ISA01) etc. Is the position of an interchange, within a batch, actually all that important?
I'll take a closer look when I get a chance. Sat evening now and I'm away for the day tomorrow - our county team is playing a big game against Dublin :-)
Thanks again Dave!! You certainly have me thinking now :-) -
17. Re: Looking for a real life ESB transformation example
OK, so I've put something together on this.
So the solution I implemented revolves around a standard SAX Parser implementation called X12nToSaxEventParser. As its name is supposed to suggest, it parses an X12N stream and generates a set of SAX Events which it fires at a standard SAX ContentHandler implementation. It has 2 support class:
1. X12nStreamReader: Wraps the stream and makes the X12N segments easier to access.
2. X12nModel: Contains definitions to help the parser convert the X12N stream to a stream of SAX events.
Here's a sample of its output after parsing the sample X12N stream in Dave's above post - actually it's 2 blocks of the above sample X12N, hence the 2 blocks inside the root <x12n> element.
So, this parser could be used by any XML transformation tools that support SAX based processing and transformation. I had to make a very small mod to Smooks in order to make it easier to specify specific SAX parsers for specific message types - available in the v 0.7.2-SNAPSHOT build from the downloads.
See the JUnit test code. The x12n-config.cdrl config file illustrates how to configure the X12nToSaxEventParser to parse messages originating from X12N based message producers ("x12n-requester" - "x12n-producer" might have been a better profile name). See X12nToSaxEventParserTest for the units test code for executing a parse. Excuse the fact that this test has no assertions - it just runs the parse :-) This test doesn't perform any actual XML based transformations on the XMLified X12N stream. This would probably be required in a real life situation (different transformations for different consumers) and is easy enough using the standard Smooks mechanisms (see tutorials) - once we have it in XML, we can do whatever we want!! -
18. Re: Looking for a real life ESB transformation example
Absolutely brilliant, Tom! Your solution provides an excellent framework from which to build. I will pursue moving toward delivering the output I've constructed here: http://www.ehealthconnect.net/ehc/test270.xml
For further thought/discussion, I should make mention of some ancillary issues (not at all requisite for pressing forward with straight transformation):
1. Validation (use of X12n implementation guides to determine correctness of both structure and content of entire EDI message). I can pretty much take ownership of this tedious issue, unless someone has extensive experience/expertise in this.
2. Looping. I will provide later further explanation of this issue. Suffice it to say at this point that there is a concept I refer to as "behavior of data", that has certain constructs repeating some number of times (see issue #1).
3. Reusable Objects. Quite a number of elements found within EDI are actually context-appropriate reuse of "standard" structures (name, address, etc.) This issue will become apparent when I publish the schema for the 270 output XML.
Thanks again, Tom.
Dave De -
19. Re: Looking for a real life ESB transformation example
Dave, do you need this for something real??
-
20. Re: Looking for a real life ESB transformation example
Dave, do you need this for something real??
Boy, Tom, the short answer is "yes". The long answer is a bit more involved. My objective is to deliver an eCommerce infrastructure based on JBESB. In truth, this concept is applicable cross-industry.
My main focus/passion is US healthcare, certainly. My experience, in fact, has been in multiple industry environments, including international.
The infrastructure of which I speak would apply for X12 (cross-industry, including healthcare, retail, manufacturing, logistics, etc.), EDIFACT, SWIFT, on and on.
Now, I can get off my "soap box'. If your question relates to timescales/existing client, the short answer is: no existing client--yet; but I would like to see this "generic" infrastructure in place by Q1 2007. In the interim, I seek a client to partner for early adoption/alpha and beta testing, etc.
Maybe my reach exceeds my grasp, but I see enormous potential for this.
Dave -
21. Re: Looking for a real life ESB transformation example
Dave,
Would you be interested in a ebMS 3.0 B2B connector? I'm working on one that should be fairly easy to integrate with JBESB. I hope to have it ready by Q4 2006. To be as backwardscompatible as possible I hope to have a ebMS2.0 version available in Q1 2007.
I see lots of opportunities for this in Europe as well. -
22. Re: Looking for a real life ESB transformation example
Ronald,
I would be most interested. Having ebMS capability along with EDI would offer a pathway to the future of eCommerce.
Your involvement in jBPM intrigues me, as well. I am of the opinion that BPM, along with data transformation, offers a powerful platform for extending a core capability to integrate an entity (public or private) within an ESB framework.
Is there a website where I can learn more about your connector (from a functional description perspective)?
I will be posting a functional description for the capability I've been describing. Most likely, I should create a new topic for this, so that those with similar interest/capabilities can merge their expertise. Agreed?
... and Thank You,
Dave -
23. Re: Looking for a real life ESB transformation example
David,
(Since this is off-topic, which I as a jBPM forum moderator hate ;-), I'll for now respond one here. Maybe we should indeed start a new thread)
I'm currently developing this 'off-line', so there is no website at all. Regarding the functionality, i'm focussing on the following items, ordered by priority
1: Full envelop compliancy (user messages and signal messages)
2: Soap one-way MEP (one way push)
3: Errorhandling
4: Reliable messaging for this mep
5: One way pull
6: Reliable messaging for this mep
7: Request response MEP
8: Reliable messaging for this MEP
After this basic functionality, I'll look into storing partner profiles, agreements etc. So on this part I am open to any requirements (I prefere to use LDAP though on the technical level) Only after this, I'll pay attention to persistent security aspects (signing etc)
Reliability is the challenge, since there are two competing standards for it, both under OASIS (????? admired and hated by me cause of their flexibility. Speed ok, TWO protocols.... stupid) Maybe JBossWS will include one, I personally have a preference to use the SUN version, WS-Reliability) -
24. Re: Looking for a real life ESB transformation example
Being a member of both WS-R TC, I can say that OASIS WS-RM is pretty much dead. Despite having superior technology to WS-RX, the realities are that everyone (including Sun) is now looking to support WS-RX. JBossWS will eventually support WS-RX, though that's further down the roadmap.
-
25. Re: Looking for a real life ESB transformation example
David,
This is a very real world example for me as well. We are currently working on createing a new ESB that will accept, among other things, inbound 834 data for Membership processing.
Very nice to see our thoughts and ideas validated by someone else. Also nice to see that the JBossESB will be something that we can use in the near future.
Stacy -
26. Re: Looking for a real life ESB transformation example
Stacy,
Understand very well the specific issues with membership enrollment/maintenance. You will find, as JBESB deploys through Beta into GA, that this facility offers a lot of flexibility in what your service can provide, including extending your internal ESB into areas such as premium processing and eligibility verification.
In my view, you and your team are definitely on the right track.
Best of luck,
David