File Listener Gateway Configuration

[colsona]

I am trying to set up a FileListenerGateway that creates an ESB message when a new file is received. I only want to propogate a file event and not send the contents of the file in the message as it may be huge. In debugging my issues I found that the FileGatewayListener looks to the actions defined in the configuration to use as the file composer. I have defined the actions but continue to get the following at start up:

09:28:23,427 WARN [AbstractFileGateway] No element found in cofiguration - Using default composer class : org.jboss.soa.esb.listeners.gateway.FileGatewayListener$PackageFileContents

My configuration is as follows:

<?xml version = "1.0" encoding = "UTF-8"?>
<jbossesb parameterReloadSecs="50" xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">


<fs-provider name="TestFile">
<fs-bus busid="test-file"/>
</fs-provider>


<service is-gateway="true" name="FileListeningService" description="Service that listens for files coming in" category="FileEvent" target-service-category="FileEventActions" target-service-name="FileActionEvent">

<fs-listener busidref="test-file" maxThreads="1" name="testFileListener" poll-frequency-seconds="10" >
<fs-message-filter input-suffix="txt" directory="C:\\FileTest" post-delete="false" post-directory="C:\\FileTest\\inProcess" error-directory="C:\\FileTest\\inError"/>
</fs-listener>





<service is-gateway="false" name="FileActionEvent" description="Service that processes receipt of files" category="FileEventActions" >

<fs-listener busidref="test-file" maxThreads="1" name="testFileListener" poll-frequency-seconds="10" >
<fs-message-filter input-suffix="txt" directory="C:\\FileTest" post-delete="false" post-directory="C:\\FileTest\\inProcess" error-directory="C:\\FileTest\\inError"/>
</fs-listener>








Thanks,
Asa

[colsona]

Sorry the configuration didn't go through completely:

<?xml version = "1.0" encoding = "UTF-8"?>
<jbossesb parameterReloadSecs="50" xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

 <providers>
 <fs-provider name="TestFile">
 <fs-bus busid="test-file"/>
 </fs-provider>
 </providers>
 <services>
 <service is-gateway="true" name="FileListeningService" description="Service that listens for files coming in" category="FileEvent" target-service-category="FileEventActions" target-service-name="FileActionEvent">
 <listeners>
 <fs-listener busidref="test-file" maxThreads="1" name="testFileListener" poll-frequency-seconds="10" >
 <fs-message-filter input-suffix="txt" directory="C:\\FileTest" post-delete="false" post-directory="C:\\FileTest\\inProcess" error-directory="C:\\FileTest\\inError"/>
 </fs-listener>
 </listeners>
 <actions>
 <action name="CreateFileMessage" class="com.spss.batch.esb.CreateFileMessage" process="receivedFile"/>
 </actions>
 </service>
 <service is-gateway="false" name="FileActionEvent" description="Service that processes receipt of files" category="FileEventActions" >
 <listeners>
 <fs-listener busidref="test-file" maxThreads="1" name="testFileListener" poll-frequency-seconds="10" >
 <fs-message-filter input-suffix="txt" directory="C:\\FileTest" post-delete="false" post-directory="C:\\FileTest\\inProcess" error-directory="C:\\FileTest\\inError"/>
 </fs-listener>
 </listeners>
 <actions>
 <action name="CreateFileMessage" class="com.spss.batch.esb.CreateFileMessage" process="receivedFile"/>
 </actions>
 </service>
 </services>
</jbossesb>

[colsona]

Sorry in trying to figure out what I was doing wrong I had a bad configuration. Here is the proper one that causes my problem:

<?xml version = "1.0" encoding = "UTF-8"?>
<jbossesb parameterReloadSecs="50" xmlns="http://anonsvn.labs.jboss.com/labs/jbossesb/trunk/product/etc/schemas/xml/jbossesb-1.0.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

 <providers>
 <fs-provider name="TestFile">
 <fs-bus busid="test-file"/>
 </fs-provider>
 </providers>
 <services>
 <service is-gateway="true" name="FileListeningService" description="Service that listens for files coming in" category="FileEvent" target-service-category="FileEventActions" target-service-name="FileActionEvent">
 <listeners>
 <fs-listener busidref="test-file" maxThreads="1" name="testFileListener" poll-frequency-seconds="10" >
 <fs-message-filter input-suffix="txt" directory="C:\\FileTest" post-delete="false" post-directory="C:\\FileTest\\inProcess" error-directory="C:\\FileTest\\inError"/>
 </fs-listener>
 </listeners>
 <actions>
 <action name="CreateFileMessage" class="com.spss.batch.esb.CreateFileMessage" process="receivedFile"/>
 </actions>
 </service>
 <service is-gateway="false" name="FileActionEvent" description="Service that processes receipt of files" category="FileEventActions" >
 <listeners>
 <fs-listener busidref="test-file" maxThreads="1" name="anotherFileListener" poll-frequency-seconds="10" >
 <fs-message-filter input-suffix="txt" directory="C:\\FileTest\\inProcess" post-delete="false" post-directory="C:\\FileTest\\inProcess\\inProcess" error-directory="C:\\FileTest\\inProcess\\inError"/>
 </fs-listener>
 </listeners>
 <actions>
 <action name="CreateFileMessage" class="com.spss.batch.esb.CreateFileMessage" process="receivedFile"/>
 </actions>
 </service>
 </services>
</jbossesb>

[bernard.tison]

Asa,

You're right. The GatewayGenerator class, responsible for generating the jbossesb-gateway.xml configuration file doesn't bind the definitions to the listener definition. You can check this ion the generated jbossesb-gateway.xml file. So when configuring itself the FilegatewayListener doesn't find your custom file composer action and instead falls back on its default composer.
I guess this is a bug. I will submit a Jira request for it.

Bernard

[marklittle]

Bernard, I've updated your JIRA issue and marked this to be fixed in the GA release. Thanks.

[kurtstam]

You are right. We don't support this right now. And I don't really understand why we need it. Why can't you wait to until after the message comes into the listener? I mean you can just drop the entire file on the gateway and then turn it into a nice message once it hits the listener?

Maybe I'm missing something.

--Kurt

[estebanschifman]

Kurt, if the files are huge, it could make sense to send a message just saying 'file XXX is ready for processing' without packaging huge contents inside the message.
The message aware action can then process the file directly.

Having said this, what is stopping Asa from specifying a different composer class that just packages the file name into the message.

[marklittle]

Pass by reference makes sense.

[marklittle]

Kurt, are you suggesting that the JIRA task Bernard registered isn't valid? This appears to relate to a configuration issue.

[tfennelly]

Does this force the general issue on "how do we handle huge messages"??

Do we need a lazy load type message impl which can also page in the message.

I can't see how doing it by reference can work consistently - what if the gateway is on one machine and the ESB aware listener is on another?

[estebanschifman]

It doesn't force anything. There are mechanisms for ftp transfers that will handle the situation Tom's talking about.

[marklittle]

"tfennelly" wrote:
Does this force the general issue on "how do we handle huge messages"??

Do we need a lazy load type message impl which can also page in the message.

I can't see how doing it by reference can work consistently - what if the gateway is on one machine and the ESB aware listener is on another?

Chunking or streaming of large messages across a network is fairly well understood (a lot of grid applications do this extremely well). It often requires an additional hand-shake protocol between the endpoints in order to negotiate the flow and throttle it back/forward dynamically over time. The current architecture doesn't support it because it's predominately by-value. The only way of doing this at the moment would be to pass by reference and then punt to the higher level to determine how the streaming of the actual message happens. That certainly puts more of an effort on the higher level developer, but at the moment there is little choice. We should look at any architecture changes for after the 4.0 GA.

[marklittle]

"estebanschifman" wrote:
It doesn't force anything. There are mechanisms for ftp transfers that will handle the situation Tom's talking about.

Yes, FTP is a good way of doing this currently. However, if you can't use FTP then other changes will be necessary.

[tfennelly]

"estebanschifman" wrote:
It doesn't force anything. There are mechanisms for ftp transfers that will handle the situation Tom's talking about.

Sure, "force" was too strong - there are prob ways of getting around the issue in the short term. It's a problem we're going to have to face some time in the future and I don't suspect it's going to be a piece of cake.

[kurtstam]

Colsona,

Are you doing this because your files are big?

--Kurt