big soap messages in camel

[wesoly]

Hi all,

I have following camel flow:

Flow receives soap messages, sends them to remote service, waits for response and sends the response back to client.

To send soap messages to remote service camel http component is used, soap message is converted to stream and added to http message's body.

Similarly when the response comes back, it's body is converted from stream to SOAPMessage object and send back to client. This convertion is done by processor:

public void process(Exchange exchange) throws Exception {

InputStream in = (InputStream) exchange.getIn().getBody();

MessageFactory factory = MessageFactory.newInstance();

SOAPMessage message = factory.createMessage(null, in);

File f = File.createTempFile("soapmsgg", ".dat");

message.writeTo(new FileOutputStream(f)); // <-- HttpProcessor.java:22

System.out.println("Message size:" + (f.length() / 1024 / 1024) + "MB");

f.delete();

exchange.getOut().setBody(message);

}

Flow works fine for responses smaller than aproximatelly 130 MB, but when the message exceeds this value following exception occures:

Caused by: org.apache.camel.CamelExchangeException: Error processing Exchange. Exchange[Message: http://Body is instance of java.io.InputStream|http://body%20is%20instance%20of%20java.io.inputstream]/]. Caused by: http://java.lang.OutOfMemoryError - null                                                                               

at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:148)                                                                               

at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:88)                                                                               

at org.apache.camel.processor.DefaultErrorHandler.process(DefaultErrorHandler.java:49)                                                                               

at org.apache.camel.processor.DefaultChannel.process(DefaultChannel.java:228)                                                                               

at org.apache.camel.processor.Pipeline.process(Pipeline.java:74)                                                                               

at org.apache.camel.processor.UnitOfWorkProcessor.processNext(UnitOfWorkProcessor.java:66)                                                                               

at org.apache.camel.processor.DelegateProcessor.process(DelegateProcessor.java:48)                                                                               

at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)                                                                               

at org.apache.camel.component.cxf.CxfConsumer$1.invoke(CxfConsumer.java:91)                                                                               

... 22 more                                                                               

Caused by: java.lang.OutOfMemoryError                                                                               

at java.io.FileOutputStream.writeBytes(Native Method)                                                                               

at java.io.FileOutputStream.write(FileOutputStream.java:260)                                                                               

at com.sun.xml.messaging.saaj.soap.MessageImpl.writeTo(MessageImpl.java:1251)                                                                               

at my.package.HttpProcessor.process(HttpProcessor.java:22)                                                                               

at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)                                                                               

at org.apache.camel.processor.DelegateProcessor.processNext(DelegateProcessor.java:53)                                                                               

at org.apache.camel.processor.DelegateProcessor.proceed(DelegateProcessor.java:82)                                                                               

at org.apache.camel.processor.interceptor.TraceInterceptor.process(TraceInterceptor.java:93)                                                                               

at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:67)                                                                               

at org.apache.camel.processor.RedeliveryErrorHandler.processExchange(RedeliveryErrorHandler.java:177)                                                                               

at org.apache.camel.processor.RedeliveryErrorHandler.processErrorHandler(RedeliveryErrorHandler.java:143)

I suspected that Xmx option was set so I started fuse with following options:

-Xms512M -Xmx2048M -XX:MaxPermSize=512m -XX:UseConcMarkSweepGC -XX:CMSIncrementalMode

But the problem still occures. Is there any other place where I should set options somehow related to memory? Or maybe someone has encountered this problem and succesfully solved it?

Edited by: wesoly on Jul 26, 2010 1:11 PM

Edited by: wesoly on Jul 26, 2010 1:12 PM

Edited by: wesoly on Jul 26, 2010 1:17 PM

[davsclaus]

Do you process those big messages concurrently? Or is there only 1 or a few messages in progress at any given time?

And why do you need a MessageFactory and the likes to write to a file? Can't you just write directly to a file?

[wesoly]

No I do not use concurrency. In my tests there is only one message processed at given time.

MessageFactory is used to created SOAPMessage from InputStream. Writing to file is an additional element used to check the size of message and using wirte method from SOAPMessage class is an easy way to achieve this. Moreover, it still does not explain why the size of messages is limited to ~130MB.

[njiang]

How about you just use SAAJ API to write a test which builds up a SOAP message from a input stream to see how big the message you test can take?

I don't think it's a good idea to hold such a big object into the memory, it will eat up all you memory sooner or later.

Willem