I/O performance down in Jboss AS 7 for Enterprise application

[kishorerouthu]

I am working with EnterpriseApplication on Jboss AS 7 platform.

Functionality of application it that it takes XML data as request

and process it and sends XML data as response to client. Here

XML request (in kilobyts) is relatively less in size when compared to

XML response ( ranges fro KB to MB). I enabled some loggings of

request processing from this i conclude that 90% of request processing

(i.e 1.parsingXML + 2.executeBusinessLogic + 3.constructsXMLResponse +

4. SendXML response to client)

is to write back the XML response to client(i.e step 4 from the above).

And observed from live that Ram usage is always 50% - 70% and

CPU load is 0.16 - 1.0

But still i couldn't found that only write response back to client

takes 90% of response time.

following is Web configuration in Standalone.xml of Jboss As 7 and target client

has same configuration which receives the response.

<subsystem xmlns="urn:jboss:domain:web:1.1" default-virtual-server="default-host">

    <configuration>

          <static-resources sendfile="524288"/>

    </configuration>

    <connector name="http" protocol="HTTP/1.1" scheme="http" socket-binding="http" enable-lookups="true" redirect-port="8443" executor="http-executor" max-connections="50"/>

    <connector name="https" protocol="HTTP/1.1" scheme="https" socket-binding="https" redirect-port="8443" secure="true" max-connections="400">

          <ssl name="ssl" key-alias="chapter8" password="123456" certificate-key-file="../standalone/configuration/cultswitch.cultuzz.de.keystore" cipher-suite="SSL_RSA_WITH_RC4_128_MD5,SSL_RSA_WITH_RC4_128_SHA" protocol="TLSv1" verify-client="false" verify-depth="10" keystore-type="PKCS12" truststore-type="PKCS12"/>

    </connector>

    <virtual-server name="default-host" enable-welcome-root="true">

          <alias name="localhost"/>

          <alias name="vela"/>

    </virtual-server>

</subsystem>

Can anybody help with the way to find the reasons for this ?

For more information :

1. Using non-transactional datasource with following pool configuration

   <pool>

     <min-pool-size>20</min-pool-size>

     <max-pool-size>50</max-pool-size>

     <prefill>false</prefill>

     <use-strict-min>false</use-strict-min>

     <flush-strategy>FailingConnectionOnly</flush-strategy>

   </pool>

2. Local EJB's with CONTAINER managed transactions.

3. Configured for recovery-environment for transactions recovery for every 2 minutes.

4. Threads subsystem configuration as follows

   <subsystem xmlns="urn:jboss:domain:threads:1.1">

        <bounded-queue-thread-pool name="http-executor">

           <core-threads count="50"/>

           <queue-length count="50"/>

           <max-threads count="50"/>

           <keepalive-time time="10" unit="seconds"/>

           </bounded-queue-thread-pool>

   </subsystem>

Hardware Specification : 8GB RAM

Processor Information (2 processors):

vendor_id          : GenuineIntel

cpu family          : 6

model                    : 44

model name          : Intel(R) Xeon(R) CPU           E5645  @ 2.40GHz

stepping          : 2

cpu MHz                    : 2400.085

cache size          : 12288 KB

physical id          : 0

siblings          : 2

core id                    : 1

cpu cores          : 2

apicid                    : 1

initial apicid          : 1

fpu                    : yes

fpu_exception          : yes

cpuid level          : 11

wp                    : yes

flags                    : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush dts mmx fxsr sse sse2 ss ht syscall nx rdtscp lm constant_tsc arch_perf

mon pebs bts xtopology tsc_reliable nonstop_tsc aperfmperf unfair_spinlock pni pclmulqdq ssse3 cx16 sse4_1 sse4_2 popcnt aes hypervisor lahf_lm arat epb dts

bogomips          : 4800.17

clflush size          : 64

cache_alignment          : 64

address sizes          : 40 bits physical, 48 bits virtual

Softwares Specification:

ApplicationType : EnterpriseApplication

Java : 7.1

Application Server : Jboss AS 7

DataBase : MySQL

EJB : 3.0

Hibenate : 4.0.1

Do i need any changes in Standalone configuration ?

Do i need any changes in I/O of OpertingSystem ?

[sfcoy]

If you have potentially large responses then your process can be improved considerably by changing steps 3 & 4 of your logic.

You can use the javax.xml.stream API to build your response and write it directly to the output stream as you go. This saves a lot of memory and the GC side effects from generating the entire XML structure in a growing buffer.

It's possible to use this in conjunction with other DOM to XML strategies such as JAXB to simplify your XML generation. In the past I have used JAXB to generate small XML sub-elements and then stream them out using XML streaming. This trick makes your fat XML generation scalable.

What are you comparing JBossAS 7.1 with when you say that your I/O performance is down?

[kishorerouthu]

Thank you Stephen Coy for your reply

Actually i build the XML response as a string and writing bytes from the string to outputStream.

Is it cuase to slow down by writing bytes to outputstream ?

Example :

This is only end of processing request before it includes lots of business calcultions over xml data

               1. byte[] rsBytes = resXML.getBytes();

                2.response.setContentType("text/xml");

                3.int contentLength = otaRSBytes.length;

                4.response.setContentLength(contentLength);

                5.response.setCharacterEncoding("UTF-8");

                6.out = response.getOutputStream();

                7.out.write(rsBytes);  

                8.out.flush(); 

                9.out.close();

90% response time takes from line 2 to line 7 .

What are you comparing JBossAS 7.1 with when you say that your I/O performance is down?

I am expecting that because of any bad configuration on jboss standalone mode some backgroud

processes are always accessing to I/O that may cause to slow down the actuall response write to outputstream.

Can you please suggest me in this way ?

[sfcoy]

What is reading the response?

If you're using a web browser you may be slowed down by the ability of the browser to read and render the response.

Try using curl or wget on a unix platform in order to avoid this possibility.

[kishorerouthu]

Thank you for your replay

Here i am not using the browser to render the response. But the response is send back to Client Program from other system

in same network ?

Is there any changes required from Client end to receive response fastly ?

[sfcoy]

Are you able to use curl or wget from a unix system to see what effect this has on the server?

What is the nature of the client?

You may need to profile both the client and server to discover where the bottlenect lies.

[kishorerouthu]

Thank Stephen for your replay

i can able to use curl to findout bottleneck .

But i don't know how to work with curl to findout where the bottlenecks are

can you provide me that inforamtion ?

What is the nature of the client?

The nature of client is also same as Server means that it has same configuration as Server with Jboss as 7

and Hardware.

Can provide me how to monitor client and server to discover I/O bottlenecks ?

[sfcoy]

Using curl is very easy.

{code:bash}[steve@steves-mbpw ~]$ curl http://www.jboss.org/overview/{code}

You can find more information about curl using:

{code:bash}[steve@steves-mbpw ~]$ man curl{code}

As curl is a native application, it is unlikely to become a read-side bottleneck. It will read the output as fast as it can be delivered.

If the output from curl is slow, then the problem is likely to be server side. If it's fast, then it will be your java client that is slowing things down.

[kishorerouthu]

Thank u Stephen Coy for your continues replies on my posts

All your previous answers were helpful for me.

But here my concern is how do i improve the  performance of send data to client from Servlet.

Here the response data encapsulated in String and size varies from 200KB to 3MB.

Following is current code snippet to send response to client

               1. byte[] rsBytes = resXML.getBytes();

               2.response.setContentType("text/xml");

               3.int contentLength = otaRSBytes.length;

               4.response.setContentLength(contentLength);

               5.response.setCharacterEncoding("UTF-8");

               6.out = response.getOutputStream();

               7.out.write(rsBytes);  

               8.out.flush(); 

               9.out.close();

What are the varies ways to send response from Server (through Servlet) to Client and Which is efficient among

them ?

How to tune Client and Server  at any level (i.e 1.ApplicationServerConfiguration, 2. OS level , 3.Java) 

in order to acheive over network latency of sending response between Server and Client. ?

***** And i already  provided all ApplicatoinServler configurations for Client and Server.

Please suggest me for the above.

Thanks in advance

[sfcoy]

What happens if you do this:

{code:java}response.setContentType("text/xml");
response.setCharacterEncoding("UTF-8");
response.getWriter().print(resXML);
{code}

What happened when you tested with curl?

[kishorerouthu]

Now my code is working with ServletOutputStream

               response.setContentType("text/xml");

               int contentLength = otaRSBytes.length;

               response.setContentLength(contentLength);

               response.setCharacterEncoding("UTF-8");

               out = response.getOutputStream();

               out.write(rsBytes); 

               out.flush();

but didn't tried with response.getWriter().print(resXML);

whether PrintWriter on String is efficient than the OutputStream ?

What happened when you tested with curl?

Here i observed that the problem is from server side .

[sfcoy]

Well, if you have 50MB of XML in a java.lang.String, this

{code:java}byte[] rsBytes = resXML.getBytes(){code}

creates a copy of the character data. Now you have consumed 100MB of memory,

and then:

{code:java}out.write(rsBytes);{code}

buffers the bytes again, consuming a further 50MB of memory.

You can mitigate the last one by adding:

{code:java}...
response.setCharacterEncoding("UTF-8");
response.setBufferSize(1024 * 1024); // 1MB
...{code}

Furthermore, resXLM.getBytes() returns the character bytes in the platform encoding. If that's not UTF-8 then all sorts of things could be going on.