8 Replies Latest reply on Oct 26, 2016 6:02 AM by ctomc

    Undertow/WildFly https-remoting: much slower than remoting: performance under load

    darrenjones

      I have been running some performance tests of our application against WildFly 8.1.0.CR1. The client application is a Java app calling SLSBs using the ejb-client API and the https-remoting protocol to take advantage of the single port WildFly configuration.

       

      As the server load increases (i.e. more users are calling the server concurrently), I am seeing that the time it takes the client application to login increases dramatically. The attached graph shows the behaviour - the blue dots on the graph are the timings for the login step and you can see the login time increasing in a staggered and odd way. The other dots are for other things the application does once logged in, and do not show such an odd pattern.

       

      Interestingly, if I switch back to using the plain remoting protocol on the old remoting port of 4447 (still over SSL, but no http-upgrade), I see no slowdown for the login step - which is why I am currently suspecting the http upgrade process.

       

      Is there a possibility of a bottleneck somewhere in the http-upgrade code of either Undertow or the remoting http upgrade handler?

       

      The only theory I have at the moment is that the http upgrade code is currently being executed on an Undertow I/O thread (I can see this when I attach a debugger) of which there are relatively few (CPU cores * 2 by default). At a period of high server load, those I/O threads may queue up a lot of I/O tasks, and when an http upgrade request comes in, it will be added to the end of the queue. It could then take a while getting the tasks at the front of the queue executed, thus taking a while for it to get round to the http upgrade task.

       

      My workaround is to go back to using the old plain remoting, but I feel that would be a shame.

       

      Any thoughts/fixes appreciated!