-
1. Re: Increasing CPU utilization on a multi-core server
wdfink Oct 21, 2013 3:06 PM (in response to bh2013)I've no problem to have all CPU's under load with EJB applications.
Do you see that the requests are really concurrent.
You may have a configuration that block real parallel working (i.e. only one thread in the pool)
Or application that is synchronized for some reason.
Which version do you use, do you have a simple reproducer?
-
2. Re: Increasing CPU utilization on a multi-core server
bh2013 Oct 21, 2013 4:11 PM (in response to wdfink)The environment is RHEL 6.2 running JBoss 5.1.2 EAP. The application is a purchased application called EMC xPression which does batch processing using other JVM instances that communicate with the JBoss application server to perform tasks. Using WebSphere this application can easily use all your CPU. We are migrating from WebSphere to JBoss and seeing our CPU utilization is limited somehow.
In order to isolate if the issue is xPression or JBoss, I wrote my own application that just consumes CPU in a tight loop for several seconds. I wrote my test to do it in either EJBs or servlets depending on the request. I can submit multiple requests to either servlets OR EJBs and never see CPU utilization go above 25%. But if I submit to each having some requests go to servlets and others go to EJBs then I can get up to 50%. It behaves like servlets can only use up to 1 CPU equivalent and EJBs can use up to 1 CPU equivalent.
I also tested this on my personal computer at home running JBoss AS 7. My home machine has 2 CPUs and runs Fedora 18. At home on my 2 CPU machine servlets can use 50% but no more and EJBs can use 50% but no more. If I submit a mix of requests using servlets and EJBs then I will see 100% utilization. But if I submit strictly servlet only requests or strictly requests using EJBs I never see JBoss using more than 50% cpu utilization on my 2 CPU computer.
-
3. Re: Increasing CPU utilization on a multi-core server
ctomc Oct 21, 2013 4:19 PM (in response to bh2013)Can you post your test code and scenario how / with what tools ware you testing doing requests?
-
4. Re: Increasing CPU utilization on a multi-core server
bh2013 Oct 23, 2013 4:02 PM (in response to ctomc)I can't post the purchased application that seems to be throttled to only using a single CPU but the code I wrote specifically to CPU is pasted below. I tried to control the number of threads by submitting multiple requests. The server at work goes to 25% on a 4 core system but will not go higher whether I hit it with 1 request or a dozen. My home machine with 2 CPUs will not go over 50% no matter how many requests I throw at it at one time. It just takes longer to process each request.
The "-server" option is used when starting JBoss otherwise I thought that might be limiting me to a single CPU. Below is the servlet code. I also put the same thing in an EJB and discovered EJB calls can take up 1 CPU core by themselves and servlet calls can take up a CPU so by submitting requests to both I can use 100% of my home machine (JBoss 7) or 50% of the server at work (JBoss 5.1.2 EAP).
// Servlet process method
protected void processRequest(HttpServletRequest request, HttpServletResponse response) throws ServletException, IOException { response.setContentType("text/html;charset=UTF-8"); long limit = 0l; String message = ""; String tmpLoopTo = request.getParameter("loopTo"); if (tmpLoopTo != null) { try { limit = Long.parseLong(tmpLoopTo); message = "Using user defined 'loopTo' limit of " + limit; } catch (NumberFormatException e) { limit = Def.DEFAULT_LIMIT; message = "Using default 'loopTo' value of " + Def.DEFAULT_LIMIT; } } long start = System.currentTimeMillis(); for (long i = 0l; i < limit; ++i) { long time = System.currentTimeMillis(); time /= time; } long end = System.currentTimeMillis(); PrintWriter out = response.getWriter(); try { /* TODO output your page here. You may use following sample code. */ out.println("<!DOCTYPE html>"); out.println("<html>"); out.println("<head>"); out.println("<title>Servlet ThreadTestNoEJB</title>"); out.println("</head>"); out.println("<body>"); out.print("<h1>Servlet ThreadTestNoEJB at " + request.getContextPath() + "</h1><p>"); out.print(message); out.print("</p><p>Processing time took "); out.print(end - start); out.println(" milliseconds.</p>"); out.println("<p><a href=\"index.jsp\">Return to Home</p>"); out.println("</body>"); out.println("</html>"); } finally { out.close(); } } -
5. Re: Increasing CPU utilization on a multi-core server
bh2013 Oct 24, 2013 12:14 PM (in response to bh2013)I was running some more tests to get some thread traces and noticed something interesting. If I submit a request in Firefox using 6 tabs loading the same page concurrently, I cannot break 1 CPU core being fully utilized. Same thing if I use 6 Chrome tabs. But if I submit some in Firefox and Some in Chrome, I can use both my CPU cores on my home machine.
Does JBoss do something to limit how much resources a single client can use?
-
6. Re: Increasing CPU utilization on a multi-core server
fsl Oct 25, 2013 9:59 AM (in response to bh2013)It looks to me your issue has nothing to do with jboss limiting anything. It looks you client -- EMC xPression, if I understand correctly -- is the cause. From that you've written, I'd guess the application sends all requests to JBoss using only one (client) thread, and so all it's requests gets serialized. To put in another way, JBoss is not getting concurrent requests, so it won't be able to spread the work among concurrent server threads.
To see how jboss can actually run your test servlet on all cores, try using the command "ab -c 30 -n 100000 <your url>", It will create -c proccesses sending -n requests to <your url>, unlike the browser, who will limit concurrent connections.
If you cannot change your client / application behaviour, I sugest a workaround: make your REST services or EJBs just get the requests and send them as messages to a JMS Queue. An MDB will get those messages and do the actual request processing. Then you'll be able to tune the MDB session and instance pool sizes to get the desired concurrency level, and put all yout CPUs to work.
It would be even better if you client could submit jobs directly to the JMS Queue, instead of sending REST/EJB calls.
-
7. Re: Increasing CPU utilization on a multi-core server
bh2013 Oct 25, 2013 10:06 AM (in response to fsl)I agree with you, Fernando. I was just mislead when my test gave similar results until I happened to try two browsers. It seems submitting requests from multiple tabs in the same browser is also a limiting factor.
You have given me good information to discuss with the vendor. Thank you.
-
8. Re: Increasing CPU utilization on a multi-core server
fsl Oct 25, 2013 10:47 AM (in response to bh2013)Actually I'd expect software like xPression (batch job contollers) to limit concurrency. Batch jobs usually are very resource intensive, a few of them can easily bring any server to it's knees. Sure they have some setting for you to tell how many simultaneous jobs to run, or how many connections to the same application server to open.
That said, it's also true that no web browser will make more than a few simultaneus concurrent connections. After all, one user cannot interact with multiple tabs at the same time. A new tab is not a new user. :-) That's why I sugested using apache "ab" (standard on all linux distros, it comes with apache httpd) to do a quick load test. But be warned, "ab" is too simplistic, if you need a real load test tool please check Apache JMeter.