Increasing CPU utilization on a multi-core server

[bh2013]

I would be surprised if this question hasn't been asked before but I am having a hard time finding it and its answer if it has.

I have a server running RHEL 6.2 which has 4 CPU cores. I am running an application on it that makes heavy use of EJBs called by external clients (external to JBoss, different JVM). I am not getting the performance I expected and noticed I am not getting over 25% CPU utilization. The calls are CPU intensive so just one request will consume 25%, or 1 core of the 4 core CPU. It doesn't matter if I submit 1 request or a dozen requests, the CPU utilization is always the same.

I wrote another J2EE application that does nothing but eat CPU cycles in a tight loop for several seconds. I did this just to see if I could get JBoss to use more than 25% CPU. I discovered something I found interesting. When the code is run inside a servlet, I cannot get more than 25% CPU utilization no matter how many concurrent requests to the servlet I make. When my code is run inside an EJB, I get the same thing. No matter how many concurrent requests to the EJB are made, I do not get above 25%. However, if I submit one request for the servlet and concurrently submit one request for the EJB, I get 50% utilization.

I am starting JBoss with the "-server" argument. Any idea why the EJB container would be constrained to a single CPU core and the web container also constrained to a single CPU core? I would like to have JBoss able to utilize the full processing power of the server.

The application I am running takes resource intensive requests from few users. A dozen clients would be considered a lot in this case. If JBoss is keeping processing power in reserve in case a thousand more requests might come in, that isn't going to happen.

Any help would be appreciated.

[wdfink]

I've no problem to have all CPU's under load with EJB applications.

Do you see that the requests are really concurrent.

You may have a configuration that block real parallel working (i.e. only one thread in the pool)

Or application that is synchronized for some reason.

Which version do you use, do you have a simple reproducer?

[bh2013]

The environment is RHEL 6.2 running JBoss 5.1.2 EAP. The application is a purchased application called EMC xPression which does batch processing using other JVM instances that communicate with the JBoss application server to perform tasks. Using WebSphere this application can easily use all your CPU. We are migrating from WebSphere to JBoss and seeing our CPU utilization is limited somehow.

In order to isolate if the issue is xPression or JBoss, I wrote my own application that just consumes CPU in a tight loop for several seconds. I wrote my test to do it in either EJBs or servlets depending on the request. I can submit multiple requests to either servlets OR EJBs and never see CPU utilization go above 25%. But if I submit to each having some requests go to servlets and others go to EJBs then I can get up to 50%. It behaves like servlets can only use up to 1 CPU equivalent and EJBs can use up to 1 CPU equivalent.

I also tested this on my personal computer at home running JBoss AS 7. My home machine has 2 CPUs and runs Fedora 18. At home on my 2 CPU machine servlets can use 50% but no more and EJBs can use 50% but no more. If I submit a mix of requests using servlets and EJBs then I will see 100% utilization. But if I submit strictly servlet only requests or strictly requests using EJBs I never see JBoss using more than 50% cpu utilization on my 2 CPU computer.

[ctomc]

Can you post your test code and scenario how / with what tools ware you testing doing requests?

[bh2013]

I can't post the purchased application that seems to be throttled to only using a single CPU but the code I wrote specifically to CPU is pasted below. I tried to control the number of threads by submitting multiple requests. The server at work goes to 25% on a 4 core system but will not go higher whether I hit it with 1 request or a dozen. My home machine with 2 CPUs will not go over 50% no matter how many requests I throw at it at one time. It just takes longer to process each request.

The "-server" option is used when starting JBoss otherwise I thought that might be limiting me to a single CPU. Below is the servlet code. I also put the same thing in an EJB and discovered EJB calls can take up 1 CPU core by themselves and servlet calls can take up a CPU so by submitting requests to both I can use 100% of my home machine (JBoss 7) or 50% of the server at work (JBoss 5.1.2 EAP).

// Servlet process method

	protected void processRequest(HttpServletRequest request, HttpServletResponse response)
	throws ServletException, IOException {
	response.setContentType("text/html;charset=UTF-8");
	long limit = 0l;
	String message = "";
	String tmpLoopTo = request.getParameter("loopTo");
	if (tmpLoopTo != null) {
	try {
	limit = Long.parseLong(tmpLoopTo);
	message = "Using user defined 'loopTo' limit of " + limit;
	} catch (NumberFormatException e) {
	limit = Def.DEFAULT_LIMIT;
	message = "Using default 'loopTo' value of " + Def.DEFAULT_LIMIT;
	}
	}
	long start = System.currentTimeMillis();
	for (long i = 0l; i < limit; ++i) {
	long time = System.currentTimeMillis();
	time /= time;
	}
	long end = System.currentTimeMillis();

	PrintWriter out = response.getWriter();
	try {
	/* TODO output your page here. You may use following sample code. */
	out.println("<!DOCTYPE html>");
	out.println("<html>");
	out.println("<head>");
	out.println("<title>Servlet ThreadTestNoEJB</title>");
	out.println("</head>");
	out.println("<body>");
	out.print("<h1>Servlet ThreadTestNoEJB at " + request.getContextPath() + "</h1><p>");
	out.print(message);
	out.print("</p><p>Processing time took ");
	out.print(end - start);
	out.println(" milliseconds.</p>");
	out.println("<p><a href=\"index.jsp\">Return to Home</p>");
	out.println("</body>");
	out.println("</html>");
	} finally {
	out.close();
	}
	}

[bh2013]

I was running some more tests to get some thread traces and noticed something interesting. If I submit a request in Firefox using 6 tabs loading the same page concurrently, I cannot break 1 CPU core being fully utilized. Same thing if I use 6 Chrome tabs. But if I submit some in Firefox and Some in Chrome, I can use both my CPU cores on my home machine.

Does JBoss do something to limit how much resources a single client can use?

[fsl]

It looks to me your issue has nothing to do with jboss limiting anything. It looks you client -- EMC xPression, if I understand correctly -- is the cause. From that you've written, I'd guess the application sends all requests to JBoss using only one (client) thread, and so all it's requests gets serialized. To put in another way, JBoss is not getting concurrent requests, so it won't be able to spread the work among concurrent server threads.

To see how jboss can actually run your test servlet on all cores, try using the command "ab -c 30 -n 100000 <your url>", It will create -c proccesses sending -n requests to <your url>, unlike the browser, who will limit concurrent connections.

If you cannot change your client / application behaviour, I sugest a workaround: make your REST services or EJBs just get the requests and send them as messages to a JMS Queue. An MDB will get those messages and do the actual request processing. Then you'll be able to tune the MDB session and instance pool sizes to get the desired concurrency level, and put all yout CPUs to work.

It would be even better if you client could submit jobs directly to the JMS Queue, instead of sending REST/EJB calls.

[bh2013]

I agree with you, Fernando. I was just mislead when my test gave similar results until I happened to try two browsers. It seems submitting requests from multiple tabs in the same browser is also a limiting factor.

You have given me good information to discuss with the vendor. Thank you.

[fsl]

Actually I'd expect software like xPression (batch job contollers) to limit concurrency. Batch jobs usually are very resource intensive, a few of them can easily bring any server to it's knees. Sure they have some setting for you to tell how many simultaneous jobs to run, or how many connections to the same application server to open.

That said, it's also true that no web browser will make more than a few simultaneus concurrent connections. After all, one user cannot interact with multiple tabs at the same time. A new tab is not a new user. :-) That's why I sugested using apache "ab" (standard on all linux distros, it comes with apache httpd) to do a quick load test. But be warned, "ab" is too simplistic, if you need a real load test tool please check Apache JMeter.