Remote Tomcat->JBoss comm performance

[katsteve]

Hoping someone can point out what's going on here under the covers of the JBoss client.

In my latest tests, 85% of the total page load time in my remote Tomcat WebServer is spent attempting to connect to, or RMI invoking in, Jboss EJB. The remaining 15% is serving up static pages or constructing JSPs. Jboss itself, however is flying comparatively. For example, the home.create() method call completes in milliseconds in Jboss (per bean statistics), but the Tomcat invocation of that method will take up to 60s. Just the java naming Context.lookup(beanName) alone is taking up to 30s. What is the Webserver/client doing for all that time? WebServer (Apple G5) CPU is jumping to, and staying at, 90-100%.


Application Architecture
4-tier: Client, WebServer, AppServer, Database
WebServer - Tomcat v5, all JSPs, Apple G5 w/2G RAM
JVM: v1.4x: -XX:+AggressiveHeap -XX:+UseParallelGC, -noclassgc, -XX:NewRatio=2
Have also tried -Xmx 1.6g -Xms 1-1.6g, -Xss 2-5m
Tomcat: server.xml for standard Http Connector on 8080:
min/maxSpareThreads=1000, maxThreads=5000, acceptCount=100
enableLookups=false, debug=0
AppServer - Jboss v3.2.2, EJBs, Windows2000 Server w/4cpu, 4G RAM
JVM: v1.4x: -Xmx 1.6G -Xms 1G, no -Xss setting, -noclassgc, -XX:+UseParallelGC
Any settings of -Xss generate OutOfMemory errors at startup.
-Xmx > 1.6G generates OutOfMemory errors at startup.
Jboss: based on Jboss default - this is the only additional app, deployed via an ear.
3 Session beans, 2 stateless, 1 stateful, several entity beans, all CMT, BMP
Jboss-service.xml: PooledInvoker mbean:
MaxPoolSize=1000
MaxClientPoolSize=500 (don't know what the diff between these attrs is?!)
NumAcceptThreads=50
Standardjboss.xml:
Stateless container-conf: MaximumSize=5000
Stateful cache-policy: min/maxCapacity=100/1000; container-conf: min/maxSize=1000/5000
BMP cache-policy: min/maxCapacity=100/1000; container-conf: min/maxSize=1000/5000, commit-option=A
-ds.xml:
2 local-tx resource mgrs, point at same database.
Both set to min-/max-pool-size = 50/100.
Database: MySQL, v4, Apple G5 w/2GRAM
Only change from out-of-the-box: --max_connections=1000
Also have used MSSQLServer2000 on Windows 4cpu/4G RAM - no noticeable difference.

Application Design/Description:
WebServer
- always starts user session.
- User may visit static pages, or query for up to 2 "game cards", or register (user info).
- Common Command-Controller/ServiceToWorker/ResourceLocator Design patterns.
AppServer
- 3 Façade Session beans, with 1 local Session bean for login security
- All Entity Beans are local.
- DTO Factory Pattern for all data to/from client
- Generic Attribute Access pattern for PrimaryKey generation
- DTOs may be cached on App Server
Database:
- All incremental long primary keys in every table.
- All keys are indexed.

Client:
Custom client session generator/script runner.
Page responses are captured but ignored, very short delays between pages, iterations.

Tests:
Create a new user session every 2 seconds, up to 500 clients. Continue until test stopped.
Randomly select a script:
1) Lookup game card, no login
2) Login, lookup game card, compare to another game card
3) Register (and be assigned to a game card), lookup my game card.
4) Login, show lists of cards


Sorry for the long windedness - I've been at this 4 months.

[triathlon98]

What are the ip addresses and server names of the two boxes?
Are you trying to access the Jboss server by IP address?

Possibly the server is replying using a computer name and this cannot be resolved in DNS.
This can be fixed by starting the server with --host ip as parameter and making sure that hostname does not resolve to that ip address (but only localhost).

You can alwyas telnet to port 1099 on your JBoss server to check what JBoss is returning when trying to do a jndi lookup.

However, I have the impression you are using a tier too much. Your app would also benifit a lot from using the embedded tomcat (in JBoss). This way, there are no serialization costs (if you configure it right) and you can gain a lot of speed.

Joachim

[katsteve]

Thanks for your insights! After some experimenting, I'm not sure I understand your solution. I can't find any documentation for a "--host ip" startup parameter (or any startup params for that matter) in the Admin/Dev guide. I did find UseHostName option in the Http Invoker service, but I thought EJB/RMI used only the jnp invoker (or is this delegated to http?) Is this the switch you meant? There's no reference to an IP Address in the HttpInvoker service. How do I ensure that the hostname doesn't resolve to the ip address as you mentioned?

I verified that my webserver machine is definitely unable to resolve the DNS (internal network) name for the app server, as I can telnet to the ip address, but not the dns name (which I can do from other machines in the network). My telnet response from 1099 has alot of gobbledy gook, but includes a reference to "http://0.0.0.8083". No mention of a dns name in it.

As for the last architecture suggestion - thanks, and I thought of that originally, but I am expecting very large loads on the webserver, and I would prefer not to share the resources with the app server.

Thanks again for any help you can give - (I'm going to flip the UseHostName switch now anyway)
-Steve

[triathlon98]

Right, that "http://0.0.0.8083" bit looks very wrong.

Now if only I could find thse docs about "--host" again :-)
Docs about this can be found when discussing running multiple jboss instances and/or multi-homed systems.

You have to add the parameter to the run.bar or run.sh script, for example

run.sh --host "192.168.0.2"

Joachim

[katsteve]

I did change the UseHostNames switch in HttpInvoker.sar - jboss-service.xml, and now my telnet comes back with the IP Address - yay! However, I re-ran the performance test, and there was no difference :-(

Could this be network traffic? All connections are 10/100.