Hi,
AS for why we went with Swing over web, well, I can say that in one application it was too complex for modern day browsers. Meaning, we had the ability to add many many lines, on going, dynamically, each line containing several input boxes. But more so it worked across 7 frames! That is, we had different displays in each frame to keep the user updated. If they changed something in one box, with javascript I had to update a page in another frame. When they clicked on a submit button I had to submit 4 frames, and hope all came back in the right order, which was never the case! IT got very complex and would definitely have been much easier to implement in a swing client.
Some applications are just extremely difficult to build rapdily in web form, and I think also part of it is security, and the other thing is the much more control over the look of the app. Web is just too limited for certain types of apps, and doesn't lend well to those that require multiple views going at one time. It works, just overly complex and too easy to see get screwed up. Other business reasons play into this as well.
Not that I don't like the web stuff. I built my own MVC framework which I am about to open source, got to know servlets/jsp, clustering (through Orion at the time), session state replication, etc. I even wrote a chapter in a Wrox book on performance and scalability of web apps. I love the stuff.
As for ejb stateless, not at all what I meant. I mean keep the session beans stateless, so that communication between your stateful web layer and your stateless logic/ejb layer, is very fast, simple, and you know ejb is doing your business logic, but doesn't need to maintain any state. Entity beans as you said are stateful in that they represent the data in the DB. So naturally you have stateful entity beans, but not in the manner where they would be failed over. I can't quite come up with a reason why entity beans would ever be used for a user state? They are cached with data, store data, yes, but they shouldn't be storing state of a user, such as user id, user name, login time, etc. I mean, actually, they would be used to store that data, but you wouldn't genenerally use entity beans to store the state that is accessed from the web tier! That would mean your web has to make a remote call to session bean, which in turn has to make either a local (hopefully) or remote call to an entity bean to get that info. And if entity bean state is not replicated, you are then forced to make sure the client ALWAYS goes to the same one server, which then becomes a single point of failure again. Thus, I see no point in entity beans ever being part of the stateful fail-over equation. However, i can see where they may want to "replicate" their data across nodes so that any server an ejb session bean uses (whether remote or local) entity bean, it has the same cached data.
As for the "meat". If you are doing in-memory replication, which basically means for every node you have that is in the same cluster, its entire HttpSession state (all objects in it) are replicated to every other node in the same cluster, you have to consider that if the memory of just one node is 4GB, and the server may actually use say 2GB of that memory for session state, and all that state has to replicate to other nodes, well, each node has to have enough memory for its own state, plus every other nodes state. Two things here. First, if any one server is using 2GB worth of HttpSession, your app is seriously in need of rewriting. Session state is for things like cart data, login info, etc. Anything that is not used that often and can be retrieved by a remote lookup (which generally is going to be pretty fast due to most two teir web/ejb setups being within the same 100MB or 1GB lan space in a rack somewhere in the same colo facility... but not always), should stay there. With entity caching, and more on the ejb side, there is no need to store stuff in the HttpSession that is easily retrievable from the ejb side of things and is not used that often. On the other hand, dynamic data such as when forms are processed, that data no doubt has to stay in the http session. Again another thing I have seen is developers often don't clean up their state! They fill it up, have a ref to it somewhere, or wait for the GC to clean up. The thing is, when you are done with a particular "state", such as a user in a cart system submits his/her order, remove the state from the HttpSession. No need to keep it lingering. Just because the application is done using the state, the user logged out, whatever, if the htttpsession data for that user is not removed, it lingers, thus taking up more memory, on every node no less.
In-memory is also the fastest stateful capability, but not always the best in all cases. Ideally you only need 2 nodes to have session failover, right? But the problem is, if one node dies, you leave one node left with no fail-over path. With a 3-node setup, if one node dies, you still have two nodes.
You would also want to test performance on two nodes only, leaving the 3rd as an extra "cushion" for peak times, as well as if one node dies. Idealy, load test for about 80 to 90% capacity on two nodes. Add a 3rd node and you should be able to do continous 60% or so (give or take..) and leave room for spikes, peak traffic.
Now, let's not confuse the issue by assuming all containers are created euqal. Jetty handles in memory and DB. In-memory is best suited for 3 nodes per cluster. If you use DB, you can use a single partition, and have all nodes store state in the DB. IT is slower, and if you don't have redundancy at the load balancer and DB layer for storing state you may introduce yet another single point of failure, but you can add node after node to scale. I personally don't like this setup.
Yes, you have to use sticky sessions. A good load balancer (Cisco) is sticky/cookie aware. They will properly take a users cookie and keep track of what server it went to, always routing the user to that server. In case of failure, they are smart enough to route the user to another server in the same cluster. It may even be possible to have the load balancer updated with cpu usage per node, requests per node, and other performance measures to ensure optimum load balancing.
I am not the authoritave figure on all this. I am going by what I have researched, seen in use, and played with myself. I feel the web stack is a better and less risky place to load balance. There are specific uses for ejb statefullness, transactions in particular, although I personally haven't seen a use for it and prefer the stateless setup for its better performance and less worry about fail-over issues. I also feel that if you are doing a web app, the web tier is by and far the best place for state. You are generating pages dynamically based on the state. So why have to make calls to the ejb tier to get that state? If a user changes state and saves it, you store it through ejb. If a user flips pages and you need to determine how to build a page based on previous selections, then using HttpSession is the only way to go for performance. I suppose a combo of both could be done as well.
Hope that helps.