We haven't had a lot of community input re usage of NLB, and we test with it or use it internal within the JBoss division, so that makes it somewhat a Not Good Idea since it will be harder to get community support. I don't have any real idea what your problem could be, only than I found a probably irrelevant post via google:
A quick glance at NLB docs make it look like its support for session affinity is not so hot. Also, it uses client addresses to pin users to a server. If your clients are sitting behind corporate firewalls, you could have a lot of users who appear to be using the same address. They'll all get pinned to the same server.
Re: mod_jk vs. a hardware load balancer, the hardware load balancer is also a potential single point of failure (unless you get two, and you can always get 2 apache boxes as well).