Are you, by any chance, running with the kernel option that shows each thread as its own process? I can't recall the option name off hand, but if you do a "ps -ef | grep java", how many entries do you get?
Based on the jps output, I suspect that 21997 is the process (and the master thread for the process) while 21998 is just a thread.
The best way to find out what is taking up all the processor is to run jstack several time, each time a few seconds apart, and look at the threads that are still in the same code. You can ignore the http threads waiting on objects - instead look for threads running your code.
Thank you for your reponse
I am running on linux so each thread has a PID
The main thread is 21997
but htop shows that pid 21997 AND 21998 are eating the CPU but jstack 21997 does not provide anything on 21008
It is like if htop is telling me processId 21997 created thread 21998 and thread 21998 is eating the CPU
But jstack is telling me process 21997 doesn't have anything on thread 21998
In my setup, the threads do not have process ids, so I cannot tell if the Java stack dump (jstack output) mentions the process id associated with each thread. But based on your observations, I would say that it doesn't. None the less, the thread identified as pid 21998 is in the jstack output. Like I wrote earlier, you might have to take several stack dumps, and compare them, to identify the busy thread.