The behavior you are seeing could be the result of a significant garbage colelction. You are not, by any chance, using the CMS collector? If so, if that collector gets behind in its duties, a full serial collection takes place, and that collection is usually very painful. One way to check this is to monitor the garbage collection data using -XX:+PrintHeapAtGC.
See my presentation at http://www.cecmg.de/doc/tagung_2007/agenda07/24-mai/2b3-peter-johnson/index.html