I suspect you ran out of perm space (where class objects are loaded). Try running the application with this option: -XX:+PrintHeapAtGC (redirect the output to a file, there could be a lot of it). You will then need to look at the heap sizes reported at each garbage collection to see if the perm space is running out of room. To adjust the perm space size, use the option: -XX:PermSize=, where is something like 128M.
I think you hit the seak spot. If I double the perm space to 128M, I can loop over 9500 'records'.
Turning on garbadge collection debugging, I see that when the perm space is getting full, it removes a long list of on-the-fly generated classes. It needs 4 Full GC cycles and more then 30 seconds to do this. At then end, the perm space and tenure space have enough free memory, but it still generates an OutOfMemoty exception.
The gc log gives the following 4 successive garbadge collections:
405.633: [Full GC 405.633: [Tenured: 197077K->194077K(932096K), 2.9671419 secs] 197235K->194077K(1036928K), [Perm : 131071K->130711K(131072K)], 2.9672640 secs] 410.008: [Full GC 410.008: [Tenured: 194077K->194452K(932096K), 2.1127909 secs] 214844K->194452K(1036928K), [Perm : 131071K->131071K(131072K)], 2.1128859 secs] 412.138: [Full GC 412.138: [Tenured: 194452K->194452K(932096K), 2.0313302 secs] 194603K->194452K(1036928K), [Perm : 131071K->131071K(131072K)], 2.0314277 secs] 414.258: [Full GC 414.258: [Tenured: 194452K->119631K(932096K), 26.2546285 secs] 194874K->119631K(1036928K), [Perm : 131071K->60643K(131072K)], 26.2547290 secs]
Any clue why after having cleaned up all the memory, and having more then enough free memory, it is still returning an OutOfMemoryError?
Does the memory allocation have some sort of 'timeout'? (I tried decreasing the perm space so it would take less time, but this did not help.)
No, there is no timeout. When an object needs to be allocated and there is no room, the app is stopped until the garbage collector is done, and then allocation happens. Unless you are running the concurrent collector.
I have never seen such a huge drop in the perm gen. The output you showed looks to be from -XX:+PrintGCDetails, not -XX:+PrintHeapAtGC. The later also prints a list of classes freed, it would be interesting to see what classes were garbage collector during that last collection.
What is the full stack trace for the OutOfMemory exception? Perhaps knowing what was being allocated might help debug this issue.