Ok, I think I got it.
It seems that the Java Hotspot VM fails to Garbage Collect if the Heap-size is to high. Therefore the Server runs for about an hour and even a shutdown and restarts has no effect. Just a reboot of the machine makes it possible to release the Heap. With a parameter of the Hotspot VM (think it was xn or xm) the max-size of the Heap can be settet.
Until now I had no problems with that.
A friend told be that i should switch to the IBM-VM. What are you thinking about that all?
Have a look at the performance analysis linked from this topic:
http://www.jboss.org/modules/bb/index.html?module=bb&op=viewtopic&t= document link referred to in the topic gives more information on IBM and SUN JVM comparisons on Linux. In general, we've had no issues with either JVM except IBM does seem to have an advantage in resource usage and speed. Ultimately, you'll need to make up your own mind.