Let's see:

6144MB for the heap
+256MB for the permgen
+256KB * number of threads for the stacks (the 256KB is a guess, I cannot offhand recall how big each thread stack is)
+xxxMB C/C++ data overhead (this is the JVM's data used to manage/run the Java app)
+xxxMB for the code (the JVM and the shared objects it uses)
-----------
7775MB total

Moral of the story - the heap is not everything, there is a lot of overhead above and beyond the heap.

By the way, do you really want a 6BG heap? I hope you are using the CMS collector or else you will wait a long time for each major garbage collection.

I suggest to not use an heap so BIG.
If you have problem of performance installed more jboss in mode cluster or use an CMS collector as Peter said.