What would cause JBoss to take up 4 more GB of memory than what is allocated?

Relevant part of the start up line: java -server -Xms10G -Xmx10G -XX:PermSize=1G -XX:MaxPermSize=1G

This instance ended up taking up 16GB of memory and 10GB of swap before killing the server.

Any ideas on what could cause that?

This is the only major application running on a RedHat system with 16GB RAM and 10GB swap.

From serverfault Ichorus

• My Guess would be runaway threads maybe, really probably better stack overflow on how to profile memory usage for a Java App and look for memory leaks.

  Keep in mind that the options -Xms and -Xmx are for heap only. There are other things that take up memory such as thread stacks etc. So maybe there were run away threads?

  You might want to look into ulimit for limit what this application can take, and start using something like Nagios to alert you when memory usage gets out of control. You running the jvm as root?

  Ichorus : "You running the jvm as root?" No. We found the answer and it is scary. One detail that I left out is that we are running on a virtualized server on VMWare. Originally we defined the memory to 8GB in VMWare. We changed it to 16GB when we decided to move that one into a production cluster. Apparently, VMWare, despite showing that there were 16GB available, really only had the original 8 (corraborated by sar data leading up to the crash). When we hit that limit, it went into swap, used that up completely and then, of course, the instance died. We are talking to VMWare now about it.

  djangofan : Run VirtualVM against that server and tell us which type of memory is getting out of hand in your VM.
  From Kyle Brandt