High VM Memory analysis and lessons learnt

This week I investigated a high VM memory utilization on a JBoss servers in production
Using the Linux top command found that the JBoss - Java process was hogging 6.8 GB (Resident Memory). We have set JVM to 6144MB JVM.
Lets look at the sizing formula - 

Max memory = [-Xmx] + [-XX:MaxPermSize] + number_of_threads * [-Xss]

6144 MB + 384 MB + 25 MB (Say 100 threads * 256 K = 25mb) = 6553 MB (Close to 6.8 GB) +
Read this article -

https://dzone.com/articles/why-does-my-java-process which says - But besides the memory consumed by your application, the JVM itself also needs some elbow room.


This means the actual memory the JBoss uses is not equal to the JVM set, but its more at least in case of JAVA 1.7.

One more thing I learnt in investigation. There were several other java processes related to machine agent. We use a APM tool and this tool has a  machine agent which is used to collect the VM resources for correlation Recently the script to restart the JBoss servers were changed from killall -9 java to kill -9 . Due to this for some reason the machine agent processes were left hanging for each day. Each process was hogging 50-60 MB which over days becomes significant given that OS also requires memory.I wonder what would come out as the root cause of the Application servers swapping. It would sound something like this - " The issue of slowness/servers crashing/swapping was due to a APM tool agent" :)