Just an idea: NIO buffers are placed outside the JVM.
EDIT:
As per 2016 it’s worth considering @Lari Hotari comment [ Why does the Sun JVM continue to consume ever more RSS memory even when the heap, etc sizes are stable? ] because back to 2009, RHEL4 had glibc < 2.10 (~2.3)
Regards.