NIO benefits should be taken with a grain of salt.
In a HTTP server, most connections are keep-alive connections, they are idle most of times. It would be a waste of resource to pre-allocate a thread for each.
For MMORPG things are very different. I guess connections are constantly busy receiving instructions from users and sending latest system state to users. A thread is needed most of time for a connection.
If you use NIO, you’ll have to constantly re-allocate a thread for a connection. It may be a inferior solution, to the simple fixed-thread-per-connection solution.
The default thread stack size is pretty large, (1/4 MB?) it’s the major reason why there can only be limited threads. Try reduce it and see if your system can support more.
However if your game is indeed very “busy”, it’s your CPU that you need to worry the most. NIO or not, it’s really hard to handle thousands of hyper active gamers on a machine.