You should not try to interfere with how Android manages your Fragment implementations. The default for the setOffScreenPageLimit should already be one. This means that Android will destroy old fragments when memory runs low. As long as you do not have a memory issue, just leave it be.
The reason why your memory increases is because Android keeps Fragment instances in memory to be able to reconnect to them instead of having to instantiate them. I recommend you account for the contingency of your Fragment instances being destroyed by the OS, saving their state if that happens, and let the OS do its job.
The delay you are experiencing could be due to some intensive computation on the UI thread. If it is, I suggest moving that out to, for example, an AsyncTask. Without the code it is, however, just a guess as to what might cause the issue. But there being only an initial delay suggests that you are loading something which might block the UI thread.
Update: Have a look at https://stackoverflow.com/a/9646622/170781 which outlines very neatly how the ViewPager handles Fragment instances.