KISS Answer:
Simple use FragmentStatePagerAdapter instead of FragmentPagerAdapter.
I got the answer.. Firstly I thought to delete this question as I am doing a very silly mistake but this answer will help someone who is facing the same problem that Instead of FragmentPagerAdapter, use FragmentStatePagerAdapter.
As @BlackHatSamurai mentioned in the comment:
The reason this works is because
FragmentStatePagerAdapterdestroys
as Fragments that aren’t being used.FragmentPagerAdapterdoes not.