You may need to remove unnecessary git objects such as dangling commits & blobs:
git fsck && git gc --prune=now
git-fsck : Verifies the connectivity and validity of the objects in the database
git-gc : Cleanup unnecessary files and optimize the local repository
You can refer here about dangling commits & blobs.