Apparently, this feature is intentional. I found out that a correct way to remove all remote branches that have been deleted from Github is to run the following command.
git fetch --prune
Then restart visual studio to remove the branches from the command palette