I had the same problem and at the time for writing this answer, there is no option of removing this from Github. So removed this tag by deleting the repository from Github and recreating a new one with the same name.
⚠️ This operation will preserve only the information stored in git, not all other Github features, i.e. issues, PRs, … If your repository has been used for a while, this might not be the right time to do this anymore.
Specific instructions:
- Make sure you have cloned your repository and all your non-stale branches are saved locally.
git fetch - Delete the original repository in the Github web UI.
- Create a new repository with the same name.
- Push the branches you want back to the remote.
git push