Unable to remove files recursively from Git

 git rm -r --cached ~/.vim/*   
 fatal: pathspec '.vim/colors' did not match any files

1/ You do not need the ‘*‘:

 git rm -r --cached ~/.vim

will take care of any tracked sub-files.

2/ fatal: pathspec '.vim/colors' did not match any files simply means one of your commands you tried before the one listed in 1/ has worked, and there is no more file to delete!

# to test that command, first reinitialize the state of the repository
# save first if you have any other current modifications
$ git reset --hard

# then check the rm works
$ git rm -r --cached ~/.vim
rm '.vim/aPath/aFile1'
rm '.vim/aSecondPath/aFile2'
rm '.vim/aThirdPath/aFile3'

# try it again
$ git rm -r --cached ~/.vim
fatal: pathspec '.vim/colors