You need to use a filter-branch along with the –prune-empty option to remove any commits that no longer introduce new changes.
git filter-branch --index-filter 'git rm --cached --ignore-unmatch -rf dir1 dir2 dirN file1 file2 fileN' --prune-empty -f HEAD
After that, if you want to recover disk space you will need to delete all the original refs that filter branch saved, expire the reflog, and garbage collect.