You can use

git filter-branch --index-filter 'git rm -r --cached --ignore-unmatch <file/dir>' HEAD

This will delete everything in the history of that file. The problem is that the file is present in the history.

This command changes the hashes of your commits which can be a real problem, especially on shared repositories. It should not be performed without understanding the consequences.

Edit: The git project now recommends that users use git filter-repo instead of git filter-branch.

Using git filter-repo

WARNING: git-filter-branch has a glut of gotchas generating mangled history
         rewrites.  Hit Ctrl-C before proceeding to abort, then use an
         alternative filtering tool such as 'git filter-repo'
         (https://github.com/newren/git-filter-repo/) instead.  See the
         filter-branch manual page for more details; to squelch this warning,
         set FILTER_BRANCH_SQUELCH_WARNING=1.

Installation

[brew|pip3|...] install git-filter-repo

Usage

To remove any file with the path prefix example/path/to/something, you can run

git filter-repo --path example/path/to/something--invert-paths

To remove any file without the path prefix example/path/to/something, you can run

git filter-repo --path example/path/to/something