How to filter out files by extension in NERDTree?

You want the NERDTreeIgnore option. For example, in your .vimrc:

let NERDTreeIgnore = ['\.pyc$']

Where NERDTreeIgnore is an array of regular expressions that match the files you want to exclude.