A selector engine is used to query a page’s DOM for particular elements, based on some sort of query (usually CSS syntax or similar).
For example, this jQuery:
$('div')
Would search for and return all of the <div> elements on the page. It uses jQuery’s selector engine to do that.
Optimizing the selector engine is a big deal because almost every operation you perform with these frameworks is based on some sort of DOM query.