ES5-Shim will only shim parts that the browsers don’t implement, so just give it to all browsers. It’ll handle the detection of what needs to be shimmed and what doesn’t.
But pay attention to the caveats listed on what shims don’t work correctly in some instances. I’ve had issues with that in the past and it causes a ton of pain until you realize the answer was super simple…