Generally speaking, Modernizr does three things.
- It adds classes indicating feature support, allowing you to apply different styling to elements depending on what features they support.
- It allows you to run feature detection to decide whether to run a script/run a polyfill or not.
- It injects html5shiv, which allows old browsers to understand HTML5 elements.
If you don’t use any of these features, then there’s no point in including Modernizr at all. If you only use it for the html5shiv then you could probably just include that instead to save a few bytes, although I doubt there’s a relevant size difference at all.
That said, you should never include feature detects in a modernizr build that you don’t use. That’s nothing but a waste.