They do different things.
Modernizr detects the availability of features in a page allowing you to provide your own polyfills for older browsers should you require that functionality. You can add support for <canvas> using a canvas tag polyfill so that canvas functionality, including it’s JavaScript interface, in browsers that don’t support the <canvas> tag.
Html5shiv adds the new html5 tags that aren’t available (<section>, <header>, etc) to older browsers. It also creates the default styles (display: block for <section> for example).
That’s it. It provides no other functionality.