My shop uses .handlebars, along with Require.js and Alex Sexton’s require-handlebars plug-in, and it all works without issue. The far more common suffix however, and the default one in that plug-in I just mentioned, is .hbs (presumably because .hbs is a 3-character extension not already taken by another file type).
You can for example use .hbs, .handlebars, or even a different extension for that matter, and it should work just fine with any sort of library (eg. Require) where the suffix could actually matter. There are no guarantees of course, but because there is no official extension library authors generally know better than to hard-code one.
I would caution against using .htm or .html for these files though … unless you have a really picky IDE. Most IDEs can be set to treat .hbs as if it were an HTML files, for syntax coloring and what not. If your’s can’t, then .htm might make sense. Otherwise I’d keep the file extension distinct, so that you can easily distinguish between the two types of files.