I recently needed to do the same thing, and this is how I did it.

1. I used the npm module html-loader, instead of es6-template-string-loader

2. Add to webpack.config.js

Webpack 3

...
module: {
    rules: [
        {
            test: /\.html$/,
            exclude: /node_modules/,
            use: {loader: 'html-loader'}
        }
    ]
}
...

Webpack 1 (deprecated, but from original answer):

...
module: {
    loaders: [
        {
            test: /\.html$/,
            loader: "html-loader"
        }
    ]
}
...

3. Use in your JS files

import template from './header.html';