How can I automate both E2E and unit tests with Yeoman & AngularJS?

As I mentioned in a comment to your question – PhantomJS saves a lot of hassle. That aside, I believe you can handle everything from within your Gruntfile and just continue to run grunt test to start the whole thing.

grunt-karma allows full customization of your karma options with some handy add-ons.

From the docs:

….

You can override any of the config file’s settings directly:

karma: {   
  unit: {
    configFile: 'karma.conf.js',
    runnerPort: 9999,
    singleRun: true,
    browsers: ['PhantomJS']   
  }
}

Sharing Configs

If you have multiple targets, it may be helpful to share common
configuration settings between them. Grunt-karma supports this by
using the options property:

karma: {
  options: {
    configFile: 'karma.conf.js',
    runnerPort: 9999,
    browsers: ['Chrome', 'Firefox']
  },
  continuous: {
    singleRun: true
    browsers: ['PhantomJS']
  },
  dev: {
    reporters: 'dots'
  }
}

Additionally you may want to snoop around in Yeoman’s generator-angular Gruntfile code to see what else may be available or at least mockable.