In short, just debug jasmine-node:
node --debug-brk node_modules/jasmine-node/lib/jasmine-node/cli.js spec/my_spec.js
If you look at the source of the jasmine-node script, it just invokes cli.js, and I found I could debug that script just fine.
I wanted to use node-inspector to debug a CoffeeScript test. Just adding the --coffee switch worked nicely, e.g.
node --debug-brk node_modules/jasmine-node/lib/jasmine-node/cli.js --coffee spec/my_spec.coffee