There really aren’t any differences.
" is processed as " which is the decimal equivalent of &x22; which is the ISO 8859-1 equivalent of ".
The only reason you may be against using " is because it was mistakenly omitted from the HTML 3.2 specification.
Otherwise it all boils down to personal preference.