I had the same problem! Tried all the scroll-* settings, didn’t help when holding down arrow. But found this on gnu.emacs.help which finally worked (for me at least):
(setq redisplay-dont-pause t)
This is what I have in .emacs for now:
(setq redisplay-dont-pause t
scroll-margin 1
scroll-step 1
scroll-conservatively 10000
scroll-preserve-screen-position 1)