Use this form:
=(B0+4)/$A$0
The $ tells excel not to adjust that address while pasting the formula into new cells.
Since you are dragging across rows, you really only need to freeze the row part:
=(B0+4)/A$0
Keyboard Shortcuts
Commenters helpfully pointed out that you can toggle relative addressing for a formula in the currently selected cells with these keyboard shortcuts:
- Windows: f4
- Mac: CommandT