Explanation
The ‘backspace’ setting controls this behavior.
From the help page:
Influences the working of <BS>, <Del>, CTRL-W and CTRL-U in Insert
mode. This is a list of items, separated by commas. Each item allows
a way to backspace over something:
value effect
indent allow backspacing over autoindent
eol allow backspacing over line breaks (join lines)
start allow backspacing over the start of insert; CTRL-W and CTRL-U
stop once at the start of insert.
Changing backspace behavior
Try adding the following to your .vimrc:
set backspace=indent,eol,start " backspace over everything in insert mode
A short-hand version of the same command:
set backspace=2