Open /etc/vim/vimrc.tiny with vi and enter the following two lines:
set nocompatible
set backspace=2