You can disable the backup file by putting this line in your .vimrc:
set nobackup
I almost always do this, as the ~ file is more annoying that useful. There is no problem with doing this, you’ll just lose the ability to revert to a backup of the file.
If you want to get rid of the temporary .swp (swap) file too, you can also set this:
set noswapfile
The swap file is created when you have a file open, and provides some backup/recovery security, in case Vim crashes while editing a file. It also can prevent multiple Vims from editing the same file. I usually just turn this off too, because I rarely have a use for it. The .swp file isn’t as annoying as the ~ file, because it goes away when you close Vim, but I still just turn that feature off.