You cannot remove the new version but you can change its version number, as long as you’re in the Prepare for Submission state (ie. you might need to remove the build from waiting for review, also known as rejecting the binary, first). Here’s how.
To edit the version number, select the falsely numbered version on the left on iTunes Connect (so NOT the “App Information”, but under that the version “1.0.1 Prepare for Submission” etc), scroll down in the right pane to “General App Information” — just below the App Icon you can see the version number and it’s editable! Change it, hit save, and you’ll see the version number change on the left side.
Update: as @MDH points out, Apple finally got around to making it easier. You can now just directly edit the version number on the main “page” of the unreleased version. Amazing.