You should be pointing it towards the Developer directory, not the Xcode application bundle. Run this:
sudo xcode-select --switch /Applications/Xcode.app/Contents/Developer
With recent versions of Xcode, you can go to Xcode ➙ Preferences… ➙ Locations and pick one of the options for Command Line Tools to set the location.