There is an icon which will let you open file itself without needing to right click:
It is also possible to change default behavior when left-clicking on the file name since version 1.30:
To change it, set in your settings.json:
"git.openDiffOnClick": false
If you set it, the logic will inverse. Now, clicking on a file name will open the file, and clicking on the “inline icon” will open the diff.
It is also possible to disable this inline icon completely with:
"git.showInlineOpenFileAction": false