If you want to clear a specific item (not the whole list) from the recently opened list you have to
- Press
Ctrl + SHIFT + P - Type
Remove From History(in earlier version it isRemove From Editor History) and press Enter - Choose/Find the file you want to clear
Source: link
Update 09.04.2020:
With the release of Visual Studio Code 1.44.0 you can easily remove a file using the UI. You see a closing x next to a file in this recently opened files view:
