As @adamesque stated, to exclude files from quick open you can either add them to files.exclude or search.exclude setting.
The issue then is that Quick Open also stores recent items, even from excluded paths.
Fortunately, VSCode now have a command File: Clear Recently Opened!
EDIT: As I needed to replace a Disk path while keeping my recent files I found where they are stored. It’s in the storage.json file which is under /Users/imac/Library/Application Support/Code/storage.json on macOS and AppData\Roaming\Code\storage.json on Windows.