I ended up doing this:
- Create a
submodulesdirectory. - Add the submodule in this directory.
- Create a symlink to the specific directory inside the submodule.
This way you have default Git submodule behaviour and in your project you only use a subset of the whole submodule.