The instance URL of any GitLab install is basically the link to the GitLab you’re trying to connect to.
For example, if your project is hosted on gitlab.example.com/yourname/yourproject then for the instance URL enter https://gitlab.example.com.
Another example, if your project is hosted on gitlab.com/username/project then the instance URL is https://gitlab.com. Though note that in the VS Code extension specifically, gitlab.com is the default so you can leave it blank in this case.