If you provide a custom segue (subclass of NSStoryboardSegue) you can get the result you are after. There are a few gotchas with this approach though:
- the custom segue will use
presentViewController:animatorso you will need to provide an animator object - because the presented view is not backed by a separate Window object, you may need to provide it with a custom NSView just to catch out mouse events that you don’t want to propagate to the underlying NSViewController’s view
- there’s also a Swift-only glitch regarding the custom segue’s
identifierproperty you need to watch out for.
As there doesn’t seem to be much documentation about this I have made a small demo project with custom segue examples in Swift and Objective-C.

I also have provided some more detail in answer to this question.