Some MVVM frameworks (e.g. MVVM Light) make use of the Mediator pattern.
So to open a new Window (or create any View) some View-specific code will subscribe to messages from the mediator and the ViewModel will send those messages.
Like this:
Subsription
Messenger.Default.Register<DialogMessage>(this, ProcessDialogMessage);
...
private void ProcessDialogMessage(DialogMessage message)
{
// Instantiate new view depending on the message details
}
In ViewModel
Messenger.Default.Send(new DialogMessage(...));
I prefer to do the subscription in a singleton class, which “lives” as long as the UI part of the application does.
To sum up: ViewModel passes messages like “I need to create a view” and the UI listens to those messages and acts on them.
There’s no “ideal” approach though, for sure.