How does the WPF Button.IsCancel property work?

Yes, it only works on dialogs as a normal window has no concept of “cancelling”, it’s the same as DialogResult.Cancel returning from ShowDialog in WinForms.

If you wanted to close a Window with escape you could add a handler to PreviewKeyDown on the window, pickup on whether it is Key.Escape and close the form:

public MainWindow()
{
    InitializeComponent();

    this.PreviewKeyDown += new KeyEventHandler(CloseOnEscape);
}

private void CloseOnEscape(object sender, KeyEventArgs e)
{
    if (e.Key == Key.Escape)
        Close();
}