I needed to use onSelectionChange on the <mat-option>, which is different than the selectionChange that you can use on the <mat-select>
It would be nice if that was in the documentation for mat-select.
Here it is working https://stackblitz.com/edit/angular-1e9gsd-34hrwg?file=app/select-overview-example.html