I figured out what was causing this problem. I need to use different AlertDialog class:
androidx.appcompat.app.AlertDialog
When I switched to this everything started working as expected. Here’s where I found the solution:
https://github.com/material-components/material-components-android/issues/162