TL;DR kill the app and run from the launcher, icon does not show up when run from Android Studio.
Adding my comment here as an answer for better visibility.
I did figure out how to get it to show. I was following this tutorial to set up a base project to recreate the issue and I noticed the note the author put right near the bottom that mentions that just running the app doesn’t show the full splash screen. You have to kill it and open the app from the launcher. Once I did that, I was able to see my splash screen. Annoying, but at least I have a way to test it now. I did go ahead and log a bug report for this as well, but I have a work around for now. Thanks for everyone’s answers/comments!