Found the solution after some searching.
You need to add a <meta> tag in your <head> containing name="theme-color", with your HEX code as the content value. For example:
<meta name="theme-color" content="#999999" />
Update:
If the android device has native dark-mode enabled, then this meta tag is ignored.
Chrome for Android does not use the color on devices with native
dark-modeenabled.
source: https://caniuse.com/#search=theme-color