CSS is only scoped within the same document. An iframe is an entire document in its own right, and so a CSS rule that applies to the page that contains that iframe cannot apply to the page that’s within that iframe.
This means that as far as HTML and CSS are concerned, html is always :root (and therefore can never be :not(:root)).
Unless you are able to transfer this CSS from the containing page to the page within the iframe (using a script for example), I don’t believe there is a way using just CSS.