Possible in theory (along with unused public members), but not with the kind of compiler ecosystem we’re used to (targeting a fixed ABI that can link separately-compiled code). Removing unused members could only be done with whole-program optimization that forbids separate libraries1.
Other compilation units might need to agree on sizeof(foo), but that wouldn’t be something you could derive from a .h if it depended on verifying that no implementation of a member function’s behaviour depended on any private members.
Remember C++ only really specifies one program, not a way to do libraries. The language ISO C++ specifies is compatible with the style of implementation we’re used to (of course), but implementations that take all the .cpp and .h files at once and produce a single self-contained non-extensible executable are possible.
If you constrain the implementation enough (no fixed ABI), aggressive whole-program application of the as-if rule becomes possible.
Footnote 1: I was going to add “or exports the size information somehow to other code being compiled” as a way to allow libraries, if the compiler could already see definitions for every member function declared in the class. But @PasserBy’s answer points out that a separately-compiled library could be the thing that used the declared private members in ways that ultimately produce externally-visible side effects (like I/O). So we’d have to fully rule them out.
Given that, public and private members are equivalent for the purposes of such an optimization.