It’s a good assumption to start with. However, the optimizer may be restricted in what it can assume in the presence of a reinterpret_cast<> or C pointer cast. Then, even though the cast itself has no associated instructions, the resulting code is slower.
For instance, if you cast an int to a pointer, the optimizer likely will have no idea what that pointer could be pointing to. As a result, it probably has to assume that a write through that pointer can change any variable. That beats very common optimizations such as storing variables in registers.