Just like you said: refactor early, refactor often.
Refactoring early means the necessary changes are still fresh on my mind. Refactoring often means the changes tend to be smaller.
Delaying refactoring only ends up making a big mess which further makes it harder to refactor. Cleaning up as soon as I notice the mess prevents it from building up and becoming a problem later.