I’d be pretty nervous about that, actually; what if you want to update an image, then change your mind? Or what if you need to build a maintenance release with old images?
If this is really a problem — and I’ve never seen this actually be a problem in practice, but I’ll take your word for it — why not just use one repo for the images, another for everything else? You can then just be lazy about syncing the image one.