My experience is in line with yours, so I use S3 exclusively.
To clarify: S4 has some slick features (e.g. dispatch on multiple arguments and slot type-checking), but I have not encountered a situation where the features outweighed the costs. Examples of the costs include: any slot change requires a full object copy and (potentially worse) the on-going changes to S4 Methods.
In short, I like the idea behind S4 but I would wait for it to mature before using it in my own code.