You use std::addressof when you have to. Sadly, “when you have to” includes anytime you are working in template code and want to turn a variable of unknown type T or T& into an honest-to-God pointer to that variable’s memory.
Because the C++ committee foolishly allowed the overloading of the reference operator (to little legitimate purpose), it is possible for a user to instantiate your template with some type that you can’t use the reference operator to get an actual pointer to. std::addressof is a way to work around users who use this dubious C++ feature in order to do what the language should have guaranteed to work to begin with.
In short, it’s a library fix for a language stupidity. Use it in template code instead of & if you want to make sure users can’t break your code. If your users can be trusted not to use this ill-conceived feature, then you can use &.