Add does nothing (returns false) if there is already a value for that key. Set does an insert or update, as necessary.
Remove + Add would leave a gap in the middle when another thread querying that key would get no clue (Set does not; the swap is typically atomic); as such, while Set has the same end result as Remove + Add, the mechanism difference is important since it could impact other callers.
For example of Add:
Return Value
Type: System.Boolean true if insertion succeeded, or false if there is an already an entry in the cache that has the same key as key.