I would say it depends on whether all the values with same key have a relationship that you want to address.
So for example, do you often go through all elements with key X, or pass them to a function, and so on? Then it is more convenient to already have them in their separate container, that you can address directly.
However, if you just have a collection of items, that may share same key value, or not, why use vectors in between? It is more convenient to run through the multimap with iterators than have a nested for-loop for the map, vector case.
Another way of looking at this: If multiple entries per key is very common, your structure is more efficient in the map, vector case. If they seldomly happen, it is the opposite.