Java 8 Stream Collecting Set

Use flatMap:

return map.values().stream().flatMap(Set::stream).collect(Collectors.toSet());

The flatMap flattens all of your sets into single stream.