The set will give much better performance (O(n) vs O(n^2) for the list), and that’s normal because set membership (the contains operation) is the very purpose of a set.
Contains for a HashSet is O(1) compared to O(n) for a list, therefore you should never use a list if you often need to run contains.