HashSet consumes about 5.5 times more memory than ArrayList for the same number of elements (although they’re both still linear), and has significantly slower iteration (albeit with the same asymptotics); a quick Google search suggests a 2-3x slowdown for HashSet iteration versus ArrayList.
If you don’t care about uniqueness or the performance of contains, then use ArrayList.