None of the previous comments address the most important difference between the two:
KeyedCollection keeps your items in the order in which they are added (the first item added is at index 0 and the last added is at the last index). Dictionary does not (or at least it is never guaranteed to do so).
This extra benefit of KeyedCollection does have a small performance cost. Under the covers, you pay the cost of maintaining both a Dictionary and a List.