You should avoid worrying about the performance implications of specific language features unless you have specific evidence (measurements) that they are actually causing a problem.
Your primary concerns should be the correctness of the code and it’s maintainability.
As a general observation, however, unnecessary casting can often be avoided in C# just by applying good OO programming practices and using generics (particularly the collections) appropriately. In those cases where you do need to perform casting, it’s highly unlikely to be a performance bottleneck unless you’re doing it in a tight loop or with types that are likely to throw an invalid cast exception.
Most real world performance problems emerge from algorithm choices or a lack of awareness of the platform itself – not from specific language features.