You are the only person who can answer the “performance hit” question. Try it both ways, measure the performance, and see what happens. The hit could be enormous or it could be nonexistant; no one reading this knows whether “enormous” to you means one microsecond or twenty minutes.
If you’re interested in what optimizations are done by the C# compiler — rather than the jitter — when the optimize switch is on, see:
http://blogs.msdn.com/ericlippert/archive/2009/06/11/what-does-the-optimize-switch-do.aspx