The C# implementation is recalculating Math.Sqrt(suspectPrime) each time through the loop, while VB only calculates it at the beginning of the loop. This is just due to the nature of the control structure. In C#, for is just a fancy while loop, while in VB it’s a separate construct.
Using this loop will even up the score:
Int32 sqrt = (int)Math.Sqrt(suspectPrime)
for (Int32 i = 2; i <= sqrt; i++) {
if (suspectPrime % i == 0)
return;
}