Is Swift really slow at dealing with numbers?

Here are optimization levels for the Swift compiler’s code generation (you can find them in Build Settings):

[-Onone] no optimizations, the default for debug.
[-O]     perform optimizations, the default for release.
[-Ofast] perform optimizations and disable runtime overflow checks and runtime type checks.

Using your code I got these times at different levels of optimization:

[-Onone]

Swift: 6110.98903417587ms
Objc:  134.006023406982ms

[-O]

Swift: 89.8249745368958ms
Objc:  85.5680108070374ms

[-Ofast]

Swift: 77.1470069885254ms
Objc:  76.3399600982666ms

Keep in mind that -Ofast is comes with risks. e.g. It will silently ignore integer and array overflows, producing nonsense results, so if you choose to use it you’ll have to guarantee yourself that overflows aren’t possible in your program.