If you use a recent version of Pandoc (latest is v2.7.3), just add (for example) these to the commandline:
-V colorlinks=true \
-V linkcolor=blue \
-V urlcolor=red \
-V toccolor=gray
No need to fiddle with the preamble!
You can investigate the built-in LaTeX template shipping with your own Pandoc by this command:
pandoc -D latex | less
Then search for link, url, color etc. to see which variables are pre-defined there. Or:
pandoc -D latex | grep --color -E '(links|color|url|file)'
will give you an idea what other “links” you may want to manipulate.