I believe Lunivore’s answer was correct at the time it was written.
In newer versions of Moq (I think since version 4.1 from 2013) it is possible to do what you want with the exact syntax you propose. That is:
mock.Setup(m => m.VirtualMethod()).CallBase();
This sets up the loose mock to call the base implementation of VirtualMethod instead of just returning default(WhatEver), but for this member (VirtualMethod) only.
As user BornToCode notes in the comments, this will not work if the method has return type void. When the VirtualMethod is non-void, the Setup call gives a Moq.Language.Flow.ISetup<TMock, TResult> which inherits the CallBase() method from Moq.Language.Flow.IReturns<TMock, TResult>. But when the method is void, we get a Moq.Language.Flow.ISetup<TMock> instead which lacks the desired CallBase() method.
Update: andrew.rockwell notes below that it works now for void methods, and apparently that was fixed in version 4.10 (from 2018).