v.which() will return the 0-based index of the type of the object currently held.
When you are retrieving the object your code must use a static type (in order to satisfy the get<T> function template) to refer to an (effectively) dynamically typed object.
You need to either test for the type (using which() or type()) and branch accordingly or use a static visitor. No matter which way you choose, you have to explicitly state the static type that you want to retrieve and it has to match the dynamic type or an exception will be thrown.
One way around this problem is instead of using a variant type directly, use a class which contains a variant type internally and then defines any implicit conversion operators necessary to use the object with minimum fuss.
I have a project called Dynamic C++ which uses this technique.