TL;DR: Just solved this issue by making sure that both tensorflow and tensorflow-estimator were in the same version. (in my case, I needed to downgrade tensorflow-estimator, so conda install tensorflow-estimator=2.1.0 solved it for me)
As you may have noticed, some tensorflow versions do not play well with certain GPUs, so I would first check some of the available builds with conda search tensorflow; then I would make sure that the chosen tensorflow build can actually recognize my GPU (with tf.config.list_physical_devices('GPU')); finally, I would search for a matching tensorflow-estimator build with conda search tensorflow-estimator and only then install it with conda install tensorflow-estimator=<chosen version> -n <my_venv>.
It should be noted, however, that all these steps are mostly useful if you have interest in using your GPU. If that is not the case, than upgrading both packages (or downgrading/upgrading them so their versions match) may be the way.