From the anaconda docs, you can uninstall with conda uninstall
Try
conda uninstall pytorch torchvision cuda80 -c soumith
Alternatively, the pytorch docs suggest
conda uninstall pytorch
pip uninstall torch
pip uninstall torch # run this command twice