How to deactivate a LVM2 physical volume to remove the drive?

lvchange -an <lvpath>
vgchange -an <vgname>

If you also want to remove the device maps you can use

dmsetup ls
dmsetup remove <name>