I don’t think you need to destroy your VM. You can make changes to VM configuration when the VM is not running and powered down.
Try using vagrant halt and then vagrant up.
It should make the modifications to the VM as specified in Vagrantfile.
and you could also do vagrant up --no-provision to avoid re-provisioning as you bring up the VM.
and vagrant reload [vm-name] --no-provision should do the same trick.