This is what I have come up so far (that also makes the borrow checker happy):
let index = xs.iter().position(|x| *x == some_x).unwrap();
xs.remove(index);
I’m still waiting to find a better way to do this as this is pretty ugly.
Note: my code assumes the element does exist (hence the .unwrap()).