Since poetry 1.2, you can do this:
poetry source add -p explicit pytorch https://download.pytorch.org/whl/cpu
poetry add --source pytorch torch torchvision
and it will install from the specified index-url.
(and this also works with https://download.pytorch.org/whl/cu118 )