It fails because you are creating an extra capturing group, meaning that the capturing group indexes will not be the same as before.
To make the s optionnal without creating a capturing group, you can simply add ?, you do not need the parenthesis.
/https?:\/\/(www\.)?vimeo.com\/(\d+)($|\/)/
To create a non-capturing group, you can use (?:), but that’s not necessary here, just showing for the example:
/http(?:s)?:\/\/(www\.)?vimeo.com\/(\d+)($|\/)/