^(\d{3}|\d{6})$
You have to have some sort of terminator otherwise \d{3} will match 1234. That’s why I put ^ and $ above. One alternative is to use lookarounds:
(?<!\d)(\d{3}|\d{6})(?!\d)
to make sure it’s not preceded by or followed by a digit (in this case). More in Lookahead and Lookbehind Zero-Width Assertions.