Add them to the allowed characters, but you’ll need to escape some of them, such as -]/\
var pattern = /^[a-zA-Z0-9!@#$%^&*()_+\-=\[\]{};':"\\|,.<>\/?]*$/
That way you can remove any individual character you want to disallow.
Also, you want to include the start and end of string placemarkers ^ and $
Update:
As elclanrs understood (and the rest of us didn’t, initially), the only special characters needing to be allowed in the pattern are &-._
/^[\w&.\-]+$/
[\w] is the same as [a-zA-Z0-9_]
Though the dash doesn’t need escaping when it’s at the start or end of the list, I prefer to do it in case other characters are added. Additionally, the + means you need at least one of the listed characters. If zero is ok (ie an empty value), then replace it with a * instead:
/^[\w&.\-]*$/