First Regex
x = x.replace(/[{()}]/g, '');
y = y.replace(/[{()}]/g, '');
In your first regex /[{()}]/g the outer square brackets [] makes a character class, it will match one of the character specified inside of it. In this case the characters { ( ) }.
Outside of the /regexp/ you have the g (global) modifier meaning that your entire regular expression will match as many times as it can , and it will not just make to the first match.
Second regex
x = x.replace(/[\[\]']+/g, '');
y = y.replace(/[\[\]']+/g, '');
In your second regex /[\[\]']+/g the outer square brackets [] makes a character class, it will match one of the character specified inside of it. In this case the characters [ ] '.
Note that the square brackets appear scaped inside the [character class] as \[ \].
After it you have specified a + quantifier, it makes the preceding rule match one or more times in a row. Note that this is redundant, even if it works, this is not quite what you want.
Outside of the /regularexpression/ you have the g (global) modifier meaning that your entire regular expression will match as many times as it can , and it will not just make to the first match.
Suggested Solution
run1.onclick = function() {
//removes "(" and ")"
output1.innerHTML = input1.value.replace(/[()]/g, '');
}
run2.onclick = function() {
//removes (){}[]
output2.innerHTML = input2.value.replace(/[\])}[{(]/g, '');
}<p>Remove ()</p>
<input id="input1" type="text" value="(123) 1234-1234">
<input id="run1" type="button" value="run">
<span id="output1"></span>
<hr>
<p>Remove ()[]{}</p>
<input id="input2" type="text" value="Hello (this) is [] a {{test}}!">
<input id="run2" type="button" value="run">
<span id="output2"></span>