It works, you just have to assign the actual result/return value. Use one of the following variations:
str = str:gsub("%s+", "")
str = string.gsub(str, "%s+", "")
I use %s+ as there’s no point in replacing an empty match (i.e. there’s no space). This just doesn’t make any sense, so I look for at least one space character (using the + quantifier).