This is because you are using the hyphen within other characters, so that grep understands it as a range, which happens to be invalid.
You are basically doing
grep "[\-']" file
This is interpreted by grep as you providing a range of characters to be checked on, like for example grep "[a-z]" file. But the range from \ to ' is invalid, hence the error.
And why the other one is working? You may be asking yourself. Because what you are doing is:
grep "['\-]" file
In this case you are looking for either the character ', \ or - in the file.
See another example of it, where I want to find characters a, - or 3 in a given string:
$ echo "23-2" | grep -o '[a-3]'
grep: Invalid range end
$ echo "23-2" | grep -o '[a3-]'
3
-
$ echo "23-2" | grep -o '[a3\-]'
3
-
So the underlying problem is that you are using an expression some character + - + another character within a [] block and it tries to be read as the range of characters between some character and another character.
How can you solve it?
If you want to match the character -, among others, just add it in the edges of the expression: as the first or last item.
From man grep:
Character Classes and Bracket Expressions
A bracket expression is a list of characters enclosed by [ and ]. It
matches any single character in that list; if the first character
of the list is the caret ^ then it matches any character not in
the list. For example, the regular expression [0123456789] matches
any single digit.Within a bracket expression, a range expression consists of two
characters separated by a hyphen. It matches any single character
that sorts between the two characters, inclusive, using the locale’s
collating sequence and character set. For example, in the default C
locale, [a-d] is equivalent to [abcd]. Many locales sort characters
in dictionary order, and in these locales [a-d] is typically
not equivalent to [abcd]; it might be equivalent to [aBbCcDd], for
example. To obtain the traditional interpretation of bracket
expressions, you can use the C locale by setting the LC_ALL
environment variable to the value C.Finally, certain named classes of characters are predefined within
bracket expressions, as follows. Their names are self explanatory,
and they are [:alnum:], [:alpha:], [:cntrl:], [:digit:],
[:graph:], [:lower:], [:print:], [:punct:], [:space:], [:upper:],
and [:xdigit:]. For example, [[:alnum:]] means the character class of
numbers and letters in the current locale. In the C locale and ASCII
character set encoding, this is the same as [0-9A-Za-z]. (Note
that the brackets in these class names are part of the symbolic
names, and must be included in addition to the brackets delimiting the
bracket expression.) Most meta-characters lose their special meaning
inside bracket expressions. To include a literal ] place it
first in the list. Similarly, to include a literal ^ place it
anywhere but first. Finally, to include a literal – place it
last.