$? is the exit status of the most recently-executed command; by convention, 0 means success and anything else indicates failure. That line is testing whether the grep command succeeded.
The grep manpage states:
The exit status is 0 if selected lines are found, and 1 if not found. If an error occurred the exit status is 2. (Note: POSIX error handling code should check for ‘2’ or greater.)
So in this case it’s checking whether any ERROR lines were found.