For GNU xargs, you can use the -r or --no-run-if-empty option:
--no-run-if-empty
-r
If the standard input does not contain any nonblanks, do not run the command. Normally, the command is run once even if there is no input. This option is a GNU extension.
The BSD version of xargs, which is used on macOS, does not run the command for an empty input, so the -r option is not needed (nor is it accepted by that version of xargs).