You can check the value of $- to see the current options; if it contains an x, it was set. You can check like so:
old_setting=${-//[^x]/}
...
if [[ -n "$old_setting" ]]; then set -x; else set +x; fi
In case it’s not familiar to you: the ${} above is a Bash Substring Replacement, which takes the variable - and replaces anything that’s not an x with nothing, leaving just the x behind (or nothing, if there was no x)