Removing the installed executable with rm is the right way to go.
In Go, go install builds a single-file binary and “installs” it by copying it to the appropriate directory (*). To “uninstall” this binary, simply remove it with rm.
It may feel “blunt force” to you, but it’s actually reassuring if you think about it. There’s little magic involved. Installation means a single binary gets placed in some directory (which is likely in your $PATH).
See also this answer for a relevant discussion of removing packages installed with go get
(*) From go help install:
Executables are installed in the directory named by the GOBIN
environment variable, which defaults to $GOPATH/bin or $HOME/go/bin if
the GOPATH environment variable is not set. Executables in $GOROOT are
installed in $GOROOT/bin or $GOTOOLDIR instead of $GOBIN.