Flutter is an SDK that you download and unpack onto a directory in your Mac. There is no automatic uninstall process in the same way that there is no automatic install process. You “installed” it by downloading a zip file and unzipping it. All you have to do is remove the contents of the directory where you unzipped it in.
Even the path addition to be able to call the flutter command from anywhere in your system has to be done manually. If you did that, then you can remove it as well from your shell’s PATH.