First, it’s better to remove VSTS agent through config.cmd remove command and the PAT is required, you don’t need to use original PAT, you can apply a new PAT with Agent Pools (read, manage) scope and use it to remove agent.
Secondly, without PAT:
Deleting agent from server:
Deleting agent service in local system through sc command if it is running as service: sc delete [service name].
After that, you can delete the agent files.