Found it https://go.dev/blog/using-go-modules#removing-unused-dependencies
go mod tidy
So basically, once the package is not being imported in any package you can perform a go mod tidy and it will safely remove the unused dependencies.
And if you are vendoring the dependencies, then run the command below to make the module changes be applied in the vendor folder:
go mod vendor