According to this post in the Swift forums, the .swiftpm directory “is intended as a place for tools that operate on packages to store state or configuration files of their own that users are not expected to edit directly as source” – it is not exclusive to Xcode support. So to add it to .gitignore could potentially have a bigger impact than just ignoring Xcode files.
You could consider ignoring .swiftpm/xcode instead, but thats also not universally recommended, because Xcode stores shared schemes there, as was mentioned in this rejected pull request.
By default, Swift Package Manager adds xcuserdata/ to the .gitignore, which is the primary ignore you need for using SPM with Xcode.