According to the statement here, you don’t need to remove connections from groups:
You should not manually remove the user from the group when the user
disconnects. This action is automatically performed by the SignalR
framework.
When a connection subscribes to a topic (which happens when you add the connection to a group), it receives a disposable which will remove the subscription when disposed (which means the connection isn’t in the group anymore). This is triggered when a connection disconnects and is removed.