The problem could be that your existing PowerShell session is “locking” the module by loading possible elements from it (such as global variables or constants) even though you are trying to unload it (Remove-Module).
The cleanest way to be sure it isn’t locked is to exit the PowerShell session. If you need to keep the session around to do “stuff” afterwards, trying starting a new PowerShell session (nested session) just before you make use of the module, then exit it at the end.