As others have indicated in the comments, undoing changes from the main branch (presumably the SOURCE branch for the shelveset you want to unshelve), should fix the issue, allowing you to continue with the merge behavior.
I am not sure if this would help or you are willing to use a new version but I had the exact same environment and issue with you. What I had done is that I uninstalled the old TFS Power Tools ( listed with a “Microsoft Team Foundation Server” prefix and/or “Microsoft Visual Studio Team … Read more
I ran into this same issue, I found the answer at the bottom of the page in one of the help files that came with The power tools. (TFPTCommandLineTool.mht) Errors TFPT Error: Unable to determine the workspace When running tfpt using a command that works with Version Control, you may receive one of the errors: … Read more