It turned out that this error only occurred when trying to inspect the Value property of the Lazy<> in the Visual Studio debugger. Doing so appeared to create a deadlock because the accessing of Value then seemed to hang the thread for a long time until the InvalidOperationException finally occurred. I could never intercept the original Exception, so I couldn’t see the inner stacktrace.
I’m just chalking this up as a bug in Visual Studio or their implementation of Lazy<>.