As others have already pointed out, valueFactory may be invoked more than once. There is a common solution that mitigates this issue – have your valueFactory return a Lazy<T> instance. Although it’s possible that multiple lazy instances will be created, the actual T value will only be created when you access Lazy<T>.Value property.
Specifically:
// Lazy instance may be created multiple times, but only one will actually be used.
// GetObjectFromRemoteServer will not be called here.
var lazyObject = dict.GetOrAdd("key", key => new Lazy<MyObject>(() => GetObjectFromRemoteServer()));
// Only here GetObjectFromRemoteServer() will be called.
// The next calls will not go to the server
var myObject = lazyObject.Value;
This method is further explained in Reed Copsey’s blog post