Object initializers should be kept for simple object initialization. If you are in the point where your object constructor has code that may fail (e.g. throwing an exception), don’t use it. Better rely on an object construction pattern, which depending on your needs may be a factory method, an abstract factory,etc…
This also ensures that all the users of your class cannot build an instance that is in an invalid state (e.g. they forget to initialize a member, or they initialize related members with incorrect values, etc…)