The answer provided by Nikos Baxevanis provides various convention-based ways to answer the question. For completeness sake, you can also do a more ad-hoc build:
var phoneBook = fixture.Build<PhoneBook>().Without(p => p.AllContacts).Create();
If you want your Fixture instance to always do this, you can Customize it:
fixture.Customize<PhoneBook>(c => c.Without(p => p.AllContacts));
Every time that Fixture instance creates an instance of PhoneBook, it’ll skip the AllContacts property, which means that you can go:
var sut = fixture.Create<OfficeBuilding>();
and the AllContacts property will remain untouched.