The wixui extension is no longer wixui.wixlib. It was rolled into the WixUIExtension.
If you are on the command line, add -ext WixUIExtension to the call to light.exe.
If you have a WiX project in Visual Studio, add a reference to “WixUIExtension” by right clicking on “References” in the Solution Explorer.
See the WiXUI Dialog Sets page for WIX3.