So it seems like pretty common issue. The point is to remove mex from your bindings:
<endpoint address="mex" binding="mexHttpBinding" name="mex" contract="IMetadataExchange"></endpoint>
Alternativelly you enable Anonymous access in IIS and in your web.config you make sure anonymous access is denied.
Hope this will help some other soul.
(I was 100% sure I tried it with mex removed. :-O )