I found the main answer inaccurate so I am responding here.
Obviously Microsoft has made an absolute mess out of Channles and Factories and Clients. Documentation is not also helpful since they seem to be there just to cover up the mess so I had to resort to testing.
With the performance issues regarding non-cached Channels, implementation changed in v3.5 to address these and added caching but that only complicated the issue.
The point is channel in ChannelFactory is in fact not different from the channel used by IClientChannel when you create a channel using ChannelFactory.CreateChannel(). It is all the same pot. Believe me not? Try:
ChannelFactory<IService> factory = new ChannelFactory<IService>();
// ...
IService service = factory.CreateChannel();
factory.Close();
service.DoIt() // Throws object disposed exception
So really, internally it is all the same channel. I personally have started disposing the channel factories and not client channels and have not encountered any issue. I have also tried doing this in a loop with creation of 100000 client channels, and only closing ChannelFactory.