I know an answer has already been accepted, but this questions is a top hit on Google and OPs question is about class libraries and not an ASP.NET Web App or a WebApi which is what the accepted answer uses.
IMO, class libraries should not use application settings and should be agnostic to such settings. If you need application settings in your class library, then you should provide those settings from your consumer. You can see an example of this On This SO Question