The source_files attribute uses Ruby file glob syntax. The pattern must be relative to the root of your project (i.e., the podspec file), so this should work for you:
s.source_files="Library/src/**/*.{h,m}"
The source attribute is not for source code files, but rather for the remote repository from which the code should be retrieved (most commonly a Git repository URL and tag). See the CocoaPods specification docs for more info.