You can’t do this in a very fine grained fashion but you can disable automatic symbol loading and then manually choose the symbols to load via the Modules window (Debug -> Windows -> Modules).
To Disable Automatic Symbol loading
- Tools -> Options -> Debugging -> Symbols
- Check “Search the above locations only when symbols are loaded manually”