Including or linking against large libraries usually won’t make a difference unless you use that stuff. Linkers should perform dead code elimination and thus ensure that at build time you won’t be getting large binaries with a lot of unused code (read your compiler/linker manual to find out more, this isn’t enforced by the C++ standard).
Including lots of headers won’t increase your binary size either (but it might substantially increase your compilation time, cfr. precompiled headers). Some exceptions stand for global objects and dynamic libraries (those can’t be stripped). I also recommend to read this passage (gcc only) regarding separating code into multiple sections.
One last notice about performances: if you use a lot of position dependent code (i.e. code that can’t just map to any address with relative offsets but needs some ‘hotpatching’ via a relocation or similar table) then there will be a startup cost.