I believe the answer to your main question lies in the difference between Anaconda vs Miniconda. Anaconda includes a long list of packages that get installed automatically into each environment that you create. Miniconda creates barebone conda virtual environments (which don’t contain many packages at all). Switching to Miniconda should substantially reduce the size/number of packages in your environments. Anaconda is about 2GB, while Miniconda is closer to 100MB.
Conda also uses hardlinks for packages installed vs conda install. A good description of hardlinks can be found here. They basically link dependencies across multiple environments like you’ve described above. Packages installed via pip are not hardlinked, so they cannot take advantage of the space savings that conda packages offer.