Google Tag Manager is by definition a performance impediment. Its only purpose is to allow non-technical people to dump random garbage very important third-party scripts on a site. If you bypass GTM by inserting the external tags/scripts that you want on your site directly, it will improve performance immediately.
If you can’t do that, you can still optimize the way GTM is loaded – it’s not true that you can’t defer. This is the default GTM implementation:
<!-- Google Tag Manager -->
<script>(function(w,d,s,l,i){w[l]=w[l]||[];w[l].push({'gtm.start':
new Date().getTime(),event:'gtm.js'});var f=d.getElementsByTagName(s)[0],
j=d.createElement(s),dl=l!='dataLayer'?'&l="+l:"';j.async=true;j.src="https://www.googletagmanager.com/gtm.js?id="+i+dl;f.parentNode.insertBefore(j,f);
})(window,document,'script','dataLayer','GTM-XXXX');</script>
<!-- End Google Tag Manager -->
Now you can’t just put the defer attribute on the script tag since that doesn’t work on inline scripts. But if you look closely at what the script does, you’ll see that it just inserts another script element that loads gtm.js from Google servers. It’s even asynchronous by default, see this line j.async=true. If you change that to j.defer=true, gtm.js will be loaded deferred. Keep in mind that some scripts/tags inserted by GTM may not expect this, so it might break something in unexpected ways.
Besides that, there are a couple minor tweaks you can do, but you won’t get around the fact that GTM is a huge unnecessary library that will impede page performance. On a sliding scale from convenience to performance, GTM is all the way on the side of convenience. If you have to justify the poor performance – show your client the test result and tell them to pick one.