(Annotated) tags won’t make Git sluggish, only harder to use in that you will have an harder time to find relevant tags amongst all the intermediate ones.
Worst case, it can make your web interface (like GitHub) sluggish if it has to display thousands of tags.
Maybe all those tags don’t need to be pushed, in which case you could consider making intermediate local “Lightweight” tags.