UPDATE ANGULAR 12
using the --configuration production
option while building the library fixed my issue
ng build --configuration production
Original answer:
using --prod
option while building the library fixed my issue
ng build yourLibraryName --prod