Update: Check this announcement: https://github.com/NuGet/Announcements/issues/49
At the moment the issue appears to be related to the Debian image.
Switch to an Ubuntu or Alpine based image instead:
FROM mcr.microsoft.com/dotnet/sdk:5.0-focal AS build-env
Follow https://github.com/NuGet/Home/issues/10491 for updates.