The documentation says:
The default_server parameter, if present, will cause the server to become the default server for the specified address:port pair.
It’s also obvious, there can be only one default server.
And it is also says:
A listen directive can have several additional parameters specific to socket-related system calls. They can be specified in any listen directive, but only once for the given address:port pair.
So, you should remove default and deferred from one of the listen 80 directives. And same applies to ipv6only=on directive as well.