listen_addresses controls which IPs the server will answer on, not which IPs the server will permit connections to authenticate from. It’s entirely reasonable and normal to use listen_addresses '*' so the server will accept incoming connections on any ip assigned to an interface on the postgresql server host, while using pg_hba.conf to control access at a finer grained level for which IPs the server will accept logins from for specific databases and users.