how to disable direct access to a web site by ip address

server {
    listen      80 default_server;
    listen      [::]:80 default_server;
    server_name "";
    return      444;
}

You need to specify default_server parameter so that all non available server requests goes to this server block which throws 444 error.

444 : CONNECTION CLOSED WITHOUT RESPONSE

ref: https://httpstatuses.com/444