Try adding the $host variable in log_format:
log_format main '$http_x_forwarded_for - $remote_user [$time_local] "$host" "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" $request_time';
http://wiki.nginx.org/HttpCoreModule#.24host:
$host
This variable is equal to line Host in the header of request or name
of the server processing the request if the Host header is not
available.This variable may have a different value from $http_host in such
cases: 1) when the Host input header is absent or has an empty value,
$host equals to the value of server_name directive; 2) when the value
of Host contains port number, $host doesn’t include that port number.
$host’s value is always lowercase since 0.8.17.