I am not an expert, but having the same problem here what I did in my nginx.conf :
log_format upstreamlog '[$time_local] $remote_addr - $remote_user - $server_name to: $upstream_addr: $request upstream_response_time $upstream_response_time msec $msec request_time $request_time';
access_log /var/log/nginx/access.log upstreamlog;
The first line define a new format, and then you tell access_log to use it