ASP.Net Core middleware cannot set status code on exception because “response has already started”

Since this is the top search result on Google, I might as well tell new comers how I came up with this error.
I was trying to use this answer by zipping files and downloading them (streaming) to the client. I returned return Ok() at the end of the actual controller action. I needed to return return new EmptyResult()