I think the one major downside you will find is that the ASGI servers are newer and therefore tested less, may have less features, fewer in number, and probably have a smaller community behind them. However, I use an ASGI server (Daphne) for everything and feel that websockets offer so much in terms of user experience that everything will eventually shift to ASGI.
Being able to use asyncio in your code is a major benefit for web programming. Instead of running 10 queries one after the other and waiting for each one to come back, you can run 10 queries at the same time, while hitting your cache and making a HTTP request simultaneously on a single thread.