So this has to do with the way caching works in Chrome. I don’t have personal experience with Chrome’s codebase, but I do know a bit about the theory of it. (I also found a reference to Chrome’s cache implementation here for the more curious: chromium disk cache)
For reference, here is a screenshot of my loading your actual Stack Overflow question in Chrome with the Network panel open and the Network Throttling option set to “Offline”. Notice that every single entry in this list is got from cache!
You’ll also notice that Chrome is spending time “downloading” every file. Why is this? Well, Chrome’s cache is a database, and that database is also compressed to save space. When you retrieve a document from cache, the price of that retrieval is not zero. Chrome has to look up the item in the cache database, and then inflate that entry into memory so Chrome can work with it. I don’t know the exact details concerning how the Network chrome-dev-tools panel shows the times, but I would guess that getting that file from disk, uncompressing it, and then parsing and working with the result is what you’re seeing reflected in “Time downloaded.”
I can’t comment on why other browsers don’t also have this delay, since I don’t have a lot of experience with them. It might be that they either use a more efficient method of getting things from cache (possible), or it could be that they keep the cache in memory at all times (unlikely), or that they’re skipping some of the integrity checks Chrome is making on cache data (possible)