I think your first hunch is correct. If(!) you can’t store everything in the store, you have to store less in the store. But I believe I can make that solution sound much better:
IndexedDB just becomes another endpoint, much like any server API you consume. When you fetch data from the server, you forward it to IndexedDB, from where your store is then populated. The store gets just what it needs and caches it as long as it doesn’t get too big or stale.
It’s really not different than, say, Facebook consuming their API. There’s never all the data for a user in the store. References are implemented with IDs and these are loaded when required.
You can keep all your logic in redux. Just create actions as usual for user actions and data changes, get the data you need and process it. The interface is still completely defined by the user data because you always have the information in the store that is needed to GET TO the rest of it when needed. It’s just somewhat condensed, i. e. you only save the total number of messages or the IDs of a mailbox until the user navigates to it.