Extreme Sharding: One SQLite Database Per User

The place where this will fail is if you have to do what’s called “shard walking” – which is finding out all the data across a bunch of different users. That particular kind of “query” will have to be done programmatically, asking each of the SQLite databases in turn – and will very likely be the slowest aspect of your site. It’s a common issue in any system where data has been “sharded” into separate databases.

If all the of the data is self-contained to the user, then this should scale pretty well – the key to making this an effective design is to know how the data is likely going to be used and if data from one person will be interacting with data from another (in your context).

You may also need to watch out for file system resources – SQLite is great, awesome, fast, etc – but you do get some caching and writing benefits when using a “standard database” (i.e. MySQL, PostgreSQL, etc) because of how they’re designed. In your proposed design, you’ll be missing out on some of that.

Leave a Comment