Redis will be faster than Postgres because Pg offers reliability guarantees on your data (when the transaction is committed, it is guaranteed to be on disk), whereas Redis has a concept of writing to disk when it feels like it, so shouldn’t be used for critical data. Redis seems like a good option for your … Read more
RocksDB is used for several (internal) reasons (as you mentioned already for example its performance). Conceptually, Kafka Streams does not need RocksDB — it is used as internal key-value cache and any other store offering similar functionality would work, too. Comment from @miguno below (rephrased): One important advantage of RocksDB in contrast to pure in-memory … Read more
I would suggest to have a look at LMDB (which is the most efficient engine for OpenLDAP, and used in a number of other open-source projects). LMDB is an embedded key/value store, with a Berkeley-DB or LevelDB like API, does not have to store everything in memory, and can support access from multiple processes. There … Read more
You can use sqlitedict which provides key-value interface to SQLite database. SQLite limits page says that theoretical maximum is 140 TB depending on page_size and max_page_count. However, default values for Python 3.5.2-2ubuntu0~16.04.4 (sqlite3 2.6.0), are page_size=1024 and max_page_count=1073741823. This gives ~1100 GB of maximal database size which fits your requirement. You can use the package … Read more
If it has to be answered in one word, its “performance”. Aerospike’s performance is much better than any clustered-nosql solutions out there. Higher performance per-node means smaller cluster which is lower TCO (Total Cost of Ownership) and maintenance. Aerospike does auto-clustering, auto-sharding, auto-rebalancing (when cluster state changes) most of which needs manual steps in other … Read more
I’ll just call out one or two common reasons (I’m sure people will be writing essay answers) With highly distributed systems, any given data set may be spread across multiple servers. When that happens, the relational constraints which the DB engine can guarantee are greatly reduced. Some of your referential integrity will need to be … Read more
I was also looking for this kind of operation. I didn’t find anything, so I did it with MULTI/EXEC: MULTI expire key1 expire key2 expire key3 EXEC
The colons have been in earlier redis versions as a concept for storing namespaced data. In early versions redis supported only strings, if you wanted to store the email and the age of ‘bob’ you had to store it all as a string, so colons were used: SET user:bob:email bob@example.com SET user:bob:age 31 They had … Read more
I had the same requirements as you but couldn’t find a suitable database. nStore was promising but the API was not nearly complete enough and not very coherent. That’s why I made NeDB, which a dependency-less embedded database for Node.js projects. You can use it with a simple require(), it is persistent, and its API … Read more
My current project actually. Storing 18,000 objects in a normalised structure: 90,000 rows across 8 different tables. Took 1 minute to retrieve and map them to our Java object model, that’s with everything correctly indexed etc. Storing them as key/value pairs using a lightweight text representation: 1 table, 18,000 rows, 3 seconds to retrieve them … Read more