There is a recipe for sorting files larger than RAM on this page, though you’d have to adapt it for your case involving CSV-format data. There are also links to additional resources there. Edit: True, the file on disk is not “larger than RAM”, but the in-memory representation can easily become much larger than available … Read more
Clean and Smudge You can use clean and smudge to compress your file. Normally, this isn’t necessary, since git will compress it internally, but since gitHub is acting weird, it may help. The main commands would be like: git config filter.compress.clean gzip git config filter.compress.smudge gzip -d GitHub will see this as a compressed file, … Read more
I am a big fan of Java’s MappedByteBuffers for situations like this. It is blazing fast. Below is a snippet I put together for you that maps a buffer to the file, seeks to the middle, and then searches backwards to a newline character. This should be enough to get you going? I have similar … Read more
Using gzip.GzipFile: import gzip with gzip.open(‘input.gz’,’rt’) as f: for line in f: print(‘got line’, line) Note: gzip.open(filename, mode) is an alias for gzip.GzipFile(filename, mode). I prefer the former, as it looks similar to with open(…) as f: used for opening uncompressed files.
Just read each line and construct a json object at this time: with open(file_path) as f: for line in f: j_content = json.loads(line) This way, you load proper complete json object (provided there is no \n in a json value somewhere or in the middle of your json object) and you avoid memory issue as … Read more
That’s how I finally did it. The idea behind is to create readable stream implementing ReadStream interface and then use pipe() method to pipe data to writable stream. var fs = require(‘fs’); var writeStream = fs.createWriteStream(‘someFile.txt’, { flags : ‘w’ }); var readStream = new MyReadStream(); readStream.pipe(writeStream); writeStream.on(‘close’, function () { console.log(‘All done!’); }); The … Read more
Use web workers for large files processing instead doing it in main thread and upload chunks of file data using file.slice(). This article helps you to handle large files in workers. change XHR send to Websocket in main thread. //Messages from worker function onmessage(blobOrFile) { ws.send(blobOrFile); } //construct file on server side based on blob … Read more