How does HDF5 compare against using something like an SQLite DB? Is that even a reasonable comparison to make? Sort of similar but not really. They’re both structured files. SQLite has features to support database queries using SQL. HDF5 has features to support large scientific datasets. They’re both meant to be high performance. Over time … Read more
UPDATE: nowadays I would choose between Parquet, Feather (Apache Arrow), HDF5 and Pickle. Pro’s and Contra’s: Parquet pros one of the fastest and widely supported binary storage formats supports very fast compression methods (for example Snappy codec) de-facto standard storage format for Data Lakes / BigData contras the whole dataset must be read into memory. … Read more
We use Python in conjunction with h5py, numpy/scipy and boost::python to do data analysis. Our typical datasets have sizes of up to a few hundred GBs. HDF5 advantages: data can be inspected conveniently using the h5view application, h5py/ipython and the h5* commandline tools APIs are available for different platforms and languages structure data using groups … Read more
I had a similar problem. The line listed with the ‘EOF inside string’ had a string that contained within it a single quote mark (‘). When I added the option quoting=csv.QUOTE_NONE it fixed my problem. For example: import csv df = pd.read_csv(csvfile, header = None, delimiter=”\t”, quoting=csv.QUOTE_NONE, encoding=’utf-8’)
Simply updating pytables with: pip install –upgrade tables worked for me.
I have found a solution that seems to work! Have a look at this: incremental writes to hdf5 with h5py! In order to append data to a specific dataset it is necessary to first resize the specific dataset in the corresponding axis and subsequently append the new data at the end of the “old” nparray. … Read more
As someone who developed a scientific project that went from using folders of files to HDF5, I think I can shed some light on the advantages of HDF5. When I began my project, I was operating on small test datasets, and producing small amounts of output, in the range of kilobytes. I began with the … Read more
Updated to use pandas 0.13.1 1) No. http://pandas.pydata.org/pandas-docs/dev/io.html#notes-caveats. There are various ways to do this, e.g. have your different threads/processes write out the computation results, then have a single process combine. 2) depending the type of data you store, how you do it, and how you want to retrieve, HDF5 can offer vastly better performance. … Read more
I strongly suggest you HDF5 instead of NetCDF. NetCDF is flat, and it gets very dirty after a while if you are not able to classify stuff. Of course classification is also a matter of debate, but at least you have this flexibility. We performed an accurate evaluation of HDF5 vs. NetCDF when I wrote … Read more