dask

Converting numpy solution into dask (numpy indexing doesn’t work in dask)

by

Chunk random_days_panel instead of historical_data and use da.map_blocks: def dask_way(sim_count, sim_days, hist_days): # shared historical data # on a cluster you’d load this on each worker, e.g. from a NPZ file historical_data = np.random.normal(111.51, 10, size=hist_days) random_days_panel = da.random.randint( 1, hist_days, size=(1, 1, sim_count, sim_days) ) future_panel = da.map_blocks( lambda chunk: historical_data[chunk], random_days_panel, dtype=float ) … Read more

Can dask parralelize reading fom a csv file?

by

Yes, dask.dataframe can read in parallel. However you’re running into two problems: Pandas.read_csv only partially releases the GIL By default dask.dataframe parallelizes with threads because most of Pandas can run in parallel in multiple threads (releases the GIL). Pandas.read_csv is an exception, especially if your resulting dataframes use object dtypes for text dask.dataframe.to_hdf(filename) forces sequential … Read more

How to see progress of Dask compute task?

by

If you’re using the single machine scheduler then do this: from dask.diagnostics import ProgressBar ProgressBar().register() http://dask.pydata.org/en/latest/diagnostics-local.html If you’re using the distributed scheduler then do this: from dask.distributed import progress result = df.id.count.persist() progress(result) Or just use the dashboard http://dask.pydata.org/en/latest/diagnostics-distributed.html

Read a large csv into a sparse pandas dataframe in a memory efficient way

by

I would probably address this by using dask to load your data in a streaming fashion. For example, you can create a dask dataframe as follows: import dask.dataframe as ddf data = ddf.read_csv(‘test.csv’) This data object hasn’t actually done anything at this point; it just contains a “recipe” of sorts to read the dataframe from … Read more

Writing Dask partitions into single file

by

Short answer No, Dask.dataframe.to_csv only writes CSV files to different files, one file per partition. However, there are ways around this. Concatenate Afterwards Perhaps just concatenate the files after dask.dataframe writes them? This is likely to be near-optimal in terms of performance. df.to_csv(‘/path/to/myfiles.*.csv’) from glob import glob filenames = glob(‘/path/to/myfiles.*.csv’) with open(‘outfile.csv’, ‘w’) as out: … Read more

Convert Pandas dataframe to Dask dataframe

by

I think you can use dask.dataframe.from_pandas: from dask import dataframe as dd sd = dd.from_pandas(df, npartitions=3) print (sd) dd.DataFrame<from_pa…, npartitions=2, divisions=(0, 1, 2)> EDIT: I find solution: import pandas as pd import dask.dataframe as dd from dask.dataframe.utils import make_meta df=pd.DataFrame({‘a’:[1,2,3],’b’:[4,5,6]}) dsk = {(‘x’, 0): df} meta = make_meta({‘a’: ‘i8’, ‘b’: ‘i8’}, index=pd.Index([], ‘i8’)) d = … Read more

python dask DataFrame, support for (trivially parallelizable) row apply?

by

map_partitions You can apply your function to all of the partitions of your dataframe with the map_partitions function. df.map_partitions(func, columns=…) Note that func will be given only part of the dataset at a time, not the entire dataset like with pandas apply (which presumably you wouldn’t want if you want to do parallelism.) map / … Read more

Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)