Solutions
- Keeping the pickle file unchanged ,upgrade your pandas version to 1.3.x and then load the pickle file.
Or
- Keeping your current pandas version unchanged, downgrade the pandas version to 1.2.x on the dumping side, and then dump a new pickle file with v1.2.x. Load it on your side with your pandas of version 1.2.x
In short
your pandas version used to dump the pickle(dump_version, probably 1.3.x) isn’t comptaible with your pandas version used to load the pickle (load_version, probably 1.2.x) . To solve it, try to upgrade the pandas version(load_version) to 1.3.x in the loading environment and then load the pickle. Or downgrade the pandas version(dump_version) to 1.2.x and then redump a new pickle. After this, you can load the new pickle with your pandas of version 1.2.x
And this has nothing to do with PySpark
In long
This issue is related to the backward imcompatibility between Pandas version 1.2.x and 1.3.x. In the version 1.2.5 and before, Pandas use the variable name new_blocks in module pandas.core.internals.blocks cf source code v1.2.5. On 2 July 2021, Pandas released version 1.3.0. In this update, Pandas changed the api, the variable name new_blocks in module pandas.core.internals.blocks has been changed to new_block cf source code v1.3.0.
This change of API will result into two imcompatiblity errors:
- If you have dumped a pickle with Pandas v1.3.x, and you try to load the pickle with Pandas v1.2.x, you will get the following error:
AttributeError: Can't get attribute 'new_block' on <module 'pandas.core.internals.blocks' from '.../site-packages/pandas/core/internals/blocks.py'>'>
Python throw this error complaining that it can not found the attribute new_block on your current pandas.core.internals.blocks because in order to pickle load an object, it has to use the exact same class used for dumping the pickle.
This is exactly your case: Having dumped the pickle with Pandas v1.3.x and try to load the pickle with Pandas v1.2.x
To reproduce the error
pip install --upgrade pandas==1.3.4
import numpy as np
import pandas as pd
df =pd.DataFrame(np.random.rand(3,6))
with open("dump_from_v1.3.4.pickle", "wb") as f:
pickle.dump(df, f)
quit()
pip install --upgrade pandas==1.2.5
import pickle
with open("dump_from_v1.3.4.pickle", "rb") as f:
df = pickle.load(f)
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-2-ff5c218eca92> in <module>
1 with open("dump_from_v1.3.4.pickle", "rb") as f:
----> 2 df = pickle.load(f)
3
AttributeError: Can't get attribute 'new_block' on <module 'pandas.core.internals.blocks' from '/opt/anaconda3/lib/python3.7/site-packages/pandas/core/internals/blocks.py'>