I could not use the codes posted so far because the codes using “multiprocessing.Pool” do not work with lambda expressions and the codes not using “multiprocessing.Pool” spawn as many processes as there are work items. I adapted the code s.t. it spawns a predefined amount of workers and only iterates through the input list if … Read more
The basic problem is that the JSON encoder json.dumps() only knows how to serialize a limited set of object types by default, all built-in types. List here: https://docs.python.org/3.3/library/json.html#encoders-and-decoders One good solution would be to make your class inherit from JSONEncoder and then implement the JSONEncoder.default() function, and make that function emit the correct JSON for … Read more
Pickle save: try: import cPickle as pickle except ImportError: # Python 3.x import pickle with open(‘data.p’, ‘wb’) as fp: pickle.dump(data, fp, protocol=pickle.HIGHEST_PROTOCOL) See the pickle module documentation for additional information regarding the protocol argument. Pickle load: with open(‘data.p’, ‘rb’) as fp: data = pickle.load(fp) JSON save: import json with open(‘data.json’, ‘w’) as fp: json.dump(data, fp) … Read more
The output file needs to be opened in binary mode: f = open(‘varstor.txt’,’w’) needs to be: f = open(‘varstor.txt’,’wb’)
Here is a list of what can be pickled. In particular, functions are only picklable if they are defined at the top-level of a module. This piece of code: import multiprocessing as mp class Foo(): @staticmethod def work(self): pass if __name__ == ‘__main__’: pool = mp.Pool() foo = Foo() pool.apply_async(foo.work) pool.close() pool.join() yields an error … Read more
You could use the pickle module in the standard library. Here’s an elementary application of it to your example: import pickle class Company(object): def __init__(self, name, value): self.name = name self.value = value with open(‘company_data.pkl’, ‘wb’) as outp: company1 = Company(‘banana’, 40) pickle.dump(company1, outp, pickle.HIGHEST_PROTOCOL) company2 = Company(‘spam’, 42) pickle.dump(company2, outp, pickle.HIGHEST_PROTOCOL) del company1 del … Read more
Try this: import pickle a = {‘hello’: ‘world’} with open(‘filename.pickle’, ‘wb’) as handle: pickle.dump(a, handle, protocol=pickle.HIGHEST_PROTOCOL) with open(‘filename.pickle’, ‘rb’) as handle: b = pickle.load(handle) print(a == b) There’s nothing about the above solution that is specific to a dict object. This same approach will will work for many Python objects, including instances of arbitrary classes … Read more