When dealing with the creating of large numbers of objects, often the single biggest performance enhancement you can use is to turn the garbage collector off. Every “generation” of objects, the garbage collector traverses all the live objects in memory, looking for objects that are a part of cycles but are not pointed at by live objects, thus are eligible for memory reclamation. See Doug Helmann’s PyMOTW GC article for some information (more can perhaps be found with google and some determination). The garbage collector is run by default every 700 or so objects created-but-not-reclaimed, with subsequent generations running somewhat less often (I forget the exact details).
Using a standard tuple instead of a Point class can save you some time (using a namedtuple is somewhere in between), and clever unpacking can save you some time, but the largest gain can be had by turning the gc off before your creation of lots of objects that you know don’t need to be gc’d, and then turning it back on afterwards.
Some code:
def orig_test_gc_off():
import time
data = get_data()
all_point_sets = []
import gc
gc.disable()
time_start = time.time()
for row in data:
point_set = []
first_points, second_points = row
# Convert points from strings to integers
first_points = map(int, first_points.split(","))
second_points = map(int, second_points.split(","))
paired_points = zip(first_points, second_points)
curr_points = [Point(p[0], p[1]) \
for p in paired_points]
all_point_sets.append(curr_points)
time_end = time.time()
gc.enable()
print "gc off total time: ", (time_end - time_start)
def test1():
import time
import gc
data = get_data()
all_point_sets = []
time_start = time.time()
gc.disable()
for index, row in enumerate(data):
first_points, second_points = row
curr_points = map(
Point,
[int(i) for i in first_points.split(",")],
[int(i) for i in second_points.split(",")])
all_point_sets.append(curr_points)
time_end = time.time()
gc.enable()
print "variant 1 total time: ", (time_end - time_start)
def test2():
import time
import gc
data = get_data()
all_point_sets = []
gc.disable()
time_start = time.time()
for index, row in enumerate(data):
first_points, second_points = row
first_points = [int(i) for i in first_points.split(",")]
second_points = [int(i) for i in second_points.split(",")]
curr_points = [(x, y) for x, y in zip(first_points, second_points)]
all_point_sets.append(curr_points)
time_end = time.time()
gc.enable()
print "variant 2 total time: ", (time_end - time_start)
orig_test()
orig_test_gc_off()
test1()
test2()
Some results:
>>> %run /tmp/flup.py
total time: 6.90738511086
gc off total time: 4.94075202942
variant 1 total time: 4.41632509232
variant 2 total time: 3.23905301094