How do I see what inside the function is making the code take so long (Should I even use cProfile?)
Yes, you can use cProfile but the way you asked the question makes me wonder if line_profiler (third party module, you need to install it) wouldn’t be a better tool.
I’m using the IPython/Jupyter bindings of this package when I want to profile a function:
%load_ext line_profiler
To actually profile a function:
%lprun -f foo foo()
# ^^^^^---- this call will be profiled
# ^^^-----------function to profile
Which produces this output:
Timer unit: 5.58547e-07 s
Total time: 17.1189 s
File: <ipython-input-1-21b5a5f52f66>
Function: foo at line 1
Line # Hits Time Per Hit % Time Line Contents
==============================================================
1 def foo():
2 10001 31906 3.2 0.1 for i in range(10000):
3 10000 30534065 3053.4 99.6 a = i**i
4 10000 75998 7.6 0.2 if i % 1000 == 0:
5 10 6953 695.3 0.0 print(i)
That includes several things that might be interesting. For example 99.6% of the time is spent in the i**i line.
- What is the best way to set about optimising my code once I know what is using the most CPU
That depends. Sometimes you need to use different functions/datastructures/algorithms – sometimes you can’t do anything. But at least you know where your bottleneck is and you can estimate how much impact a change at the bottleneck or somewhere else would have.