I have seen similar behavior couple times and it was related to lowering cpu frequency for power saving. If you have rooted device you can try going to:
cd /sys/devices/system/cpu/
And for all cpu’s you see (replace cpuX by cpu0, cpu1 and etc) do:
echo 1 > cpuX/online
echo performance > cpuX/cpufreq/scaling_governor
If this is the case then unfortunately I don’t think you can easily workaround this problem.