Adding a shadowPath should give you a huge performance boost. The following example assumes you only want the shadow on the sides of your view
CGPathRef path = [UIBezierPath bezierPathWithRect:view.bounds].CGPath;
[view.layer setShadowPath:path];
EDIT:
On default a CALayer draws a shadow during animations, the following code allows you to cache the shadow as a bitmap and reuse it instead of redrawing it:
self.view.layer.shouldRasterize = YES;
// Don't forget the rasterization scale
// I spent days trying to figure out why retina display assets weren't working as expected
self.view.layer.rasterizationScale = [UIScreen mainScreen].scale;