While none of us have access to the internal review process (which appears to be continuously changing), there are a few things that can be said based on the responses that people have received.
First, Apple has no access to your source code, so they do not review that. You submit a binary as part of an application bundle, along with your other application resources.
They do, however, appear to scan your application’s binary for certain symbols that indicate the use of private APIs. A number of applications started crashing after iPhone OS updates because they used these private APIs, so Apple has been cracking down on this.
There are plenty of applications on the store that have memory leaks or other performance issues. I know that I’ve submitted versions of my applications that had subtle leaks (since fixed) and had no problems with review. Therefore, it does not appear that they do any sort of performance testing or profiling.
The only place where a memory leak causes a problem doing review is when that memory leak gets so bad that the application crashes when the reviewer is testing it. If your application crashes at any point during the review process, it will be rejected.
Beyond that, they have a checklist of user interface elements that they check for proper usage of (no persistent selections on table view rows, etc.). If your application deviates significantly from the Human Interface Guidelines when using these standard UI elements, you may get rejected.
Apple is very careful about copyright, particularly with their own images and artwork, so you may run into trouble if you use copyrighted material improperly.
Most of the rejection reasons you will face are preventable by making your application stable and by following platform guidelines, but some are not. Certain classes of applications have been rejected due to their intended use, and again the classes of applications that are allowed on the store change on a regular basis. This can add frustration and uncertainty when dealing with the App Store, but the vast majority of application types will never run into problems (as can be seen in the diversity of applications currently available).