How to crop biggest rectangle out of an image

As I have previously done something similar, I have experienced with hough transforms, but they were much harder to get right for my case than using contours. I have the following suggestions to help you get started:

1. Generally paper (edges, at least) is white, so you may have better luck by going to a colorspace like YUV which better separates luminosity:

   image_yuv = cv2.cvtColor(image,cv2.COLOR_BGR2YUV)
   image_y = np.zeros(image_yuv.shape[0:2],np.uint8)
   image_y[:,:] = image_yuv[:,:,0]
   

2. The text on the paper is a problem. Use a blurring effect, to (hopefully) remove these high frequency noises. You may also use morphological operations like dilation as well.

   image_blurred = cv2.GaussianBlur(image_y,(3,3),0)
   

3. You may try to apply a canny edge-detector, rather than a simple threshold. Not necessarily, but may help you:

    edges = cv2.Canny(image_blurred,100,300,apertureSize = 3)
   

4. Then find the contours. In my case I only used the extreme outer contours. You may use CHAIN_APPROX_SIMPLE flag to compress the contour

   contours,hierarchy = cv2.findContours(edges,cv2.RETR_EXTERNAL,cv2.CHAIN_APPROX_SIMPLE)
   

5. Now you should have a bunch of contours. Time to find the right ones. For each contour cnt, first find the convex hull, then use approaxPolyDP to simplify the contour as much as possible.

   hull = cv2.convexHull(cnt)
   simplified_cnt = cv2.approxPolyDP(hull,0.001*cv2.arcLength(hull,True),True)
   

6. Now we should use this simplified contour to find the enclosing quadrilateral. You may experiment with lots of rules you come up with. The simplest method is picking the four longest longest segments of the contour, and then create the enclosing quadrilateral by intersecting these four lines. Based on your case, you can find these lines based on the contrast the line makes, the angle they make and similar things.

7. Now you have a bunch of quadrilaterals. You can now perform a two step method to find your required quadrilateral. First you remove those ones that are probably wrong. For example one angle of the quadrilateral is more than 175 degrees. Then you can pick the one with the biggest area as the final result. You can see the orange contour as one of the results I got at this point:
   

8. The final step after finding (hopefully) the right quadrilateral, is transforming back to a rectangle. For this you can use findHomography to come up with a transformation matrix.

   (H,mask) = cv2.findHomography(cnt.astype('single'),np.array([[[0., 0.]],[[2150., 0.]],[[2150., 2800.]],[[0.,2800.]]],dtype=np.single))
   

   The numbers assume projecting to letter paper. You may come up with better and more clever numbers to use. You also need to reorder the contour points to match the order of coordinates of the letter paper. Then you call warpPerspective to create the final image:

   final_image = cv2.warpPerspective(image,H,(2150, 2800))
   

   This warping should result in something like the following (from my results before):
   

I hope this helps you to find an appropriate approach in your case.