Here is a sketch of an O(n^2 log n) solution.
First, the preliminaries shared with other answers.
When we have a line passing through some rectangles, we can translate it to any of the two sides until it passes through a corner of some rectangle.
After that, we fix that corner as the center of rotation and rotate the line to any of the two sides until it passes through another corner.
During the whole process, all points of intersection between our line and rectangle sides stayed on these sides, so the number of intersections stayed the same, as did the number of rectangles crossed by the line.
As a result, we can consider only lines which pass through two rectangle corners, which is capped by O(n^2), and is a welcome improvement compared to the infinite space of arbitrary lines.
So, how do we efficiently check all these lines?
First, let us have an outer loop which fixes one point A and then considers all lines passing through A.
There are O(n) choices of A.
Now, we have one point A fixed, and want to consider all lines AB passing through all other corners B.
In order to do that, first sort all other corners B according to the polar angle of AB, or, in other words, angle between axis Ox and vector AB.
Angles are measured from -PI to +PI or from 0 to 2 PI or otherwise, the point in which we cut the circle to sort angles can be arbitrary.
The sorting is done in O(n log n).
Now, we have points B1, B2, …, Bk sorted by the polar angle around point A (their number k is something like 4n-4, all corners of all rectangles except the one where point A is a corner).
First, look at the line AB1 and count the number of rectangles crossed by that line in O(n).
After that, consider rotating AB1 to AB2, then AB2 to AB3, all the way to ABk.
The events which happen during the rotation are as follows:
-
When we rotate to ABi, and Bi is the first corner of some rectangle in our order, the number of rectangles crossed increases by 1 as soon as the rotating line hits Bi.
-
When we rotate to ABj, and Bj is the last corner of some rectangle in our order, the number of rectangles crossed decreases by 1 as soon as the line rotates past Bj.
Which corners are first and last can be established with some O(n) preprocessing, after the sort, but before considering the ordered events.
In short, we can rotate to the next such event and update the number of rectangles crossed in O(1).
And there are k = O(n) events in total.
What’s left to do is to track the global maximum of this quantity throughout the whole algorithm.
The answer is just this maximum.
The whole algorithm runs in O(n * (n log n + n + n)), which is O(n^2 log n), just as advertised.