In the case of 11 line segments, you have 330 ways of choosing four segments. You could determine the likelihood of each combination making a quadrilateral, and grade that way.
It is possible to have a Hough transform detect forms other than lines, though it becomes harder to visualise as the accumulator space would require more than two dimensions. Circles can be found in three dimensions (midX, midY, radius), ellipses in four (I believe). I’m not sure exactly how few parameters you’d need to model a quadrilateral, and I believe that the performance of the Hough transform starts to drop off when you get higher than three dimensions. The accumulator space becomes so large that the noise ratio increases significantly.
Here’s a related question that may have some interesting answers for you.
Let us know how you get on!
EDIT
I took a stab at this problem today, and uploaded my solution to GitHub. There is too much code to post here.
Here’s a screenshot showing the output:
The solution I took is basically what I described above before this edit.
- Find all combinations of four lines
- Find all permutations of those four lines
- Evaluate the likelihood that those four lines form a quadrilateral
- Take the best match
The evaluation works by calculating a crude error score. This is the sum of two different types of error:
- The deviation at each corner from 90 degrees (I use the sum of squared errors across all four corners)
- When the line segments intersect within the line segment, it’s likely not a valid corner
The second type of error could possibly be determined in a more robust way. It was necessary to find a solution for your sample data set.
I haven’t experimented with other data sets. It may need some tweaking to make it more robust. I have tried to avoid using too many parameters so that it should be straightforward to adjust to a particular environment. For example to control sensitivity to occlusion, as seen in your sample image.
It finds the solution in about 160ms on my laptop. However I haven’t made any performance optimisations. I expect that the methods of finding combinations/permutations could be significantly optimised if you needed this to run closer to real-time, as is often the case with computer vision experiments.