I suppose this draft titled “Principles of the Same-Origin Policy” explains (albeit briefly) what’s pretty much in everyone’s head:
In principle, user agents could treat every URL as a separate principal and isolate each document from every other URL unless the document explicitly indicated that it trusted that URL. Unfortunately, this design is cumbersome for developers because web applications often consist of a number of resource acting in concert.
As an approximation, user agents group URLs together into protection domains called origins. In particular, two URLs are part of the same origin (i.e., represent the same principal) if they have the same scheme, host, and port.
In short: it would be much harder to build the Web if everything was subject to SOP.