An HTML entity reference is, depending on HTML version either an SGML entity or an XML entity (HTML inherits entities from the underlying technology). Entities are a way of inserting chunks of content defined elsewhere into the document.
All HTML entities are single-character entities, and are hence basically the same as character references (technically they are different to character references, but as there are no multi-character entities defined, the distinction has no impact on HTML).
When an HTML processor sees, for example — it replaces it with the content of that entity reference with the appropriate entity, based on the section in the DTD that says:
<!ENTITY mdash CDATA "—" -- em dash, U+2014 ISOpub -->
So it replaces the entity reference with the entity — which is in turn a character reference that gets replaced by the character — (U+2014). In reality unless you are doing this with a general-purpose XML or SGML processor that doesn’t understand HTML directly, this will really be done in one step.
Now, what would we replace your hypothetical &br; with to cause a line-break to happen? We can’t do so with a newline character, or even the lesser known U+2028 LINE SEPARATOR (which semantically in plain text has the same meaning as <br/> in HTML), because they are whitespace characters which are not significant in most HTML code, which is something that you should be grateful for as writing HTML would be much harder if we couldn’t format for readability within the source code.
What we need is not an entity, but a way to indicate semantically that the rendered content contains a line-break at this point. We also need to not indicate anything else (we can already indicate a line-break by beginning or ending a block element, but that’s not what we want). The only reasonable way to do so is to have an element that means exactly that, and so we have the <br/> element, with its related tag being put into the source code.