<aside> seems appropriate. The current spec with relevant sections bolded:
The aside element represents a section
of a page that consists of content
that is tangentially related to the
content around the aside element, and
which could be considered separate
from that content. Such sections are
often represented as sidebars in
printed typography.The element can be used for
typographical effects like pull quotes
or sidebars, for advertising, for
groups of nav elements, and for other
content that is considered separate
from the main content of the page.
In this case, a modal is “tangentially related” to the action that caused it. While you might normally expect an aside to be in a sidebar, one of the purposes of semantic content is to enable versatility that’s unrestricted by physical page characteristics. The last phrase of the spec seems to imply just this versatile use case.