With a modal dialog, you set your application in a particular mode (a different “state” if you will), whereby only actions pertaining to that “mode” are accepted, hence preventing UI actions outside of the dialog.
At Andreas’ prompting I thought I may have to dig dusty Windows API books, as often, the etymology/origin of a word or expression that has became broadly accepted is only found in early documentation, but in fact we still see this referenced in an online glossary from MS. The Modal entry reads (emphasis is mine):
modal
Restrictive or limited interaction due to operating in a mode. Modal often describes a secondary window that restricts a user’s interaction with the owner window. See also: modeless.