How do I get countifs to select all non-blank cells in Excel?

Use a criteria of "<>". It will count anything which isn’t an empty cell, including #NAME? or #DIV/0!. As to why it works, damned if I know, but Excel seems to understand it.

Note: works nicely in Google Spreadsheet too