Need to use doesNotExist() instead.
Found here.
If the view is there in the view hierarchy but in an invisible state (visibility is set to ‘INVISIBLE’), use not(isDisplayed). However, if the view is not there at all in the view hierarchy (e.g. visibility set to ‘GONE’), doesNotExist() is used.