Many others have mentioned to just use display: none but as you know this has an entirely different behavior than using the visibility property.
One thing you can do is to use hidden / inherit instead of hidden / visible. inherit will cause an element to be visible by default, unless one of its parents is not. Which is what you want.