You can transform either column ‘a’ or ‘b’ so they are both either float64 or bool. However, an easier solution that preserves the data type of your features is this:
DF3 = DF[(DF['a'] == 0) | (DF['b'] == 0)]
A common operation is the use of boolean vectors to filter the data. The operators are: | for or, & for and, and ~ for not. These must be grouped by using parentheses.