When you cast the first argument to Object yourself, the compiler will match the method without using autoboxing (JLS3 15.12.2):
The first phase (§15.12.2.2) performs
overload resolution without permitting
boxing or unboxing conversion, or the
use of variable arity method
invocation. If no applicable method is
found during this phase then
processing continues to the second
phase.
If you don’t cast it explicitly, it will go to the second phase of trying to find a matching method, allowing autoboxing, and then it is indeed ambiguous, because your second argument can be matched by boolean or Object.
The second phase (§15.12.2.3) performs
overload resolution while allowing
boxing and unboxing, but still
precludes the use of variable arity
method invocation.
Why, in the second phase, doesn’t the compiler choose the second method because no autoboxing of the boolean argument is necessary? Because after it has found the two matching methods, only subtype conversion is used to determine the most specific method of the two, regardless of any boxing or unboxing that took place to match them in the first place (§15.12.2.5).
Also: the compiler can’t always choose the most specific method based on the number of auto(un)boxing needed. It can still result in ambiguous cases. For example, this is still ambiguous:
public class Test {
static void f(Object a, boolean b) {}
static void f(int a, Object b) {}
static void m(int a, boolean b) { f(a, b); } // ambiguous
}
Remember that the algorithm for choosing a matching method (compile-time step 2) is fixed and described in the JLS. Once in phase 2 there is no selective autoboxing or unboxing. The compiler will locate all the methods that are accessible (both methods in these cases) and applicable (again the two methods), and only then chooses the most specific one without looking at boxing/unboxing, which is ambiguous here.