What values can a constructor return to avoid returning this?

The exact condition is described on the [[Construct]] internal property, which is used by the new operator:

From the ECMA-262 3rd. Edition Specification:

13.2.2 [[Construct]]

When the [[Construct]] property for a Function object F is
called, the following steps are taken:

1. Create a new native ECMAScript object.
2. Set the [[Class]] property of Result(1) to "Object".
3. Get the value of the prototype property of F.
4. If Result(3) is an object, set the [[Prototype]] property of Result(1) to Result(3).
5. If Result(3) is not an object, set the [[Prototype]] property of Result(1) to the original Object prototype object as
   described in 15.2.3.1.
6. Invoke the [[Call]] property of F, providing Result(1) as the this value and
   providing the argument list passed into [[Construct]] as the
   argument values.
7. If Type(Result(6)) is
   Object then return Result(6).
8. Return Result(1).

Look at steps 7 and 8, the new object will be returned only if the
type of Result(6) (the value returned from the F constructor
function) is not an Object.