Different generic behaviour when using lambda instead of explicit anonymous inner class

tldr:

1. There is a bug in javac that records the wrong enclosing method for lambda-embedded inner classes. As a result, type variables on the actual enclosing method cannot be resolved by those inner classes.
2. There are arguably two sets of bugs in the java.lang.reflect API implementation:
   • Some methods are documented as throwing exceptions when nonexistent types are encountered, but they never do. Instead, they allow null references to propagate.
   • The various Type::toString() overrides currently throw or propagate a NullPointerException when a type cannot be resolved.

The answer has to do with the generic signatures that usually get emitted in class files that make use of generics.

Typically, when you write a class that has one or more generic supertypes, the Java compiler will emit a Signature attribute containing the fully parameterized generic signature(s) of the class’s supertype(s). I’ve written about these before, but the short explanation is this: without them, it would not be possible to consume generic types as generic types unless you happened to have the source code. Due to type erasure, information about type variables gets lost at compilation time. If that information were not included as extra metadata, neither the IDE nor your compiler would know that a type was generic, and you could not use it as such. Nor could the compiler emit the necessary runtime checks to enforce type safety.

javac will emit generic signature metadata for any type or method whose signature contains type variables or a parameterized type, which is why you are able to obtain the original generic supertype information for your anonymous types. For example, the anonymous type created here:

TypeToken<?> token = new TypeToken<List<? extends CharSequence>>() {};

…contains this Signature:

LTypeToken<Ljava/util/List<+Ljava/lang/CharSequence;>;>;

From this, the java.lang.reflection APIs can parse the generic supertype information about your (anonymous) class.

But we already know that this works just fine when the TypeToken is parameterized with concrete types. Let’s look at a more relevant example, where its type parameter includes a type variable:

static <F> void test() {
    TypeToken sup = new TypeToken<F[]>() {};
}

Here, we get the following signature:

LTypeToken<[TF;>;

Makes sense, right? Now, let’s look at how the java.lang.reflect APIs are able to extract generic supertype information from these signatures. If we peer into Class::getGenericSuperclass(), we see that the first thing it does is call getGenericInfo(). If we haven’t called into this method before, a ClassRepository gets instantiated:

private ClassRepository getGenericInfo() {
    ClassRepository genericInfo = this.genericInfo;
    if (genericInfo == null) {
        String signature = getGenericSignature0();
        if (signature == null) {
            genericInfo = ClassRepository.NONE;
        } else {
            // !!!  RELEVANT LINE HERE:  !!!
            genericInfo = ClassRepository.make(signature, getFactory());
        }
        this.genericInfo = genericInfo;
    }
    return (genericInfo != ClassRepository.NONE) ? genericInfo : null;
}

The critical piece here is the call to getFactory(), which expands to:

CoreReflectionFactory.make(this, ClassScope.make(this))

ClassScope is the bit we care about: this provides a resolution scope for type variables. Given a type variable name, the scope gets searched for a matching type variable. If one is not found, the ‘outer’ or enclosing scope is searched:

public TypeVariable<?> lookup(String name) {
    TypeVariable<?>[] tas = getRecvr().getTypeParameters();
    for (TypeVariable<?> tv : tas) {
        if (tv.getName().equals(name)) {return tv;}
    }
    return getEnclosingScope().lookup(name);
}

And, finally, the key to it all (from ClassScope):

protected Scope computeEnclosingScope() {
    Class<?> receiver = getRecvr();

    Method m = receiver.getEnclosingMethod();
    if (m != null)
        // Receiver is a local or anonymous class enclosed in a method.
        return MethodScope.make(m);

    // ...
}

If a type variable (e.g., F) is not found on the class itself (e.g., the anonymous TypeToken<F[]>), then the next step is to search the enclosing method. If we look at the disassembled anonymous class, we see this attribute:

EnclosingMethod: LambdaTest.test()V

The presence of this attribute means that computeEnclosingScope will produce a MethodScope for the generic method static <F> void test(). Since test declares the type variable W, we find it when we search the enclosing scope.

So, why doesn’t it work inside a lambda?

To answer this, we must understand how lambdas get compiled. The body of the lambda gets moved into a synthetic static method. At the point where we declare our lambda, an invokedynamic instruction gets emitted, which causes a TypeToken implementation class to be generated the first time we hit that instruction.

In this example, the static method generated for the lambda body would look something like this (if decompiled):

private static /* synthetic */ Object lambda$test$0() {
    return new LambdaTest$1();
}

…where LambdaTest$1 is your anonymous class. Let’s dissassemble that and inspect our attributes:

Signature: LTypeToken<TW;>;
EnclosingMethod: LambdaTest.lambda$test$0()Ljava/lang/Object;

Just like the case where we instantiated an anonymous type outside of a lambda, the signature contains the type variable W. But EnclosingMethod refers to the synthetic method.

The synthetic method lambda$test$0() does not declare type variable W. Moreover, lambda$test$0() is not enclosed by test(), so the declaration of W is not visible inside it. Your anonymous class has a supertype containing a type variable that your the class doesn’t know about because it’s out of scope.

When we call getGenericSuperclass(), the scope hierarchy for LambdaTest$1 does not contain W, so the parser cannot resolve it. Due to how the code is written, this unresolved type variable results in null getting placed in the type parameters of the generic supertype.

Note that, had your lambda had instantiated a type that did not refer to any type variables (e.g., TypeToken<String>) then you would not run into this problem.

Conclusions

(i) There is a bug in javac. The Java Virtual Machine Specification §4.7.7 (“The EnclosingMethod Attribute”) states:

It is the responsibility of a Java compiler to ensure that the method identified via the method_index is indeed the closest lexically enclosing method of the class that contains this EnclosingMethod attribute. (emphasis mine)

Currently, javac seems to determine the enclosing method after the lambda rewriter runs its course, and as a result, the EnclosingMethod attribute refers to a method that never even existed in the lexical scope. If EnclosingMethod reported the actual lexically enclosing method, the type variables on that method could be resolved by the lambda-embedded classes, and your code would produce the expected results.

It is arguably also a bug that the signature parser/reifier silently allows a null type argument to be propagated into a ParameterizedType (which, as @tom-hawtin-tackline points out, has ancillary effects like toString() throwing a NPE).

My bug report for the EnclosingMethod issue is now online.

(ii) There are arguably multiple bugs in java.lang.reflect and its supporting APIs.

The method ParameterizedType::getActualTypeArguments() is documented as throwing a TypeNotPresentException when “any of the actual type arguments refers to a non-existent type declaration”. That description arguably covers the case where a type variable is not in scope. GenericArrayType::getGenericComponentType() should throw a similar exception when “the underlying array type’s type refers to a non-existent type declaration”. Currently, neither appears to throw a TypeNotPresentException under any circumstances.

I would also argue that the various Type::toString overrides should merely fill in the canonical name of any unresolved types rather than throwing a NPE or any other exception.

I have submitted a bug report for these reflection-related issues, and I will post the link once it is publicly visible.

Workarounds?

If you need to be able to reference a type variable declared by the enclosing method, then you can’t do that with a lambda; you’ll have to fall back to the longer anonymous type syntax. However, the lambda version should work in most other cases. You should even be able to reference type variables declared by the enclosing class. For example, these should always work:

class Test<X> {
    void test() {
        Supplier<TypeToken<X>> s1 = () -> new TypeToken<X>() {};
        Supplier<TypeToken<String>> s2 = () -> new TypeToken<String>() {};
        Supplier<TypeToken<List<String>>> s3 = () -> new TypeToken<List<String>>() {};
    }
}

Unfortunately, given that this bug has apparently existed since lambdas were first introduced, and it has not been fixed in the most recent LTS release, you may have to assume the bug remains in your clients’ JDKs long after it gets fixed, assuming it gets fixed at all.