Thread: Generics & 'Type Erasure': disingenuous documentation?

Java 5 and onwards allows type variables and parametrized types to be used in declarations. 'Type Erasure' is a grandiose term describing a "process" whereby the compiler eliminates all references to these type variables and parameters in the generated byte code. The reason given is to make Java 5 and onwards compatible with earlier versions of Java, which do not support Generics.

What I suspect is the case is that Generics has been added as incomplete, "design time only" support for parameterized types, which the compiler IGNORES at the point of actual generation of byte code -- hardly what I would call a "process of type erasure". I suspect that the compiler simply does nothing different, and generates the same byte code as the equivalent raw types. The only compiler difference is in its design time validation.

Backward compatibility is a good reason for allowing aliasing of raw types with generic types. It is not a good reason for allowing heap pollution. Heap pollution causes pending runtime exceptions which should be trapped immediately. This suggests that parametrized type information is important at runtime, even when you require backward compatibility.

Can anyone prove me wrong, perhaps by creating a concise example of a generic type that compiles into different byte code from its equivalent raw type?

No we can't because the compiler doesn't generate any new code at all; Generics is a compile time thingy; even the runtime casts are still present, even if the compiler 'knows' that a generic type is of a matching concrete type. This is caused by the backwards compatibility b.t.w. Hava a look at the following snippet:

Java Code:
import java.util.ArrayList;
import java.util.List;

public class T {
public static void main(String[] args) {
List<String> list= new ArrayList<String>();

list.add("foo");
System.out.println(list.get(0)+"bar");
}

}
The compiler 'knows' that the list will only contain objects of the type String, but still there's a runtime cast:

Java Code:
public class T extends java.lang.Object{
public T();
Code:
0: aload_0
1: invokespecial #8; //Method java/lang/Object."<init>":()V
4: return

public static void main(java.lang.String[]);
Code:
0: new #16; //class java/util/ArrayList
3: dup
4: invokespecial #18; //Method java/util/ArrayList."<init>":()V
7: astore_1
8: aload_1
9: ldc #19; //String foo
11: invokeinterface #21, 2; //InterfaceMethod java/util/List.add:(Ljava/lang/Object;)Z
16: pop
17: getstatic #27; //Field java/lang/System.out:Ljava/io/PrintStream;
20: new #33; //class java/lang/StringBuilder
23: dup
24: aload_1
25: iconst_0
26: invokeinterface #35, 2; //InterfaceMethod java/util/List.get:(I)Ljava/lang/Object;
31: checkcast #39; //class java/lang/String
34: invokestatic #41; //Method java/lang/String.valueOf:(Ljava/lang/Object;)Ljava/lang/String;
37: invokespecial #45; //Method java/lang/StringBuilder."<init>":(Ljava/lang/String;)V
40: ldc #48; //String bar
42: invokevirtual #50; //Method java/lang/StringBuilder.append:(Ljava/lang/String;)Ljava/lang/StringBuilder;
45: invokevirtual #54; //Method java/lang/StringBuilder.toString:()Ljava/lang/String;
48: invokevirtual #58; //Method java/io/PrintStream.println:(Ljava/lang/String;)V
51: return

}
See line #31; the cast wouldn't be needed if there were no 'raw' types. The only benefit is type safety during compile time and no explicit casts in the source code.

thank you very much
The presence of the cast adds weight to the argument that the compiler generates the same code as it does for a raw type. I just don't know why this isn't being stated upfront in documentation, but is being obfuscated in terms like "type erasure" which eventually seem to mean no more than "no change to the byte code generator".

Did you see my point about heap pollution? Backward compatibility is not a reason for this "type erasure". Type is still needed at runtime for throwing exceptions at the correct point.

Yep, the generated code is the same as before generics saw the light. Those casts are still needed during runtime because of the 'backward compatibility'. I don't understand your remark on 'heap pollution'; a raw type can store any type of object and only if that cast fails, an Exception will be thrown. The cast can only fail if some 'backward compatible' code goofed ...

My point about heap pollution is this. If a non-compatible type parameter is used for a generic type through being aliased with a raw type, the runtime allows heap pollution. The runtime will put the rogue object on the heap. It doesn't complain at that point! It should!

It is only if you later assign the object to a "reifiable" type that it can give you a runtime exception.

So, backward compatibility does not mean you should erase type information.

The runtime does not throw an exception on the add method. It defers it until the get.

Originally Posted by cyberion

.
The runtime does not throw an exception on the add method. It defers it until the get.

I'm not sure what your issue is. It doesn't surprise me that Java doesn't complain about the add since the List variable, l, is not generic (and the compiler even warns you of this problem). If you made the variable generic, then you'd get a compile-time error from the code above and there wouldn't be any risk of "pollution".

--- Update ---

that's all just because generics are a compiler issue, nothing on earth forbids anyone to use a List<T> as if it were a raw List (because of the 'backward compatibility' issue). I wish there would've been a compiler flag that forbids those 'raw' types and generates a hard error if a parameterized type were attempted to use as such. You're right, a 'non T' type can be stored in a List<T> without notice, or until it is attempted to use that 'non T' as if it were a T (hence the generated casts).

Originally Posted by fortioni

)that's all just because generics are a compiler issue.

But generics are not a "compiler issue". They've only implemented it as compiler issue and worded the documentation to make it sound OK.

I do not believe it is correct for the runtime to place something on the heap and then LATER turn round and say, "oh, look what I've found on the heap! That shouldn't be there!"

The reason for this problem is precisely because they have implemented generics as a compiler issue when it is more than that.

--- Update ---

This is a usage issue, not a backward compatibility issue. Whenever would your new code want your legacy code to use List<T> in a non-T way?

Ok, I reword that phrase: generics are a compile time issue in Java and hence its weakness. OTOH, when you want to go all the way, you end up with possible 'code bloat' as in C++'s templates; I guess the Java language designers wanted to avoid that.

--- Update ---

Originally Posted by cyberion

This is a usage issue, not a backward compatibility issue. Whenever would your new code want your legacy code to use List<T> in a non-T way?

you're assuming a reasonable user; assumptions are always wrong ;-)

Then that is what the documentation should say ;-)

It shouldn't obfuscate the "weakness" with excuses about backward compatibility. Backward compatibility does not force type erasure.

Originally Posted by cyberion

Then that is what the documentation should say ;-)

It shouldn't obfuscate the "weakness" with excuses about backward compatibility. Backward compatibility does not force type erasure.

I gave it some more thoughts, but, suppose the requirements were:

a) old code should keep on compiling and
b) old code should keep on running and
c) generics are supposed to be a compile time thing only (a consequence of a) and b)?).

then, 'type erasure' is the only solution ... of course that documentation only brags about it.

Originally Posted by fortioni

c) generics are supposed to be a compile time thing only (a consequence of a) and b)?).

then, 'type erasure' is the only solution ... of course that documentation only brags about it.

But c) is not a consequence of a) and b). Nor is type erasure the only solution.

Throwing a runtime exception at the point of heap pollution (which requires runtime generic type information) violates neither a) nor b).

The reason it can't be done is because of type erasure, not backward compatibility!

If I am using a parametrized type, it must be because I intend to access the object in a way that requires it to be of that type. Otherwise there would be no need to constrain it! Calling a legacy function that corrupts the parametrized type is misuse and should be flagged when it happens. Ideally, the error shouldn't be deferred to the point where the object is accessed.

The idea that I could use a parametrized type for one part of its life cycle and then allow it to be something different in the latter part of its life cycle, is not what backward compatibility is supposed to support. Such a thing would be bad practice.

Originally Posted by cyberion

But c) is not a consequence of a) and b). Nor is type erasure the only solution.

.

I don't know if it's the only solution, but it is the reasoning that the documentations give:

"Type erasure enables Java applications that use generics to maintain binary compatibility with Java libraries and applications that were created before generics."

Heap pollution is a result of the implemented solution and not the requirement.

The documentation fails to mention that type erasure is a compromise that is more than just ensuring backward compatibility.

I agree with fortioni that possible compiler and output bloat may be a factor.

I think I was also a little hasty in saying type erasure is a simple matter of ignoring type information at generation time. I now recognise that, once you have decided to employ type erasure, deciding what the equivalent raw type would be is not necessarily a simple matter! With inheritance and correlated parametrization to consider, things could become complex and scenarios could arise that were not there before generics.

--- Update ---

Here is another example of what I am finding disingenuous:

Java interprets varargs as arrays.
Java Code:
public static void faultyMethod(List<String>... l) {
// Valid
Object[] objectArray = l;
objectArray[0] = Arrays.asList(new Integer(42));
// ClassCastException thrown here
String s = l[0].get(0);
}
Method argument, "l", should first become List<String>[]

The line labelled "Valid" should actually give an unchecked warning.

Only at generation time should the <String> parameter be stripped, giving a List[].

But what unfortunately seems to be happening is that the compiler does it all in one hit and immediately translates the varargs parameter l to List[] in anticipation of type erasure. That is why there is no warning, not because the line is valid.

This is a glitch.

Why have they labeled it "valid" and then pretended to explain it?

If l were an explicit List<String>[], it would not anticipate type erasure. It would erase the type at the correct point and an unchecked error would be given!

The line labelled valid will never give an unchecked warning, even if the parameter were a List<String>[].
Java Code:
private static void test(List<String>[] l) {
Object[] o = l; // This is valid
// do stuff
}

Originally Posted by fortioni

The line labelled valid will never give an unchecked warning, even if the parameter were a List<String>[].
Java Code:
private static void test(List<String>[] l) {
Object[] o = l; // This is valid
// do stuff
}

You are right. It does not give an unchecked warning. Question is, shouldn't it?

I find this ambiguous..


So we have a "valid" statement which can potentially introduce heap pollution. What does "valid" mean then? Something is obviously skew-whiff.

I mean, am I supposed to believe that creating a List<String>[] is essentially invalid, or is it an artifact of the Java Language? Is it not odd that the assignment statement, which is the clear candidate for incompatibility issues, is being called the valid statement? I would say that validity is being controlled by circumstance rather than essence.

No it shouldn't.
Casting an array of something to an Object[] is not something that needs checking since everything is rooted in the Object class.

It's possibly a side effect of the nature of arrays, but doesn't strike me as all that unusual.

It all, essentially, boils down to..."don't write bollocks code".

Originally Posted by fortioni

No it shouldn't.
Casting an array of something to an Object[] is not something that needs checking since everything is rooted in the Object class.

It's possibly a side effect of the nature of arrays, but doesn't strike me as all that unusual.

It all, essentially, boils down to..."don't write bollocks code".

Well that is your opinion and I think it sidesteps the issue. The documentation says that it can lead to heap pollution and you are saying it doesn't need checking.

And as for what you think it boils down to, that is a cop out.

I wouldn't be surprised if what you are today defending, gets fixed. Then you'll be telling everyone how logical the new way is! :-)

I expect to see generic type information in the runtime in some future release. I don't really believe it is all that difficult. They just haven't got round to doing it. They can stamp an array with its type, so why not a List. They don't have to use the stamp to break backward compatibility, just to trap heap pollution at the point where it happens.

But that would not fix the issue of casting SomeClass[] to an Object[], which has nothing at all to do with type erasure.
Which is the point I was replying to.

The above that you complained about being valid, is just as valid without touching Collections at all...

Originally Posted by fortioni

But that would not fix the issue of casting SomeClass[] to an Object[], which has nothing at all to do with type erasure.
Which is the point I was replying to.

The above that you complained about being valid, is just as valid without touching Collections at all...

Yes it would. The documentation states that the assignment introduces possible heap pollution. If you classes are stamped with their type at runtime, their wouldn't be any heap pollution. It would throw a runtime exception.

Which has nothing to do with type erasure, since that is something specific to generics, and:
Java Code:

Object[] objects = new SomeClass[]();


predates them by several years.

You are conflating two different things here.
Type erasure and everything extending Object.
The former may well get "fixed" (though I wouldn't hold my breath since you'll get a runtime exception as soon as you try and use an array that's been buggered up), the latter is not likely to.

Originally Posted by fortioni

Which has nothing to do with type erasure, since that is something specific to generics, and:
Java Code:

Object[] objects = new SomeClass[]();


predates them by several years.

You are conflating two different things here.
Type erasure and everything extending Object.
The former may well get "fixed" (though I wouldn't hold my breath since you'll get a runtime exception as soon as you try and use an array that's been buggered up), the latter is not likely to.

Huh? Heap corruption has nothing to do with type erasure??? ***scratches head***

I am not conflating the issues. I am looking at them as a whole. You are looking at the assignment in isolation. Of course you can assign any reference type to object, and any array reference type to object array.

Put it this way, suppose I had a complex method which generated a warning about the conversion of its parameterized arguments. I didn't write the function, but I need to check where there may be possible misuse. The varargs themselves are not essentially the issue. That is just an artifact of how the Java compiler has been made to behave. The issue is at the assignment to object. That is where the vulnerability is exposed and where the compiler should flag. As it is, I could be left with some rather complex compilation analysis or be forced to relinquish the parametrization which may have been used in a totally proper way.

We're obviously talking at cross-purposes.

The array assignment marked as Valid in the code snippet from the docs has always been valid, and is valid for things that do not involve generics. Therefore that assignment (which is the only thing I have been talking about here) is nothing to do with type erasure. It is simply down to the nature of extending from a base object (and the associated logic behind arrays).

So, as I said, it's a part of the language and is most unlikely to be changed.

I know it has been there since day 1. Object is at the top of the reference type hierarchy and can be assigned any reference type.

But, when you evolve a language, there is impact. You can't just say that what was valid in earlier versions remains totally valid. You are tweaking the semantic perspective on the language and there is regression. So, I find the argument that it is valid because it was valid before, to be wrong.

If you try to answer the question in my last post, you will see that the assignment to object ought now to be flagged, however valid it used to be before generics hit.

(Reminds me of the same thing in the world of science. You can't build a world perspective in way that is entirely incremental and without review. Perspectives break, just like Newtonian Dynamics)

Why should the cast to Object[] be flagged though?
Will you flag casts to List<SomeClass>[] as well?
How about any cast?
Because that is essentially what you are suggesting...casting is always suspicious.
Which, to be frank, is nonsense.

Almost any artifact of a language is open to misuse.

I'm curious, in your work have you encountered a problem with this aspect of Java?
Because in over a decade working in it I honestly can't think of any in the stuff I've written or reviewed.

Same goes for type erasure as well, as a matter of fact.
Now, automatic primitive wrapping...that keeps causing trouble.

Originally Posted by fortioni

Why should the cast to Object[] be flagged though?
Will you flag casts to List<SomeClass>[] as well?
How about any cast?
Because that is essentially what you are suggesting...casting is always suspicious.
Which, to be frank, is nonsense.

Almost any artifact of a language is open to misuse.

I'm curious, in your work have you encountered a problem with this aspect of Java?
Because in over a decade working in it I honestly can't think of any in the stuff I've written or reviewed.

Same goes for type erasure as well, as a matter of fact.
Now, automatic primitive wrapping...that keeps causing trouble.

No. The "nonsense" suggestion is entirely your own. Nowhere have I suggested flag all casts. If a cast fails at runtime, you know exactly where to look in the source code because it will tell you.

You are still avoiding addressing the scenario in my last but one post, which is why you think any old cast is equivalent.

The problem here is impact analysis. Earlier versions of Java didn't even flag the function declaration in the varargs problem. They only flagged the invocation. Now they've move the compiler error closer to the source of the error. But, as I said, please address the scenario, so that you comment more accurately on my perspective.