Literals aren't limited to only numbers. They can be booleans, characters, strings, numbers, null, etc. (not a complete list).
They're called literals because they represent an absolute value which cannot be interpreted as anything other else. We call them literals because we need a name to refer to what these things are.
There is a big difference between these:
int val = '4'; // not actually the integer literal 4, but the character literal '4' which in ASCII is equivalent to 52
int val2 = 4; // this is the integer literal 4
int val3 = (int) 4f; // this is the floating point literal 4f. Because floats are not implicitly castable to int, we must explicitly cast it.
// ...
There are complicated reasons (some of which is historical) why integer literals and long literals are treated differently, and I won't pretend to know/understand all of it. My guess is that it has something to do with implementation details of the compiler (especially on older 32-bit systems), and possibly has historical reasons coming from C/C++ where ints originally held 16-bit values and longs held 32-bit values.