Why doesn’t this code simply print letters A to Z?

From the docs:

PHP follows Perl’s convention when dealing with arithmetic operations on character variables and not C’s.

For example, in Perl 'Z'+1 turns into 'AA', while in C 'Z'+1 turns into '[' ( ord('Z') == 90, ord('[') == 91 ).

Note that character variables can be incremented but not decremented and even so only plain ASCII characters (a-z and A-Z) are supported.

From Comments:-
It should also be noted that <= is a lexicographical comparison, so 'z'+1 ≤ 'z'. (Since 'z'+1 = 'aa' ≤ 'z'. But 'za' ≤ 'z' is the first time the comparison is false.) Breaking when $i == 'z' would work, for instance.

Example here.