Internally SQL Server is summing the values (to divide by the count later) and storing them in the columns data type – in this case an int – which isn’t large enough to hold the sum – if you cast the value as a bigint first it will sum the values also storing those values in a bigint – which is probably large enough, then the average calculation can proceed.
select avg(cast(number as bigint)) from LargeNumbers