How to read a text file line-by-line in Go when some lines are long enough to cause “bufio.Scanner: token too long” errors?

by

You can also do:

scanner := bufio.NewScanner(file)
buf := make([]byte, 0, 64*1024)
scanner.Buffer(buf, 1024*1024)
for scanner.Scan() {
    // do your stuff
}

The second argument to scanner.Buffer() sets the maximum token size. In the above example you will be able to scan the file as long as none of the lines is larger than 1MB.

Leave a Comment