Concat multiple slices in golang

This question has been answered already, but I wanted to post this here because the accepted answer is not the most efficient.

The reason is that creating an empty slice and then appending can lead to many unnecessary allocations.

The most efficient way would be to pre-allocate a slice and copy the elements into it. Below is a package which implements the concatenation both ways. If you benchmark you can see that pre-allocating is ~2x faster and allocates much less memory.

Benchmark Results:

go test . -bench=. -benchmem
testing: warning: no tests to run
BenchmarkConcatCopyPreAllocate-8    30000000            47.9 ns/op        64 B/op          1 allocs/op
BenchmarkConcatAppend-8             20000000           107 ns/op         112 B/op          3 allocs/op

Package concat:

package concat

func concatCopyPreAllocate(slices [][]byte) []byte {
    var totalLen int
    for _, s := range slices {
        totalLen += len(s)
    }
    tmp := make([]byte, totalLen)
    var i int
    for _, s := range slices {
        i += copy(tmp[i:], s)
    }
    return tmp
}

func concatAppend(slices [][]byte) []byte {
    var tmp []byte
    for _, s := range slices {
        tmp = append(tmp, s...)
    }
    return tmp
}

Benchmark tests:

package concat

import "testing"

var slices = [][]byte{
    []byte("my first slice"),
    []byte("second slice"),
    []byte("third slice"),
    []byte("fourth slice"),
    []byte("fifth slice"),
}

var B []byte

func BenchmarkConcatCopyPreAllocate(b *testing.B) {
    for n := 0; n < b.N; n++ {
        B = concatCopyPreAllocate(slices)
    }
}

func BenchmarkConcatAppend(b *testing.B) {
    for n := 0; n < b.N; n++ {
        B = concatAppend(slices)
    }
}

Leave a Comment

Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)