size_t is for objects, off_t is for files.
mmap merges the two concepts, pretty much by definition. Personally I think I’d use size_t, since no matter what else it is, a mapped file is also an array in (virtual) memory.
size_t is standard C++, off_t is Posix, and off64_t is a GNU extension that goes with the functions fopen64, ftello64, etc. I think it should always be the same type as off_t on 64 bit GNU systems, but don’t bet your company on that without checking.
Should it be relevant, off_t is signed whereas size_t is unsigned. But the signed counterpart to size_t is ptrdiff_t, so when you need a signed type it doesn’t automatically mean you should use off_t or off64_t.