syscallis the default way of entering kernel mode onx86-64. This instruction is not available in 32 bit modes of operation on Intel processors.sysenteris an instruction most frequently used to invoke system calls in 32 bit modes of operation. It is similar tosyscall, a bit more difficult to use though, but that is the kernel’s concern.int 0x80is a legacy way to invoke a system call and should be avoided.
The preferred way to invoke a system call is to use vDSO, a part of memory mapped in each process address space that allows to use system calls more efficiently (for example, by not entering kernel mode in some cases at all). vDSO also takes care of more difficult, in comparison to the legacy int 0x80 way, handling of syscall or sysenter instructions.
Also, see this and this.