Here are the major components of a subsystem:
- User-mode server. The server creates a (A)LPC port and listens for and handles client requests.
- User-mode client DLL. In the DLL_INIT_ROUTINE, you can connect to the port set up by the server. This DLL will expose your subsystem’s API, and some functions will require communication with the server.
- Kernel-mode support driver (you might not need this).
You will want to store process or thread state in either your server or driver. If you’re storing it in the server, you might need something like NtRegisterThreadTerminatePort to ensure you get to clean up when a process or thread exits. If you’re using a driver, you need PsSetCreateProcessNotifyRoutine.
And lastly, if you’re on XP and below, you can add new system calls. You can do this by calling KeAddSystemServiceTable. To invoke the system calls from user-mode, you need to create stubs like this (for x86):
; XyzCreateFooBar(__out PHANDLE FooBarHandle, __in ACCESS_MASK DesiredAccess, ...)
mov eax, SYSTEM_CALL_NUMBER
mov edx, 0x7ffe0300
call [edx]
retn 4
On Vista and above you can no longer add new system service tables because there is only room for two: the kernel’s system calls and win32k’s system calls.
After a bit of Googling I found this: http://winntposix.sourceforge.net/. I think it’s very similar to what you’re looking for, and uses a lot of the things I have mentioned.