The X window system generally does the drawing – you then use a toolkit such as Qt or GTK on top of raw Xlib to provide event loops, drag and drop, starting apps on mouseclicks and all the other ‘desktop’ stuff
It’s fairly easy to work directly with Xlib and opengl or if you just want to learn opengl the glut provides the framework you need to display a window, handle mouse/keyboard events and so on.