How do the nl80211 library & cfg80211 work?

To be able to control wireless drivers from userspace, some IPC communication processes between kernel and userspace are used.

• At first ioctl with vendor dependent APIs was used.
• In 1996, Jean Tourrilhes creates wireless extensions (WE or WEXT).

The Wireless Extension (WE) is a generic API allowing a driver to
expose to the user space configuration and statistics specific to
common Wireless LANs.

• In 2006, John Linville creates mac80211 and Johannes Berg creates cfg80211 and nl80211. Together it is intended to replace wireless extensions.

  +-------------+
  |             |
  |  Userspace  |
  |             |
  +-------------+
        ^
  - - - | - - - - 
        | nl80211
        v
  +-------------+
  |             |
  |  cfg80211   |
  |             |
  +-------------+
  +-------------+
  |             |
  |  mac80211   |
  |   driver    |
  |             |
  +-------------+
  

An important point is that nl80211/cfg80211/mac80211 no longer use ioctl, they use netlink.

So, tools like iw, hostapd or the wpa_supplicant use some netlink libraries (like libnl or libnl-tiny) and the netlink interface public header which is of course nl80211.h.

There is not so much documentations, but I advise you to read the libnl documentation and then the iw source code (because iw use libnl).