Thanks to @StevePiercy I found this documentation: https://www.sphinx-doc.org/en/master/usage/configuration.html#confval-html_favicon
conf.py
html_favicon = 'favicon.ico'
I put favicon.ico in my source folder, next to my .rst-files and it is working as expected:
https://global-coffee-data-standard.readthedocs.io/en/latest/index.html