How to reference a requirements.txt in the pyproject.toml of a setuptools project?

Use dynamic metadata:

[project]
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}

Note that this will use a requirements.txt–like syntax; each line must conform to PEP 508, so flags like -r, -c, and -e are not supported inside this requirements.txt.

Additionally:

If you are using an old version of setuptools, you might need to ensure that all files referenced by the file directive are included in the sdist (you can do that via MANIFEST.in or using plugins such as setuptools-scm, please have a look on [sic] Controlling files in the distribution for more information).

Changed in version 66.1.0: Newer versions of setuptools will automatically add these files to the sdist.

If you want to use optional-dependencies, say, with a requirements-dev.txt, you will need to put an extra group, as follows (credit to Billbottom):

[project]
dynamic = ["dependencies"]
[tool.setuptools.dynamic]
dependencies = {file = ["requirements.txt"]}
optional-dependencies = {dev = { file = ["requirements-dev.txt"] }}