Using the norecursedirs option in pytest.ini or tox.ini can save a lot of collection time, depending on what other files you have in your working directory. My collection time is roughly halved for a suite of 300 tests when I have that in place (0.34s vs 0.64s).
If you’re already using tox like I am, you just need to add the following in your tox.ini:
[pytest]
norecursedirs = docs *.egg-info .git appdir .tox
You can also add it in a free-standing pytest.ini file.
The pytest documentation has more details on pytest configuration files.