How to give delay between each requests in scrapy?

There is a setting for that:

DOWNLOAD_DELAY

Default: 0

The amount of time (in secs) that the downloader should wait before
downloading consecutive pages from the same website. This can be used
to throttle the crawling speed to avoid hitting servers too hard.

DOWNLOAD_DELAY = 0.25    # 250 ms of delay

Read the docs: https://doc.scrapy.org/en/latest/index.html

Leave a Comment