Is it possible to pass a variable from start_requests() to parse() for each individual request?

You can use scrapy.Request meta attribute:

import scrapy

class MySpider(scrapy.Spider):
    name="myspider"

    def start_requests(self):
        urls = [...]
        for index, url in enumerate(urls):
            yield scrapy.Request(url, meta={'index':index})

    def parse(self, response):
        print(response.url)
        print(response.meta['index'])

Leave a Comment

Hata!: SQLSTATE[HY000] [1045] Access denied for user 'divattrend_liink'@'localhost' (using password: YES)