You can use scrapy.Request meta attribute: import scrapy class MySpider(scrapy.Spider): name=”myspider” def start_requests(self): urls = […] for index, url in enumerate(urls): yield scrapy.Request(url, meta={‘index’:index}) def parse(self, response): print(response.url) print(response.meta[‘index’])
The twisted reactor cannot be restarted. A work around for this is to let the celery task fork a new child process for each crawl you want to execute as proposed in the following post: Running Scrapy spiders in a Celery task This gets around the “reactor cannot be restart-able” issue by utilizing the multiprocessing … Read more
Since Scrapy 1.2.0, a new setting FEED_EXPORT_ENCODING is introduced. By specifying it as utf-8, JSON output will not be escaped. That is to add in your settings.py: FEED_EXPORT_ENCODING = ‘utf-8’