You should run scrapy crawl spider_name command being in a scrapy project folder, where scrapy.cfg file resides. From the docs: Crawling To put our spider to work, go to the project’s top level directory and run: scrapy crawl dmoz
The twisted reactor cannot be restarted. A work around for this is to let the celery task fork a new child process for each crawl you want to execute as proposed in the following post: Running Scrapy spiders in a Celery task This gets around the “reactor cannot be restart-able” issue by utilizing the multiprocessing … Read more
This is what you’d use the meta Keyword for. def parse(self, response): for sel in response.xpath(‘//tbody/tr’): item = HeroItem() # Item assignment here url=”https://” + item[‘server’] + ‘.battle.net/’ + sel.xpath(‘td[@class=”cell-BattleTag”]//a/@href’).extract()[0].strip() yield Request(url, callback=self.parse_profile, meta={‘hero_item’: item}) def parse_profile(self, response): item = response.meta.get(‘hero_item’) item[‘weapon’] = response.xpath(‘//li[@class=”slot-mainHand”]/a[@class=”slot-link”]/@href’).extract()[0].split(“https://stackoverflow.com/”)[4] yield item Also note, doing sel = Selector(response) is a waste … Read more
response.css(‘mytag *::text’) The * will visit all the inner tags of mytag and ::text will get the text of each of them
By default, CrawlerProcess‘s .start() will stop the Twisted reactor it creates when all crawlers have finished. You should call process.start(stop_after_crawl=False) if you create process in each iteration. Another option is to handle the Twisted reactor yourself and use CrawlerRunner. The docs have an example on doing that.
Okay here is how I got Scrapy working with my Django project that uses Celery for queuing up what to crawl. The actual workaround came primarily from joehillen’s code located here http://snippets.scrapy.org/snippets/13/ First the tasks.py file from celery import task @task() def crawl_domain(domain_pk): from crawl import domain_crawl return domain_crawl(domain_pk) Then the crawl.py file from multiprocessing … Read more
Since Scrapy 1.2.0, a new setting FEED_EXPORT_ENCODING is introduced. By specifying it as utf-8, JSON output will not be escaped. That is to add in your settings.py: FEED_EXPORT_ENCODING = ‘utf-8’
You can write custom middleware for duplicate removal and add it in settings import os from scrapy.dupefilter import RFPDupeFilter class CustomFilter(RFPDupeFilter): “””A dupe filter that considers specific ids in the url””” def __getid(self, url): mm = url.split(“&refer”)[0] #or something like that return mm def request_seen(self, request): fp = self.__getid(request.url) if fp in self.fingerprints: return True … Read more
Move your USER_AGENT line to the settings.py file, and not in your scrapy.cfg file. settings.py should be at same level as items.py if you use scrapy startproject command, in your case it should be something like myproject/settings.py
Make sure you have set the “name” property of the spider. Example: class campSpider(BaseSpider): name=”campSpider” Without the name property, the scrapy manager will not be able to find your spider.