Scrapy: how to disable or change log?
You need to add the following to your settings.py document: LOG_LEVEL = ‘INFO’ See the documentation.
scrapy
How to access scrapy settings from item Pipeline
UPDATE (2021-05-04) Please note that this answer is now ~7 years old, so it’s validity can no longer be ensured. In addition it is using Python2 The way to access your Scrapy settings (as defined in settings.py) from within your_spider.py is simple. All other answers are way too complicated. The reason for this is the … Read more
Scrapy throws ImportError: cannot import name xmlrpc_client
I’ve just fixed this issue on my OS X. Please backup your files first. sudo rm -rf /Library/Python/2.7/site-packages/six* sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six* sudo pip install six Scrapy 1.0.0 is ready to go. If you encounter an error rm: /System/Library/… Operation not permitted Please try to disable System Integrity Protection See Operation Not Permitted when on … Read more
Scrapy: Follow link to get additional Item data?
Please, first read the docs to understand what i say. The answer: To scrape additional fields which are on other pages, in a parse method extract URL of the page with additional info, create and return from that parse method a Request object with that URL and pass already extracted data via its meta parameter. … Read more
python No module named service_identity [duplicate]
pip install service_identity It is complaining that you do not have service_identity. More information on the module can be found here on PyPI Note that this is a rather unusual scenario, because service_identity is not a requirement of the twisted package, whose developers for some reason chose not to add it to its list of … Read more
CrawlerProcess vs CrawlerRunner
Scrapy’s documentation does a pretty bad job at giving examples on real applications of both. CrawlerProcess assumes that scrapy is the only thing that is going to use twisted’s reactor. If you are using threads in python to run other code this isn’t always true. Let’s take this as an example. from scrapy.crawler import CrawlerProcess … Read more
Force my scrapy spider to stop crawling
In the latest version of Scrapy, available on GitHub, you can raise a CloseSpider exception to manually close a spider. In the 0.14 release note doc is mentioned: “Added CloseSpider exception to manually close spiders (r2691)” Example as per the docs: def parse_page(self, response): if ‘Bandwidth exceeded’ in response.body: raise CloseSpider(‘bandwidth_exceeded’) See also: http://readthedocs.org/docs/scrapy/en/latest/topics/exceptions.html?highlight=closeSpider
Is it possible to pass a variable from start_requests() to parse() for each individual request?
You can use scrapy.Request meta attribute: import scrapy class MySpider(scrapy.Spider): name=”myspider” def start_requests(self): urls = […] for index, url in enumerate(urls): yield scrapy.Request(url, meta={‘index’:index}) def parse(self, response): print(response.url) print(response.meta[‘index’])
Passing arguments to process.crawl in Scrapy python
pass the spider arguments on the process.crawl method: process.crawl(spider, input=”inputargument”, first=”James”, last=”Bond”)
Access Django models with scrapy: defining path to Django project
I think the main misconception is the package path vs the settings module path. In order to use django’s models from an external script you need to set the DJANGO_SETTINGS_MODULE. Then, this module has to be importable (i.e. if the settings path is myproject.settings, then the statement from myproject import settings should work in a … Read more