.writerow() requires a sequence (”, (), []) and places each index in it’s own column of the row, sequentially. If your desired string is not an item in a sequence, writerow() will iterate over each letter in your string and each will be written to your CSV in a separate cell. after you import csv … Read more
There are a number of ways to do it, but, none will always work. Here are the two easiest: if it’s a known finite set of websites: in your scraper convert each url from the normal url to the print url for a given site (cannot really be generalized across sites) Use the arc90 readability … Read more
wget \ –recursive \ –no-clobber \ –page-requisites \ –html-extension \ –convert-links \ –restrict-file-names=windows \ –domains www.website.com \ –no-parent \ www.website.com Read more about it here.
Just loop through all the <strong> tags and use next_sibling to get what you want. Like this: for strong_tag in soup.find_all(‘strong’): print(strong_tag.text, strong_tag.next_sibling) Demo: from bs4 import BeautifulSoup html=””‘ <p> <strong class=”offender”>YOB:</strong> 1987<br /> <strong class=”offender”>RACE:</strong> WHITE<br /> <strong class=”offender”>GENDER:</strong> FEMALE<br /> <strong class=”offender”>HEIGHT:</strong> 5’05”<br /> <strong class=”offender”>WEIGHT:</strong> 118<br /> <strong class=”offender”>EYE COLOR:</strong> GREEN<br /> … Read more
Node.io seems to take the cake 🙂
I setup a PhantomJs Cloud Service, and it pretty much does what you are asking. It took me about 5 weeks of work implement. The biggest problem you’ll run into is the known-issue of memory leaks in PhantomJs. The way I worked around this is to cycle my instances every 50 calls. The second biggest … Read more
Provide the meta tag name as the first argument to find(). Then, use keyword arguments to check the specific attributes: title = soup.find(“meta”, property=”og:title”) url = soup.find(“meta”, property=”og:url”) print(title[“content”] if title else “No meta title given”) print(url[“content”] if url else “No meta url given”) The if/else checks here would be optional if you know that … Read more
Scrapy cannot interpret javascript. If you absolutely must interact with the javascript on the page, you want to be using Selenium. If using Scrapy, the solution to the problem depends on what the button is doing. If it’s just showing content that was previously hidden, you can scrape the data without a problem, it doesn’t … Read more
I’m guessing that you need to assign the Exception to a variable. As shown in the Python 3 tutorial: def fails(): x = 1 / 0 try: fails() except Exception as ex: print(ex) To give a brief explanation, as is a pseudo-assignment keyword used in certain compound statements to assign or alias the preceding statement … Read more
This is how you do it. from urllib import request, parse data = parse.urlencode(<your data dict>).encode() req = request.Request(<your url>, data=data) # this will make the method “POST” resp = request.urlopen(req)