Scrapy is a Web-spider or web scraper framework, You give Scrapy a root URL to start crawling, then you can specify constraints on how many (number of) URLs you want to crawl and fetch,etc. It is a complete framework for web-scraping or crawling. While BeautifulSoup is a parsing library which also does a pretty good … Read more
Try this li = soup.find(‘li’, {‘class’: ‘text’}) children = li.findChildren(“a” , recursive=False) for child in children: print(child)
Here’s a short snippet using the SoupStrainer class in BeautifulSoup: import httplib2 from bs4 import BeautifulSoup, SoupStrainer http = httplib2.Http() status, response = http.request(‘http://www.nytimes.com’) for link in BeautifulSoup(response, parse_only=SoupStrainer(‘a’)): if link.has_attr(‘href’): print(link[‘href’]) The BeautifulSoup documentation is actually quite good, and covers a number of typical scenarios: https://www.crummy.com/software/BeautifulSoup/bs4/doc/ Edit: Note that I used the SoupStrainer class … Read more
Activate the virtualenv, and then install BeautifulSoup4: $ pip install BeautifulSoup4 When you installed bs4 with easy_install, you installed it system-wide. So your system python can import it, but not your virtualenv python. If you do not need bs4 to be installed in your system python path, uninstall it and keep it in your virtualenv. … Read more
.find_all() returns list of all found elements, so: input_tag = soup.find_all(attrs={“name” : “stainfo”}) input_tag is a list (probably containing only one element). Depending on what you want exactly you either should do: output = input_tag[0][‘value’] or use .find() method which returns only one (first) found element: input_tag = soup.find(attrs={“name”: “stainfo”}) output = input_tag[‘value’]
You should post your example document, because the code works fine: >>> import BeautifulSoup >>> soup = BeautifulSoup.BeautifulSoup(‘<html><body><div id=”articlebody”> … </div></body></html’) >>> soup.find(“div”, {“id”: “articlebody”}) <div id=”articlebody”> … </div> Finding <div>s inside <div>s works as well: >>> soup = BeautifulSoup.BeautifulSoup(‘<html><body><div><div id=”articlebody”> … </div></div></body></html’) >>> soup.find(“div”, {“id”: “articlebody”}) <div id=”articlebody”> … </div>
You are using Python 2 methodology instead of Python 3. Change: outfile=open(‘./immates.csv’,’wb’) To: outfile=open(‘./immates.csv’,’w’) and you will get a file with the following output: SNo,States,Dist,Population 1,Andhra Pradesh,13,49378776 2,Arunachal Pradesh,16,1382611 3,Assam,27,31169272 4,Bihar,38,103804637 5,Chhattisgarh,19,25540196 6,Goa,2,1457723 7,Gujarat,26,60383628 ….. In Python 3 csv takes the input in text mode, whereas in Python 2 it took it in binary mode. … Read more
Once upon a time I stumbled with this issue. If you’re using macOS go to Macintosh HD > Applications > Python3.6 folder (or whatever version of python you’re using) > double click on “Install Certificates.command” file. 😀
You can use find_all in the following way to find every a element that has an href attribute, and print each one: # Python2 from BeautifulSoup import BeautifulSoup html=””‘<a href=”some_url”>next</a> <span class=”class”><a href=”another_url”>later</a></span>”’ soup = BeautifulSoup(html) for a in soup.find_all(‘a’, href=True): print “Found the URL:”, a[‘href’] # The output would be: # Found the URL: … Read more
\xa0 is actually non-breaking space in Latin1 (ISO 8859-1), also chr(160). You should replace it with a space. string = string.replace(u’\xa0′, u’ ‘) When .encode(‘utf-8’), it will encode the unicode to utf-8, that means every unicode could be represented by 1 to 4 bytes. For this case, \xa0 is represented by 2 bytes \xc2\xa0. Read … Read more