The following should work soup = BeautifulSoup(htmlstring) soup.findAll(‘div’, style=”width=300px;”) There are couple of ways to search for tags. https://www.crummy.com/software/BeautifulSoup/bs4/doc/#searching-the-tree For more text to understand and use it http://lxml.de/elementsoup.html
I agree that the Scrapy docs give off that impression. But, I believe, as I found for myself, that if you are patient with Scrapy, and go through the tutorials first, and then bury yourself into the rest of the documentation, you will not only start to understand the different parts to Scrapy better, but … Read more
To simply suppress the warning and continue with processing this works: import warnings from bs4 import UserWarning warnings.filterwarnings(“ignore”, category=UserWarning, module=”bs4″)
Extract the string from the element: volume = soup.findAll(“span”, {“id”: “volume”})[0].string
I actually dealt with this myself, in the hackiest way possible: by post-processing the result. r = re.compile(r’^(\s*)’, re.MULTILINE) def prettify_2space(s, encoding=None, formatter=”minimal”): return r.sub(r’\1\1′, s.prettify(encoding, formatter)) Actually, I monkeypatched prettify_2space in place of prettify in the class. That’s not essential to the solution, but let’s do it anyway, and make the indent width a … Read more
soup.find(“div”, {“class”:”real number”})[‘data-value’] Here you are searching for a div element, but the span has the “real number” class in your example HTML data, try instead: soup.find(“span”, {“class”: “real number”, “data-value”: True})[‘data-value’] Here we are also checking for presence of data-value attribute. To find elements having “real number” or “fake number” classes, you can make … Read more
if tag.find(‘child_tag_name’):
You can access tag’s attributes by treating the tag like a dictionary (documentation): for tag in soup.find_all(class_=”bookmark blurb group”) : print tag.get(‘id’) The reason tag.id didn’t work is that it is equivalent to tag.find(‘id’), which results into None since there is no id tag found (documentation).
BeautifulSoup supports CSS selectors which allow you to select elements based on the content of particular attributes. This includes the selector *= for contains. The following will return all div elements with a class attribute containing the text ‘listing-col-‘: for EachPart in soup.select(‘div[class*=”listing-col-“]’): print EachPart.get_text()
From the documentation: As of Beautiful Soup version 4.9.0, when lxml or html.parser are in use, the contents of <script>, <style>, and <template> tags are not considered to be ‘text’, since those tags are not part of the human-visible content of the page. So basically the accepted answer from falsetru above is all good, but … Read more