As justhalf points out above, my question here is essentially a duplicate of this question. The HTML content reported itself as UTF-8 encoded and, for the most part it was, except for one or two rogue invalid UTF-8 characters. This apparently confuses BeautifulSoup about which encoding is in use, and when trying to first decode … Read more
This you won’t be able to do with requests and BeautifulSoup as the page that you want to extract the information from loads the rest of the entries through JS when you scroll down. You can do this using selenium which opens a real browser and you can pass page down key press events programmatically. … Read more
Use soup.findAll(attrs={‘class’: None}) Quoting from docs: You can use attrs if you need to put restrictions on attributes whose names are Python reserved words, like class, for, or import; or attributes whose names are non-keyword arguments to the Beautiful Soup search methods: name, recursive, limit, text, or attrs itself.
.findAll() works for nested li elements: for ul in uls: for li in ul.findAll(‘li’): print(li) Output: <li>List items</li> <li>Etc…</li> <li>List items</li> <li>Nested list items</li> <li>Nested list items</li> <li>List items</li>
You are trying to install BeautifulSoup 3, which is not Python 3 compatible. As the Pycharm error window explains: Make sure you use a version of Python supported by this package. Currently you are using Python 3.5. However, you want to install BeautifulSoup 4 instead; the project name for that series has changed to beautifulsoup4. … Read more
ele is already a tag, try doing this: soup = BeautifulSoup(‘<html><p>line 1</p><div><a>line 2</a></div></html>’) print(soup.find(‘p’).nextSibling.name) So in your example it would be just print(ele.name)
You can’t directly do a comprehension with an OrderedDict. You can, however, use a generator in the constructor for OrderedDict. Try this on for size: import requests from bs4 import BeautifulSoup from collections import OrderedDict soup = BeautifulSoup(html, ‘html.parser’) tables = soup.find_all(‘table’) rows = tables[1].find_all(‘tr’) t_data = OrderedDict((row.th.text, row.td.text) for row in rows if row.td)
You could also use findAll to get all the rows in a list and after that just use the slice syntax to access the elements that you need: rows = soup.findAll(‘tr’)[4::5]
You can pass a function to findAll: >>> print soupHandler.findAll(‘div’, id=lambda x: x and x.startswith(‘post-‘)) [<div id=”post-45″>…</div>, <div id=”post-334″>…</div>] Or a regular expression: >>> print soupHandler.findAll(‘div’, id=re.compile(‘^post-‘)) [<div id=”post-45″>…</div>, <div id=”post-334″>…</div>]
Just ignore NavigableString objects while iterating through the tree: from bs4 import BeautifulSoup, NavigableString, Tag response = requests.get(url) soup = BeautifulSoup(response.text, ‘html.parser’) for body_child in soup.body.children: if isinstance(body_child, NavigableString): continue if isinstance(body_child, Tag): print(body_child.name)