I have a suspicion that this is related to the parser that BS will use to read the HTML. They document is here, but if you’re like me (on OSX) you might be stuck with something that requires a bit of work: You’ll notice that in the BS4 documentation page above, they point out that … Read more
I was getting the same UnicodeEncodeError when saving scraped web content to a file. To fix it I replaced this code: with open(fname, “w”) as f: f.write(html) with this: with open(fname, “w”, encoding=”utf-8″) as f: f.write(html) If you need to support Python 2, then use this: import io with io.open(fname, “w”, encoding=”utf-8″) as f: f.write(html) … Read more
You can refine your search to only find those divs with a given class using BS3: mydivs = soup.find_all(“div”, {“class”: “stylelistrow”})
You need to read the Python Unicode HOWTO. This error is the very first example. Basically, stop using str to convert from unicode to encoded text / bytes. Instead, properly use .encode() to encode the string: p.agent_info = u’ ‘.join((agent_contact, agent_telno)).encode(‘utf-8’).strip() or work entirely in unicode.