There are a number of ways to do it, but, none will always work. Here are the two easiest: if it’s a known finite set of websites: in your scraper convert each url from the normal url to the print url for a given site (cannot really be generalized across sites) Use the arc90 readability … Read more
If you are fluent with BeautifulSoup, you could just add soupselect to your libs. Soupselect is a CSS selector extension for BeautifulSoup. Usage: from bs4 import BeautifulSoup as Soup from soupselect import select import urllib soup = Soup(urllib.urlopen(‘http://slashdot.org/’)) select(soup, ‘div.title h3’) [<h3><span><a href=”https://science.slashdot.org/”>Science</a>:</span></h3>, <h3><a href=”https://slashdot.org/articles/07/02/28/0120220.shtml”>Star Trek</h3>, ..]
Language: JavaScript Library: jQuery $.each($(‘a[href]’), function(){ console.debug(this.href); }); (using firebug console.debug for output…) And loading any html page: $.get(‘http://stackoverflow.com/’, function(page){ $(page).find(‘a[href]’).each(function(){ console.debug(this.href); }); }); Used another each function for this one, I think it’s cleaner when chaining methods.
Always choose WHATWG over W3C, no exceptions. Anne van Kesteren, (a WHATWG member who was a major contributor to the the HTML specification prior to the WHATWG and W3C versions diverging, and who remains a major contributor to the WHATWG specification) describes the current situation between WHATWG and W3C as follows on his blog: The … Read more
First, your task fits into the information extraction area of research. There are mainly 2 levels of complexity for this task: extract from a given html page or a website with the fixed template (like Amazon). In this case the best way is to look at the HTML code of the pages and craft the … Read more
you can do this soup.findAll(True, {‘class’:[‘class1’, ‘class2’]}) example: >>> from bs4 import BeautifulSoup >>> soup = BeautifulSoup(‘<html><body><div class=”class1″></div><div class=”class2″></div><div class=”class3″></div></body></html>’) >>> soup.findAll(True, {“class”:[“class1”, “class2″]}) [<div class=”class1″></div>, <div class=”class2”></div>]
JavaScript doesn’t actually see a web page in the form of text-based HTML, but rather as a tree structure known as the DOM, or Document Object Model. The order of HTML element attributes in the DOM is not defined (in fact, as Svend comments, they’re not even part of the DOM), so the idea of … Read more
Just get the string representation: html_content = str(listing) This is a non-prettified version. If you want a prettified one, use prettify() method: html_content = listing.prettify()
So how does a HTML parser work? Doesn’t it use regular expressions to parse? Well, no. If you reach back in your brain to a theory of computation course, if you took one, or a compilers course, or something similar, you may recall that there are different kinds of languages and computational models. I’m not … Read more
You can use the npm modules jsdom and htmlparser to create and parse a DOM in Node.JS. Other options include: BeautifulSoup for python you can convert you html to xhtml and use XSLT HTMLAgilityPack for .NET CsQuery for .NET (my new favorite) The spidermonkey and rhino JS engines have native E4X support. This may be … Read more