I’ve done some of this recently, and here are my experiences. There are three basic approaches: Regular Expressions. Most flexible, easiest to use with loosely-structured info and changing formats. Harder to do structural/tag analysis, but easier to do text matching. Built in validation of data formatting. Harder to maintain than others, because you have to … Read more
Html Agility Pack This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don’t HAVE to understand XPATH nor XSLT to use it, don’t worry…). It is a .NET code library that allows you to parse “out of the web” HTML files. The parser is very … Read more
I found using hpple quite useful to parse messy HTML. Hpple project is a Objective-C wrapper on the XPathQuery library for parsing HTML. Using it you can send an XPath query and receive the result . Requirements: -Add libxml2 includes to your project Menu Project->Edit Project Settings Search for setting “Header Search Paths” Add a … Read more
from BeautifulSoup import BeautifulSoup import re html_text = “”” <h2>this is cool #12345678901</h2> <h2>this is nothing</h2> <h1>foo #126666678901</h1> <h2>this is interesting #126666678901</h2> <h2>this is blah #124445678901</h2> “”” soup = BeautifulSoup(html_text) for elem in soup(text=re.compile(r’ #\S{11}’)): print elem.parent Prints: <h2>this is cool #12345678901</h2> <h2>this is interesting #126666678901</h2> <h2>this is blah #124445678901</h2>
Html Agility Pack This is an agile HTML parser that builds a read/write DOM and supports plain XPATH or XSLT (you actually don’t HAVE to understand XPATH nor XSLT to use it, don’t worry…). It is a .NET code library that allows you to parse “out of the web” HTML files. The parser is very … Read more
Try this: from bs4 import BeautifulSoup from bs4.element import Comment import urllib.request def tag_visible(element): if element.parent.name in [‘style’, ‘script’, ‘head’, ‘title’, ‘meta’, ‘[document]’]: return False if isinstance(element, Comment): return False return True def text_from_html(body): soup = BeautifulSoup(body, ‘html.parser’) texts = soup.findAll(text=True) visible_texts = filter(tag_visible, texts) return u” “.join(t.strip() for t in visible_texts) html = urllib.request.urlopen(‘http://www.nytimes.com/2009/12/21/us/21storm.html’).read() … Read more
$url=”http://example.com”; $html = file_get_contents($url); $doc = new DOMDocument(); @$doc->loadHTML($html); $tags = $doc->getElementsByTagName(‘img’); foreach ($tags as $tag) { echo $tag->getAttribute(‘src’); }
Use ( ) in regexp and group(1) in python to retrieve the captured string (re.search will return None if it doesn’t find the result, so don’t use group() directly): title_search = re.search(‘<title>(.*)</title>’, html, re.IGNORECASE) if title_search: title = title_search.group(1)
The best piece of code I found for extracting text without getting javascript or not wanted things : from urllib.request import urlopen from bs4 import BeautifulSoup url = “http://news.bbc.co.uk/2/hi/health/2284783.stm” html = urlopen(url).read() soup = BeautifulSoup(html, features=”html.parser”) # kill all script and style elements for script in soup([“script”, “style”]): script.extract() # rip it out # get … Read more