You can set the user agent from an alias a = Mechanize.new a.user_agent_alias=”Mac Safari” Available aliases are stored in the AGENT_ALIASES constant. p Mechanize::AGENT_ALIASES Otherwise, use #user_agent to set your custom user agent. a = Mechanize.new a.user_agent=”Custom agent”
Try: br.select_form(nr=0) to select the first form In Mechanize source, def select_form(self, name=None, predicate=None, <b>nr=None</b>): “”” … nr, if supplied, is the sequence number of the form (where 0 is the first). “””
#!/usr/bin/ruby1.8 require ‘nokogiri’ require ‘pp’ html = <<-EOS (The HTML from the question goes here) EOS doc = Nokogiri::HTML(html) rows = doc.xpath(‘//table/tbody[@id=”threadbits_forum_251″]/tr’) details = rows.collect do |row| detail = {} [ [:title, ‘td[3]/div[1]/a/text()’], [:name, ‘td[3]/div[2]/span/a/text()’], [:date, ‘td[4]/text()’], [:time, ‘td[4]/span/text()’], [:number, ‘td[5]/a/text()’], [:views, ‘td[6]/text()’], ].each do |name, xpath| detail[name] = row.at_xpath(xpath).to_s.strip end detail end pp details … Read more
The link you included in your question is simply a wrapper that executes urllib’s read() function, which catches any incomplete read exceptions for you. If you don’t want to implement this entire patch, you could always just throw in a try/catch loop where you read your links. For example: try: page = urllib2.urlopen(urls).read() except httplib.IncompleteRead, … Read more
oh you need to ignore the robots.txt br = mechanize.Browser() br.set_handle_robots(False)
I think this talk (at pycon 2009), has the answers for what you’re looking for (Asheesh Laroia has lots of experience on the matter). And he points out the good and the bad from most of your listing Scrape the Web: Strategies for programming websites that don’t expect it (Part 1 of 3) Scrape the … Read more
Receiving a status 429 is not an error, it is the other server “kindly” asking you to please stop spamming requests. Obviously, your rate of requests has been too high and the server is not willing to accept this. You should not seek to “dodge” this, or even try to circumvent server security settings by … Read more
This is working as documented. Any paths specified in PYTHONPATH are documented as normally coming after the working directory but before the standard interpreter-supplied paths. sys.path.append() appends to the existing path. See here and here. If you want a particular directory to come first, simply insert it at the head of sys.path: import sys sys.path.insert(0,’/path/to/mod_directory’) … Read more