web-scraping

Python Selenium Timeout Exception Catch

by

Almost of your code is working fine except the Driver.Close(). It should be Driver.close(). The TimeoutException will be thrown when the page is not loaded within specific time. See my code below: from selenium import webdriver from selenium.common.exceptions import TimeoutException Driver = webdriver.Firefox() try: Driver.set_page_load_timeout(1) Driver.get(“http://www.engadget.com”) except TimeoutException as ex: isrunning = 0 print(“Exception has … Read more

Beautiful Soup if Class “Contains” or Regex?

by

BeautifulSoup supports CSS selectors which allow you to select elements based on the content of particular attributes. This includes the selector *= for contains. The following will return all div elements with a class attribute containing the text ‘listing-col-‘: for EachPart in soup.select(‘div[class*=”listing-col-“]’): print EachPart.get_text()

How can I use Python’s Requests to fake a browser visit a.k.a and generate User Agent? [duplicate]

by

Provide a User-Agent header: import requests url=”http://www.ichangtou.com/#company:data_000008.html” headers = {‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36’} response = requests.get(url, headers=headers) print(response.content) FYI, here is a list of User-Agent strings for different browsers: List of all Browsers As a side note, there is a pretty useful third-party package called … Read more

How can I download images on a page using puppeteer?

by

If you want to skip the manual dom traversal you can write the images to disk directly from the page response. Example: const puppeteer = require(‘puppeteer’); const fs = require(‘fs’); const path = require(‘path’); (async () => { const browser = await puppeteer.launch(); const page = await browser.newPage(); page.on(‘response’, async response => { const url … Read more

Scrape a webpage and navigate by clicking buttons

by

Zombie.js and Node.io run on JSDOM, hence your options are either going with JSDOM (or any equivalent wrapper), a headless browser (PhantomJS, SlimerJS) or Cheerio. JSDOM is fairly slow because it has to recreate DOM and CSSOM in Node.js. PhantomJS/SlimerJS are proper headless browsers, thus performances are ok and those are also very reliable. Cheerio … Read more

Is it possible to use Selenium WebDriver to drive PhantomJS?

by

PhantomJS now includes the GhostDriver project. You are also suggested to use PhantomJS directly or with a convenience library such as CasperJS. CasperJS is specifically designed to make it easy to do sequential operations to web pages, perfect for many automation tasks. Disclaimer: I am the author of PhantomJS. Edit: As noted in Nick’s answer, … Read more