web-scraping
Python follow redirects and then download the page?
Use requests as the other answer states, here is an example. The redirect will be in r.url. In the example below the http is redirected to https For HEAD: In [1]: import requests …: r = requests.head(‘http://github.com’, allow_redirects=True) …: r.url Out[1]: ‘https://github.com/’ For GET: In [1]: import requests …: r = requests.get(‘http://github.com’) …: r.url Out[1]: … Read more
Headless browser for C# (.NET)? [closed]
There are some options: WebKit.Net (free) Awesomium It is based on Chrome/WebKit and works like a charm. There is a free license available but also a commercial one and if need be you can buy the source code 🙂 HTML Agility Pack (free) (An HTML Parser library, NOT a headless browser) This helps with extracting … Read more
Wait page to load before getting data with requests.get in python 3
It doesn’t look like a problem of waiting, it looks like the element is being created by JavaScript, requests can’t handle dynamically generated elements by JavaScript. A suggestion is to use selenium together with PhantomJS to get the page source, then you can use BeautifulSoup for your parsing, the code shown below will do exactly … Read more
“TypeError: object of type ‘Response’ has no len()”
You are getting response.content. But it return response body as bytes (docs). But you should pass str to BeautifulSoup constructor (docs). So you need to use the response.text instead of getting content.
How do I call a Javascript function from Python?
Find a JavaScript interpreter that has Python bindings. (Try Rhino? V8? SeaMonkey?). When you have found one, it should come with examples of how to use it from python. Python itself, however, does not include a JavaScript interpreter.
Scrape tables into dataframe with BeautifulSoup
Pandas already has a built-in method to convert the table on the web to a dataframe: table = soup.find_all(‘table’) df = pd.read_html(str(table))[0]
How can I scrape a page with dynamic content (created by JavaScript) in Python?
EDIT Sept 2021: phantomjs isn’t maintained any more, either EDIT 30/Dec/2017: This answer appears in top results of Google searches, so I decided to update it. The old answer is still at the end. dryscape isn’t maintained anymore and the library dryscape developers recommend is Python 2 only. I have found using Selenium’s python library … Read more
How does reCAPTCHA 3 know I’m using Selenium/chromedriver?
reCaptcha Websites can easily detect the network traffic and identify your program as a BOT. Google have already released 5(five) reCAPTCHA to choose from when creating a new site. While four of them are active and reCAPTCHA v1 being shutdown. reCAPTCHA versions and types reCAPTCHA v3 (verify requests with a score): reCAPTCHA v3 allows you … Read more
Page content is loaded with JavaScript and Jsoup doesn’t see it
JSoup is an HTML parser, not some kind of embedded browser engine. This means that it’s completely unaware of any content that is added to the DOM by Javascript after the initial page load. To get access to that type of content you will need an embedded browser component, there are a number of discussions … Read more