When you login to the site, it is probably setting an authorised session cookie that needs to be sent on subsequent requests to maintain the session. You can get the cookie like this: Connection.Response res = Jsoup.connect(“http://www.example.com/login.php”) .data(“username”, “myUsername”, “password”, “myPassword”) .method(Method.POST) .execute(); Document doc = res.parse(); String sessionId = res.cookie(“SESSIONID”); // you will need … Read more
oh you need to ignore the robots.txt br = mechanize.Browser() br.set_handle_robots(False)
I use webkit as a headless browser in Python via pyqt / pyside: http://www.riverbankcomputing.co.uk/software/pyqt/download http://developer.qt.nokia.com/wiki/Category:LanguageBindings::PySide::Downloads I particularly like webkit because it is simple to setup. For Ubuntu you just use: sudo apt-get install python-qt4 Here is an example script: http://webscraping.com/blog/Scraping-JavaScript-webpages-with-webkit/
This is a very general question but a very nice topic! Definitely upvoted 🙂 However I am not satisfied with the answers provided so far, so I decided to write a rather lengthy answer on this. The reason I am not satisfied is that the answers are basically all true (I especially like the answer … Read more
Scraping generally encompasses 3 steps: first you GET or POST your request to a specified URL next you receive the html that is returned as the response finally you parse out of that html the text you’d like to scrape. To accomplish steps 1 and 2, below is a simple php class which uses Curl … Read more
Node.io seems to take the cake 🙂
You can also use Js2Py which is written in pure python and is able to both execute and translate javascript to python. Supports virtually whole JavaScript even labels, getters, setters and other rarely used features. import js2py js = “”” function escramble_758(){ var a,b,c a=”+1 ” b=’84-‘ a+=’425-‘ b+=’7450’ c=”9″ document.write(a+c+b) } escramble_758() “””.replace(“document.write”, “return … Read more
You can use the WebClient Using System.Net; using(WebClient client = new WebClient()) { string downloadString = client.DownloadString(“http://www.gooogle.com”); }
I’m working in travel industry as a software architect / project lead on the precisely kind of project you describe – in our region we work with suppliers directly, but for outgoing we connect to several aggregators. To answer your question… some data you have, some you get in various ways, and some you have … Read more
I tried Fred’s and Cameron Tinker’s answers, but only –ssl-protocol=any option seem to help me: phantomjs –ssl-protocol=any test.js Also I think it should be way safer to use –ssl-protocol=any as you still are using encryption, but –ignore-ssl-errors=true will ignore (duh) all ssl errors, including malicious ones.