JSoup is an HTML parser, not some kind of embedded browser engine. This means that it’s completely unaware of any content that is added to the DOM by Javascript after the initial page load. To get access to that type of content you will need an embedded browser component, there are a number of discussions … Read more
You can easily set proxy System.setProperty(“http.proxyHost”, “192.168.5.1”); System.setProperty(“http.proxyPort”, “1080”); Document doc = Jsoup.connect(“www.google.com”).get();
You might try setting the referrer header as well: doc = Jsoup.connect(“https://www.facebook.com/”) .userAgent(“Mozilla/5.0 (Windows; U; WindowsNT 5.1; en-US; rv1.8.1.6) Gecko/20070725 Firefox/2.0.0.6”) .referrer(“http://www.google.com”) .get();
With Jsoup: final String html = “<p> <span> foo </span> <em> bar <a> foobar </a> baz </em> </p>”; Document doc = Jsoup.parse(html); System.out.println(doc.text()); Output: foo bar foobar baz If you want only the text of p-tag, use this instead of doc.text(): doc.select(“p”).text(); … or only body: doc.body().text(); Linebreak: final String html = “<p><strong>Tarthatatlan biztonsági viszonyok</strong></p>” … Read more
When you login to the site, it is probably setting an authorised session cookie that needs to be sent on subsequent requests to maintain the session. You can get the cookie like this: Connection.Response res = Jsoup.connect(“http://www.example.com/login.php”) .data(“username”, “myUsername”, “password”, “myPassword”) .method(Method.POST) .execute(); Document doc = res.parse(); String sessionId = res.cookie(“SESSIONID”); // you will need … Read more
Use a HTML parser like Jsoup. This has my preference above the other HTML parsers available in Java since it supports jQuery like CSS selectors. Also, its class representing a list of nodes, Elements, implements Iterable so that you can iterate over it in an enhanced for loop (so there’s no need to hassle with … Read more
I think you can do Jsoup.connect(“…”).timeout(10 * 1000).get(); which sets timeout to 10s.
The real solution that preserves linebreaks should be like this: public static String br2nl(String html) { if(html==null) return html; Document document = Jsoup.parse(html); document.outputSettings(new Document.OutputSettings().prettyPrint(false));//makes html() preserve linebreaks and spacing document.select(“br”).append(“\\n”); document.select(“p”).prepend(“\\n\\n”); String s = document.html().replaceAll(“\\\\n”, “\n”); return Jsoup.clean(s, “”, Whitelist.none(), new Document.OutputSettings().prettyPrint(false)); } It satisfies the following requirements: if the original html contains newline(\n), … Read more