Python 3 urllib.parse.urlencode(query, doseq=False, […]) Convert a mapping object or a sequence of two-element tuples, which may contain str or bytes objects, to a percent-encoded ASCII text string. — Python 3 urllib.parse docs A dict is a mapping. Legacy Python urllib.urlencode(query[, doseq]) Convert a mapping object or a sequence of two-element tuples to a “percent-encoded” … Read more
In Python 3.x, you need to import urllib.parse.quote: >>> import urllib.parse >>> urllib.parse.quote(“châteu”, safe=””) ‘ch%C3%A2teu’ According to Python 2.x urllib module documentation: NOTE The urllib module has been split into parts and renamed in Python 3 to urllib.request, urllib.parse, and urllib.error.
Nope, BeautifulSoup, by itself, does not support XPath expressions. An alternative library, lxml, does support XPath 1.0. It has a BeautifulSoup compatible mode where it’ll try and parse broken HTML the way Soup does. However, the default lxml HTML parser does just as good a job of parsing broken HTML, and I believe is faster. … Read more
This works in Python 2.x. For Python 3 look in the docs: import urllib.request with urllib.request.urlopen(“http://www.python.org”) as url: s = url.read() # I’m guessing this would output the html source code ? print(s)
Python 2 Using urllib.urlretrieve import urllib urllib.urlretrieve(“http://www.gunnerkrigg.com//comics/00000001.jpg”, “00000001.jpg”) Python 3 Using urllib.request.urlretrieve (part of Python 3’s legacy interface, works exactly the same) import urllib.request urllib.request.urlretrieve(“http://www.gunnerkrigg.com//comics/00000001.jpg”, “00000001.jpg”)
If you really want to handle with HTTP using Python, I highly recommend Requests: HTTP for Humans. The POST quickstart adapted to your question is: >>> import requests >>> r = requests.post(“http://bugs.python.org”, data={‘number’: ‘12524’, ‘type’: ‘issue’, ‘action’: ‘show’}) >>> print(r.status_code, r.reason) 200 OK >>> print(r.text[:300] + ‘…’) <!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Transitional//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd”> … Read more
Python 2 From the documentation: urllib.quote(string[, safe]) Replace special characters in string using the %xx escape. Letters, digits, and the characters ‘_.-‘ are never quoted. By default, this function is intended for quoting the path section of the URL.The optional safe parameter specifies additional characters that should not be quoted — its default value is … Read more
This isn’t a solution to your specific problem, but I’m putting it here because this thread is the top Google result for “SSL: CERTIFICATE_VERIFY_FAILED”, and it lead me on a wild goose chase. If you have installed Python 3.6 on OSX and are getting the “SSL: CERTIFICATE_VERIFY_FAILED” error when trying to connect to an https:// … Read more
I was getting the same UnicodeEncodeError when saving scraped web content to a file. To fix it I replaced this code: with open(fname, “w”) as f: f.write(html) with this: with open(fname, “w”, encoding=”utf-8″) as f: f.write(html) If you need to support Python 2, then use this: import io with io.open(fname, “w”, encoding=”utf-8″) as f: f.write(html) … Read more
Python 2 What you’re looking for is urllib.quote_plus: safe_string = urllib.quote_plus(‘string_of_characters_like_these:$#@=?%^Q^$’) #Value: ‘string_of_characters_like_these%3A%24%23%40%3D%3F%25%5EQ%5E%24′ Python 3 In Python 3, the urllib package has been broken into smaller components. You’ll use urllib.parse.quote_plus (note the parse child module) import urllib.parse safe_string = urllib.parse.quote_plus(…)