Strictly speaking URIs can’t contain non-ASCII characters; what you have there is an IRI. To convert an IRI to a plain ASCII URI: non-ASCII characters in the hostname part of the address have to be encoded using the Punycode-based IDNA algorithm; non-ASCII characters in the path, and most of the other parts of the address … Read more
From the docs Note that params output from urlencode is encoded to bytes before it is sent to urlopen as data: data = urllib.parse.urlencode(d).encode(“utf-8”) req = urllib.request.Request(url) with urllib.request.urlopen(req,data=data) as f: resp = f.read() print(resp)
No reason to work line by line (small chunks AND requires Python to find the line ends for you!-), just chunk it up in bigger chunks, e.g.: # from urllib2 import urlopen # Python 2 from urllib.request import urlopen # Python 3 response = urlopen(url) CHUNK = 16 * 1024 with open(file, ‘wb’) as f: … Read more
Wikipedias stance is: Data retrieval: Bots may not be used to retrieve bulk content for any use not directly related to an approved bot task. This includes dynamically loading pages from another website, which may result in the website being blacklisted and permanently denied access. If you would like to download bulk content or mirror … Read more
See this thread How to send JSON as part of multipart POST-request Do not set the Content-type header yourself, leave that to pyrequests to generate def send_request(): payload = {“param_1”: “value_1”, “param_2”: “value_2”} files = { ‘json’: (None, json.dumps(payload), ‘application/json’), ‘file’: (os.path.basename(file), open(file, ‘rb’), ‘application/octet-stream’) } r = requests.post(url, files=files) print(r.content)
Do it in stages, and modify the object, like this: # make a string with the request type in it: method = “POST” # create a handler. you can specify different handlers here (file uploads etc) # but we go for the default handler = urllib2.HTTPHandler() # create an openerdirector instance opener = urllib2.build_opener(handler) # … Read more
In the meantime urllib2 seems to verify server certificates by default. The warning, that was shown in the past disappeared for 2.7.9 and I currently ran into this problem in a test environment with a self signed certificate (and Python 2.7.9). My evil workaround (don’t do this in production!): import urllib2 import ssl ctx = … Read more
There are very few cases where you want to use except:. Doing this captures any exception, which can be hard to debug, and it captures exceptions including SystemExit and KeyboardInterupt, which can make your program annoying to use.. At the very simplest, you would catch urllib2.URLError: try: urllib2.urlopen(“http://example.com”, timeout = 1) except urllib2.URLError, e: raise … Read more
Not quite. Creating a Request object does not actually send the request, and Request objects have no Read() method. (Also: read() is lowercase.) All you need to do is pass the Request as the first argument to urlopen() and that will give you your response. import urllib2 request = urllib2.Request(“http://www.google.com”, headers={“Accept” : “text/html”}) contents = … Read more
I think a more reliable way of closing a connection is to tell the sever explicitly to close it in a way compliant with HTTP specification: HTTP/1.1 defines the “close” connection option for the sender to signal that the connection will be closed after completion of the response. For example, Connection: close in either the … Read more