Deprecated is one thing, might become deprecated at some point in the future is another. If it suits your needs, I’d continuing using urlretrieve. That said, you can use shutil.copyfileobj: from urllib.request import urlopen from shutil import copyfileobj with urlopen(my_url) as in_stream, open(‘my_filename’, ‘wb’) as out_file: copyfileobj(in_stream, out_file)
I’ve found a minimalistic solution to force urrlib3 to use either ipv4 or ipv6. This method is used by urrlib3 for creating new connection both for Http and Https. You can specify in it any AF_FAMILY you want to use. import socket import requests.packages.urllib3.util.connection as urllib3_cn def allowed_gai_family(): “”” https://github.com/shazow/urllib3/blob/master/urllib3/util/connection.py “”” family = socket.AF_INET if … Read more
I had this same problem. The verify parameter refers to the server’s certificate. You want the cert parameter to specify your client certificate. import requests cert_file_path = “cert.pem” key_file_path = “key.pem” url = “https://example.com/resource” params = {“param_1”: “value_1”, “param_2”: “value_2”} cert = (cert_file_path, key_file_path) r = requests.get(url, params=params, cert=cert)
Quick Answer Option 1 import requests s = requests.session() s.cookies.set(“COOKIE_NAME”, “the cookie works”, domain=”example.com”) Option 2 import requests s = requests.session() # Note that domain keyword parameter is the only optional parameter here cookie_obj = requests.cookies.create_cookie(domain=”example.com”,name=”COOKIE_NAME”,value=”the cookie works”) s.cookies.set_cookie(cookie_obj) Detailed Answer I do not know if this technique was valid when the original question was … Read more
As @maak pointed out, you need to first send a message to the bot before the bot can send messages to you.
Install the grequests module which works with gevent (requests is not designed for async): pip install grequests Then change the code to something like this: import grequests class Test: def __init__(self): self.urls = [ ‘http://www.example.com’, ‘http://www.google.com’, ‘http://www.yahoo.com’, ‘http://www.stackoverflow.com/’, ‘http://www.reddit.com/’ ] def exception(self, request, exception): print “Problem: {}: {}”.format(request.url, exception) def async(self): results = grequests.map((grequests.get(u) for … Read more
In urllib3 POST is not allowed as a retried method by default (since it can cause multiple inserts). You can force it though: Retry(total=3, allowed_methods=frozenset([‘GET’, ‘POST’])) See https://urllib3.readthedocs.io/en/latest/reference/urllib3.util.html#urllib3.util.retry.Retry
You need to send a complete form. The easiest way to find out what Facebook expects is to use something like Google Chrome’s developer tools to monitor your web requests. To make your life easier I’ve monitored my own login on Facebook, and reproduced it below (with private information redacted, obviously) with the unimportant information … Read more
The simplest way is to “shim” the session’s request function: import requests import functools s = requests.Session() s.request = functools.partial(s.request, timeout=3) # now all get, post, head etc requests should timeout after 3 seconds # following will fail s.get(‘https://httpbin.org/delay/6’) # we can still pass higher timeout when needed # following will succeed s.get(‘https://httpbin.org/delay/6’, timeout=7)
requests handles redirects for you, see redirection and history. Set allow_redirects=False if you don’t want requests to handle redirections, or you can inspect the redirection responses contained in the r.history list. Demo: >>> import requests >>> url=”https://httpbin.org/redirect-to” >>> params = {“status_code”: 301, “url”: “https://stackoverflow.com/q/22150023”} >>> r = requests.get(url, params=params) >>> r.history [<Response [301]>, <Response [302]>] … Read more