python-requests – Page 3

Python ‘requests’ library – define specific DNS?

December 9, 2023 by Tarik

requests uses urllib3, which ultimately uses httplib.HTTPConnection as well, so the techniques from https://stackoverflow.com/questions/4623090/python-set-custom-dns-server-for-urllib-requests (now deleted, it merely linked to Tell urllib2 to use custom DNS) still apply, to a certain extent. The urllib3.connection module subclasses httplib.HTTPConnection under the same name, having replaced the .connect() method with one that calls self._new_conn. In turn, this delegates … Read more

openssl, python requests error: “certificate verify failed”

December 5, 2023 by Tarik

If I run the following command from my development box: $ openssl s_client -connect github.com:443 I get the following last line of output: Verify return code: 20 (unable to get local issuer certificate) You are missing DigiCert High Assurance EV CA-1 as a root of trust: $ openssl s_client -connect github.com:443 CONNECTED(00000003) depth=1 C = … Read more

Why does HTTP POST request body need to be JSON enconded in Python?

December 2, 2023 by Tarik

Apparently your API requires JSON-encoded and not form-encoded data. When you pass a dict in as the data parameter, the data is form-encoded. When you pass a string (like the result of json.dumps), the data is not form-encoded. Consider this quote from the requests documentation: Typically, you want to send some form-encoded data — much … Read more

Python requests – quickly know if response is json parsable

December 2, 2023 by Tarik

Depending on the consistency of the response, you could check if the returned headers include content-type application/json: resp.headers.get(‘content-type’) == ‘application/json’

Why do I receive a timeout error from Pythons requests module?

December 1, 2023 by Tarik

Per https://requests.readthedocs.io/en/latest/user/quickstart/#timeouts, that is the expected behavior. As royhowie mentioned, wrap it in a try/except block (e.g.: try: requests.post(url, headers, timeout=10) except requests.exceptions.Timeout: print “Timeout occurred” )

Why is Python 3 http.client so much faster than python-requests?

November 29, 2023 by Tarik

Based on profiling both, the main difference appears to be that the requests version is doing a DNS lookup for every request, while the http.client version is doing so once. # http.client ncalls tottime percall cumtime percall filename:lineno(function) 1974 0.541 0.000 0.541 0.000 {method ‘recv_into’ of ‘_socket.socket’ objects} 1000 0.020 0.000 0.045 0.000 feedparser.py:470(_parse_headers) 13000 … Read more

How do I get the IP address from a http request using the requests library?

November 28, 2023 by Tarik

It turns out that it’s rather involved. Here’s a monkey-patch while using requests version 1.2.3: Wrapping the _make_request method on HTTPConnectionPool to store the response from socket.getpeername() on the HTTPResponse instance. For me on python 2.7.3, this instance was available on response.raw._original_response. from requests.packages.urllib3.connectionpool import HTTPConnectionPool def _make_request(self,conn,method,url,**kwargs): response = self._old_make_request(conn,method,url,**kwargs) sock = getattr(conn,’sock’,False) if … Read more

URI encoding in Python Requests package

November 26, 2023 by Tarik

You can use requests.utils.quote (which is just a link to urllib.parse.quote) to convert your text to url encoded format. >>> import requests >>> requests.utils.quote(‘[email protected]’) ‘test%2Buser%40gmail.com’

python requests link headers

November 25, 2023 by Tarik

There is already a way provided by requests to access links header response.links It returns the dictionary of links header value which can easily parsed further using response.links[‘next’][‘url’] to get the required values.

How can I use Python’s Requests to fake a browser visit a.k.a and generate User Agent? [duplicate]

October 29, 2023 by Tarik

Provide a User-Agent header: import requests url=”http://www.ichangtou.com/#company:data_000008.html” headers = {‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36’} response = requests.get(url, headers=headers) print(response.content) FYI, here is a list of User-Agent strings for different browsers: List of all Browsers As a side note, there is a pretty useful third-party package called … Read more