It seems the page rejects GET requests that do not identify a User-Agent. I visited the page with a browser (Chrome) and copied the User-Agent header of the GET request (look in the Network tab of the developer tools): import requests url=”http://worldagnetwork.com/” headers = {‘User-Agent’: ‘Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_5) AppleWebKit/537.36 (KHTML, like … Read more
I could reproduce the error with a minimal example: foo.py: class MyError(Exception): pass class A: def inner(self): err = MyError(“FOO”) print(type(err)) raise err def outer(self): try: self.inner() except MyError as err: print (“catched “, err) return “OK” Test without mocking : class FooTest(unittest.TestCase): def test_inner(self): a = foo.A() self.assertRaises(foo.MyError, a.inner) def test_outer(self): a = foo.A() … Read more
Why does the former install 3 additional packages? Using requests[security] instead of requests will install three additional packages: pyOpenSSL cryptography idna These are defined in extras_requires, as optional features with additional dependencies. Are there any things that I need to take care about when I push the code to production? You’d want to make sure … Read more
This you won’t be able to do with requests and BeautifulSoup as the page that you want to extract the information from loads the rest of the entries through JS when you scroll down. You can do this using selenium which opens a real browser and you can pass page down key press events programmatically. … Read more
The solution is to use tuples when passing parameters to the files argument: import requests requests.post(‘http://example.com/example/asdfas’, files={‘value_1’: (None, ‘12345’), ‘value_2’: (None, ‘67890’)}) Works as expected: ‘Accept’: ‘*/*’, ‘Accept-Encoding’: ‘gzip, deflate, compress’, ‘Content-Length’: ‘228’, ‘User-Agent’: ‘python-requests/2.2.1 CPython/3.3.2 Windows/7’, ‘Content-Type’: ‘multipart/form-data; boundary=85e90a4bbb05474ca1e23dbebdd68ed9’ –85e90a4bbb05474ca1e23dbebdd68ed9 Content-Disposition: form-data; name=”value_1″ 12345 –85e90a4bbb05474ca1e23dbebdd68ed9 Content-Disposition: form-data; name=”value_2″ 67890 –85e90a4bbb05474ca1e23dbebdd68ed9–
No need to implement your own! in most cases Requests has built in support for proxies, for basic authentication: proxies = { ‘https’ : ‘https://user:password@proxyip:port’ } r = requests.get(‘https://url’, proxies=proxies) see more on the docs Or in case you need digest authentication HTTPDigestAuth may help. Or you might need try to extend it like yutaka2487 … Read more
This is the typical symptom of an unrelated requests.py (or requests.pyc) file sitting in your current directory, or somewhere else on the PYTHONPATH. If this is the case, remove or rename it, as it’s shadowing the module you really want to import.
You will lose time switching between reads and writes, and the limit of the chunk size is AFAIK only the limit of what you can store in memory. So as long as you aren’t very concerned about keeping memory usage down, go ahead and specify a large chunk size, such as 1 MB (e.g. 1024 … Read more
Looking at stack trace you’ve provided your error is caused by httplib.BadStatusLine exception, which, according to docs, is: Raised if a server responds with a HTTP status code that we don’t understand. In other words something that is returned (if returned at all) by proxy server cannot be parsed by httplib that does actual request. … Read more