Messa’s answer only works if the server isn’t bothering to check the content-type header. You’ll need to specify a content-type header if you want it to really work. Here’s Messa’s answer modified to include a content-type header: import json import urllib2 data = json.dumps([1, 2, 3]) req = urllib2.Request(url, data, {‘Content-Type’: ‘application/json’}) f = urllib2.urlopen(req) … Read more
I think this talk (at pycon 2009), has the answers for what you’re looking for (Asheesh Laroia has lots of experience on the matter). And he points out the good and the bad from most of your listing Scrape the Web: Strategies for programming websites that don’t expect it (Part 1 of 3) Scrape the … Read more
The HTTPError is a file-like object. You can catch it and then read its contents. try: resp = urllib2.urlopen(url) contents = resp.read() except urllib2.HTTPError, error: contents = error.read()
Basically what you do is correct. Looking at redmine docs you linked to, it seems that suffix after the dot in the url denotes type of posted data (.json for JSON, .xml for XML), which agrees with the response you get – Processing by AttachmentsController#upload as XML. I guess maybe there’s a bug in docs … Read more
How can I tell if the data at a URL is gzipped? This checks if the content is gzipped and decompresses it: from StringIO import StringIO import gzip request = urllib2.Request(‘http://example.com/’) request.add_header(‘Accept-encoding’, ‘gzip’) response = urllib2.urlopen(request) if response.info().get(‘Content-Encoding’) == ‘gzip’: buf = StringIO(response.read()) f = gzip.GzipFile(fileobj=buf) data = f.read() Does urllib2 automatically uncompress the data … Read more
proxy = urllib2.ProxyHandler({‘http’: ‘127.0.0.1’}) opener = urllib2.build_opener(proxy) urllib2.install_opener(opener) urllib2.urlopen(‘http://www.google.com’)
response.read() returns an instance of bytes while StringIO is an in-memory stream for text only. Use BytesIO instead. From What’s new in Python 3.0 – Text Vs. Data Instead Of Unicode Vs. 8-bit The StringIO and cStringIO modules are gone. Instead, import the io module and use io.StringIO or io.BytesIO for text and data respectively.
The problem could be that the Python libraries, per HTTP-Standard, first send an unauthenticated request, and then only if it’s answered with a 401 retry, are the correct credentials sent. If the Foursquare servers don’t do “totally standard authentication” then the libraries won’t work. Try using headers to do authentication: import urllib2, base64 request = … Read more
Cookie is just another HTTP header. import urllib2 opener = urllib2.build_opener() opener.addheaders.append((‘Cookie’, ‘cookiename=cookievalue’)) f = opener.open(“http://example.com/”) See urllib2 examples for other ways how to add HTTP headers to your request. There are more ways how to handle cookies. Some modules like cookielib try to behave like web browser – remember what cookies did you get … Read more
Python 3 from urllib.error import HTTPError Python 2 from urllib2 import HTTPError Just catch HTTPError, handle it, and if it’s not Error 404, simply use raise to re-raise the exception. See the Python tutorial. Here is a complete example for Python 2: import urllib2 from urllib2 import HTTPError try: urllib2.urlopen(“some url”) except HTTPError as err: … Read more