\xa0 is actually non-breaking space in Latin1 (ISO 8859-1), also chr(160). You should replace it with a space. string = string.replace(u’\xa0′, u’ ‘) When .encode(‘utf-8’), it will encode the unicode to utf-8, that means every unicode could be represented by 1 to 4 bytes. For this case, \xa0 is represented by 2 bytes \xc2\xa0. Read … Read more
The data is UTF-8 encoded bytes escaped with URL quoting, so you want to decode, with urllib.parse.unquote(), which handles decoding from percent-encoded data to UTF-8 bytes and then to text, transparently: from urllib.parse import unquote url = unquote(url) Demo: >>> from urllib.parse import unquote >>> url=”example.com?title=%D0%BF%D1%80%D0%B0%D0%B2%D0%BE%D0%B2%D0%B0%D1%8F+%D0%B7%D0%B0%D1%89%D0%B8%D1%82%D0%B0″ >>> unquote(url) ‘example.com?title=правовая+защита’ The Python 2 equivalent is urllib.unquote(), … Read more
Answering myself as the FAQ of this site encourages it. This works for me: Mostly characters äåö are not a problematic as the default character set used by browsers and tomcat/java for webapps is latin1 ie. ISO-8859-1 which “understands” those characters. To get UTF-8 working under Java+Tomcat+Linux/Windows+Mysql requires the following: Configuring Tomcat’s server.xml It’s necessary … Read more
Rather than mess with .encode and .decode, specify the encoding when opening the file. The io module, added in Python 2.6, provides an io.open function, which allows specifying the file’s encoding. Supposing the file is encoded in UTF-8, we can use: >>> import io >>> f = io.open(“test”, mode=”r”, encoding=”utf-8″) Then f.read returns a decoded … Read more
Unfortunately, the file.encoding property has to be specified as the JVM starts up; by the time your main method is entered, the character encoding used by String.getBytes() and the default constructors of InputStreamReader and OutputStreamWriter has been permanently cached. As Edward Grech points out, in a special case like this, the environment variable JAVA_TOOL_OPTIONS can … Read more
In Python 3, UTF-8 is the default source encoding (see PEP 3120), so Unicode characters can be used anywhere. In Python 2, you can declare in the source code header: # -*- coding: utf-8 -*- …. This is described in PEP 0263. Then you can use UTF-8 in strings: # -*- coding: utf-8 -*- u … Read more
UTF-8 is a variable-length encoding. In the case of UTF-8, this means that storing one code point requires one to four bytes. However, MySQL’s encoding called “utf8” (alias of “utf8mb3”) only stores a maximum of three bytes per code point. So the character set “utf8″/”utf8mb3” cannot store all Unicode code points: it only supports the … Read more
Why do we need Unicode? In the (not too) early days, all that existed was ASCII. This was okay, as all that would ever be needed were a few control characters, punctuation, numbers and letters like the ones in this sentence. Unfortunately, today’s strange world of global intercommunication and social media was not foreseen, and … Read more
UTF-8 is a multibyte encoding that can represent any Unicode character. ISO 8859-1 is a single-byte encoding that can represent the first 256 Unicode characters. Both encode ASCII exactly the same way.
The UTF-8 BOM is a sequence of bytes at the start of a text stream (0xEF, 0xBB, 0xBF) that allows the reader to more reliably guess a file as being encoded in UTF-8. Normally, the BOM is used to signal the endianness of an encoding, but since endianness is irrelevant to UTF-8, the BOM is … Read more