Updated answer for 2015 & beyond: import ( “encoding/xml” “golang.org/x/net/html/charset” ) reader := bytes.NewReader(theXml) decoder := xml.NewDecoder(reader) decoder.CharsetReader = charset.NewReaderLabel err = decoder.Decode(&parsed)
To read any file in the UTF-8 encoding it must be created in the UTF-8 before. Use some editor which supports encoding switching. Or create that file from IDE with encoding option for properties files. E.g. IDEA: http://blog.jetbrains.com/idea/2013/03/use-the-utf-8-luke-file-encodings-in-intellij-idea/
Java used UCS-2 before transitioning over UTF-16 in 2004/2005. The reason for the original choice of UCS-2 is mainly historical: Unicode was originally designed as a fixed-width 16-bit character encoding. The primitive data type char in the Java programming language was intended to take advantage of this design by providing a simple data type that … Read more
First, let’s demonstrate that head is actually working correctly: $ printf ‘\xef\xbb\xbf’ >file $ head -c 3 file $ head -c 3 file | hexdump -C 00000000 ef bb bf |…| 00000003 Now, let’s create a working function has_bom. If your grep supports -P, then one option is: $ has_bom() { head -c3 “$1” | … Read more
Nope, all it’s there for is to tell the browser which character set to decode your response with.
East Asian languages typically require less storage in UTF-16 (2 bytes is enough for 99% of East-Asian language characters) than UTF-8 (typically 3 bytes is required). Of course, for Western lanagues, UTF-8 is usually smaller (1 byte instead of 2). For mixed files like HTML (where there’s a lot of markup) it’s much of a … Read more
In your case this page has wrong utf-8 data which confuses BeautifulSoup and makes it think that your page uses windows-1252, you can do this trick: soup = BeautifulSoup.BeautifulSoup(content.decode(‘utf-8′,’ignore’)) by doing this you will discard any wrong symbols from the page source and BeautifulSoup will guess the encoding correctly. You can replace ‘ignore’ by ‘replace’ … Read more
Open the file in Notepad++ and click Encoding->Convert to UTF-8. (Do not click Encode in UTF-8 because it won’t actually convert the characters.)