I noticed the exact same issue when logging onto servers running Red Hat from an OSX Lion machine. Try adding or editing the ~/.profile file for it to correctly export your locale settings upon initiating a new session. export LC_ALL=en_US.UTF-8 export LANG=en_US.UTF-8 These two lines added to the file should suffice to set the locale … Read more
Three words for you: Byte Order Mark (BOM) That’s the representation for the UTF-8 BOM in ISO-8859-1. You have to tell your editor to not use BOMs or use a different editor to strip them out. To automatize the BOM’s removal you can use awk as shown in this question. As another answer says, the … Read more
What you’re asking for is extremely hard. If possible, getting the user to specify the encoding is the best. Preventing an attack shouldn’t be much easier or harder that way. However, you could try doing this: iconv(mb_detect_encoding($text, mb_detect_order(), true), “UTF-8”, $text); Setting it to strict might help you get a better result.
So what’s the problem, It’s a ’ (RIGHT SINGLE QUOTATION MARK – U+2019) character which is being decoded as CP-1252 instead of UTF-8. If you check the encodings table, then you see that this character is in UTF-8 composed of bytes 0xE2, 0x80 and 0x99. If you check the CP-1252 code page layout, then you’ll … Read more
None of these answers solved the problem for me. The root cause being: You cannot store 4-byte characters in MySQL with the utf-8 character set. MySQL has a 3 byte limit on utf-8 characters (yes, it’s wack, nicely summed up by a Django developer here) To solve this you need to: Change your MySQL database, … Read more
Here is a simpler method (hack) that gives you back the setdefaultencoding() function that was deleted from sys: import sys # sys.setdefaultencoding() does not exist, here! reload(sys) # Reload does the trick! sys.setdefaultencoding(‘UTF8’) (Note for Python 3.4+: reload() is in the importlib library.) This is not a safe thing to do, though: this is obviously … Read more
As you know the string is coming in as Encoding.Default you could simply use: byte[] bytes = Encoding.Default.GetBytes(myString); myString = Encoding.UTF8.GetString(bytes); Another thing you may have to remember: If you are using Console.WriteLine to output some strings, then you should also write Console.OutputEncoding = System.Text.Encoding.UTF8;!!! Or all utf8 strings will be outputed as gbk…
As per the documentation: This allows you to switch from the default ASCII to other encodings such as UTF-8, which the Python runtime will use whenever it has to decode a string buffer to unicode. This function is only available at Python start-up time, when Python scans the environment. It has to be called in … Read more
Instead of using FileWriter, create a FileOutputStream. You can then wrap this in an OutputStreamWriter, which allows you to pass an encoding in the constructor. Then you can write your data to that inside a try-with-resources Statement: try (OutputStreamWriter writer = new OutputStreamWriter(new FileOutputStream(PROPERTIES_FILE), StandardCharsets.UTF_8)) // do stuff }
>>> u’aあä’.encode(‘ascii’, ‘ignore’) ‘a’ Decode the string you get back, using either the charset in the the appropriate meta tag in the response or in the Content-Type header, then encode. The method encode(encoding, errors) accepts custom handlers for errors. The default values, besides ignore, are: >>> u’aあä’.encode(‘ascii’, ‘replace’) b’a??’ >>> u’aあä’.encode(‘ascii’, ‘xmlcharrefreplace’) b’aあä’ >>> u’aあä’.encode(‘ascii’, … Read more