In Python 3, UTF-8 is the default source encoding (see PEP 3120), so Unicode characters can be used anywhere. In Python 2, you can declare in the source code header: # -*- coding: utf-8 -*- …. This is described in PEP 0263. Then you can use UTF-8 in strings: # -*- coding: utf-8 -*- u … Read more
UTF-8 is a variable-length encoding. In the case of UTF-8, this means that storing one code point requires one to four bytes. However, MySQL’s encoding called “utf8” (alias of “utf8mb3”) only stores a maximum of three bytes per code point. So the character set “utf8″/”utf8mb3” cannot store all Unicode code points: it only supports the … Read more
Why do we need Unicode? In the (not too) early days, all that existed was ASCII. This was okay, as all that would ever be needed were a few control characters, punctuation, numbers and letters like the ones in this sentence. Unfortunately, today’s strange world of global intercommunication and social media was not foreseen, and … Read more
OK, I have found the problem. I use some reporting plugins. In the documentation of the failsafe-maven-plugin I found, that the <encoding> configuration – of course – uses ${project.reporting.outputEncoding} by default. So I added the property as a child element of the project element and everything is fine now: <properties> <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding> <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding> </properties> See also … Read more
A simple workaround is to use Google Spreadsheet. Paste (values only if you have complex formulas) or import the sheet then download CSV. I just tried a few characters and it works rather well. NOTE: Google Sheets does have limitations when importing. See here. NOTE: Be careful of sensitive data with Google Sheets. EDIT: Another … Read more
To expand on the answers others have given: We’ve got lots of languages with lots of characters that computers should ideally display. Unicode assigns each character a unique number, or code point. Computers deal with such numbers as bytes… skipping a bit of history here and ignoring memory addressing issues, 8-bit computers would treat an … Read more
The main difference is sorting accuracy (when comparing characters in the language) and performance. The only special one is utf8_bin which is for comparing characters in binary format. utf8_general_ci is somewhat faster than utf8_unicode_ci, but less accurate (for sorting). The specific language utf8 encoding (such as utf8_swedish_ci) contain additional language rules that make them the … Read more
URLEncoder is the way to go. You only need to keep in mind to encode only the individual query string parameter name and/or value, not the entire URL, for sure not the query string parameter separator character & nor the parameter name-value separator character =. String q = “random word £500 bank $”; String url … Read more
If you already have a byte array then you will need to know what type of encoding was used to make it into that byte array. For example, if the byte array was created like this: byte[] bytes = Encoding.ASCII.GetBytes(someString); You will need to turn it back into a string like this: string someString = … Read more
When you have some binary data that you want to ship across a network, you generally don’t do it by just streaming the bits and bytes over the wire in a raw format. Why? because some media are made for streaming text. You never know — some protocols may interpret your binary data as control … Read more