Don’t worry about that. According to the same information source: this library component should be retired to Annex D, along side , until a suitable replacement is standardized. So, you can still use it until a new standardized, more-secure version is done.
Simply use the “utf-8-sig” codec: fp = open(“file.txt”) s = fp.read() u = s.decode(“utf-8-sig”) That gives you a unicode string without the BOM. You can then use s = u.encode(“utf-8”) to get a normal UTF-8 encoded string back in s. If your files are big, then you should avoid reading them all into memory. The … Read more
I believe there are a lot of good articles about this around the Web, but here is a short summary. Both UTF-8 and UTF-16 are variable length encodings. However, in UTF-8 a character may occupy a minimum of 8 bits, while in UTF-16 character length starts with 16 bits. Main UTF-8 pros: Basic ASCII characters … Read more
I’ve been struggling with this problem for a while, and just discovered (for me) a perfect solution: $ git config –global diff.tool vimdiff # or merge.tool to get merging too! $ git difftool commit1 commit2 git difftool takes the same arguments as git diff would, but runs a diff program of your choice instead of … Read more
The term “surrogate pair” refers to a means of encoding Unicode characters with high code-points in the UTF-16 encoding scheme. In the Unicode character encoding, characters are mapped to values between 0x0 and 0x10FFFF. Internally, Java uses the UTF-16 encoding scheme to store strings of Unicode text. In UTF-16, 16-bit (two-byte) code units are used. … Read more
Why do we need Unicode? In the (not too) early days, all that existed was ASCII. This was okay, as all that would ever be needed were a few control characters, punctuation, numbers and letters like the ones in this sentence. Unfortunately, today’s strange world of global intercommunication and social media was not foreseen, and … Read more
UTF-8 has an advantage in the case where ASCII characters represent the majority of characters in a block of text, because UTF-8 encodes these into 8 bits (like ASCII). It is also advantageous in that a UTF-8 file containing only ASCII characters has the same encoding as an ASCII file. UTF-16 is better where ASCII … Read more