The UTF-8 BOM is a sequence of bytes at the start of a text stream (0xEF, 0xBB, 0xBF) that allows the reader to more reliably guess a file as being encoded in UTF-8. Normally, the BOM is used to signal the endianness of an encoding, but since endianness is irrelevant to UTF-8, the BOM is … Read more
Python 3.x makes a clear distinction between the types: str = ‘…’ literals = a sequence of Unicode characters (Latin-1, UCS-2 or UCS-4, depending on the widest character in the string) bytes = b’…’ literals = a sequence of octets (integers between 0 and 255) If you’re familiar with: Java or C#, think of str … Read more
For those people still arriving at this question in 2020 or later, there are newer options that may be better than both of these. For example, utf8_unicode_520_ci. All these collations are for the UTF-8 character encoding. The differences are in how text is sorted and compared. _unicode_ci and _general_ci are two different sets of rules … Read more
Unicode arrows heads: ▲ – U+25B2 BLACK UP-POINTING TRIANGLE ▼ – U+25BC BLACK DOWN-POINTING TRIANGLE ▴ – U+25B4 SMALL BLACK UP-POINTING TRIANGLE ▾ – U+25BE SMALL BLACK DOWN-POINTING TRIANGLE For ▲ and ▼ use ▲ and ▼ respectively if you cannot include Unicode characters directly (use UTF-8!). Note that the font support for the smaller … Read more
Unicode decoding takes place before any other lexical translation. The key benefit of this is that it makes it trivial to go back and forth between ASCII and any other encoding. You don’t even need to figure out where comments begin and end! As stated in JLS Section 3.3 this allows any ASCII based tool … Read more
You need to read the Python Unicode HOWTO. This error is the very first example. Basically, stop using str to convert from unicode to encoded text / bytes. Instead, properly use .encode() to encode the string: p.agent_info = u’ ‘.join((agent_contact, agent_telno)).encode(‘utf-8’).strip() or work entirely in unicode.