UTF-8 has an advantage in the case where ASCII characters represent the majority of characters in a block of text, because UTF-8 encodes these into 8 bits (like ASCII). It is also advantageous in that a UTF-8 file containing only ASCII characters has the same encoding as an ASCII file. UTF-16 is better where ASCII … Read more
read_csv takes an encoding option to deal with files in different formats. I mostly use read_csv(‘file’, encoding = “ISO-8859-1”), or alternatively encoding = “utf-8” for reading, and generally utf-8 for to_csv. You can also use one of several alias options like ‘latin’ or ‘cp1252’ (Windows) instead of ‘ISO-8859-1’ (see python docs, also for numerous other … Read more
To expand on the answers others have given: We’ve got lots of languages with lots of characters that computers should ideally display. Unicode assigns each character a unique number, or code point. Computers deal with such numbers as bytes… skipping a bit of history here and ignoring memory addressing issues, 8-bit computers would treat an … Read more
Unidecode is the correct answer for this. It transliterates any unicode string into the closest possible representation in ascii text. Example: accented_string = u’Málaga’ # accented_string is of type ‘unicode’ import unidecode unaccented_string = unidecode.unidecode(accented_string) # unaccented_string contains ‘Malaga’and is of type ‘str’
The text uses combining characters, also known as combining marks. See section 2.11 of Combining Characters in the Unicode Standard (PDF). In Unicode, character rendering does not use a simple character cell model where each glyph fits into a box with given height. Combining marks may be rendered above, below, or inside a base character … Read more
Here’s how I’d do it – For Schemas (or Databases – they are synonyms): SELECT default_character_set_name FROM information_schema.SCHEMATA WHERE schema_name = “schemaname”; For Tables: SELECT CCSA.character_set_name FROM information_schema.`TABLES` T, information_schema.`COLLATION_CHARACTER_SET_APPLICABILITY` CCSA WHERE CCSA.collation_name = T.table_collation AND T.table_schema = “schemaname” AND T.table_name = “tablename”; For Columns: SELECT character_set_name FROM information_schema.`COLUMNS` WHERE table_schema = “schemaname” AND table_name … Read more
Use the ensure_ascii=False switch to json.dumps(), then encode the value to UTF-8 manually: >>> json_string = json.dumps(“ברי צקלה”, ensure_ascii=False).encode(‘utf8’) >>> json_string b'”\xd7\x91\xd7\xa8\xd7\x99 \xd7\xa6\xd7\xa7\xd7\x9c\xd7\x94″‘ >>> print(json_string.decode()) “ברי צקלה” If you are writing to a file, just use json.dump() and leave it to the file object to encode: with open(‘filename’, ‘w’, encoding=’utf8′) as json_file: json.dump(“ברי צקלה”, json_file, … Read more
There’s not really any “raw string“; there are raw string literals, which are exactly the string literals marked by an ‘r’ before the opening quote. A “raw string literal” is a slightly different syntax for a string literal, in which a backslash, \, is taken as meaning “just a backslash” (except when it comes right … Read more
string? wstring? std::string is a basic_string templated on a char, and std::wstring on a wchar_t. char vs. wchar_t char is supposed to hold a character, usually an 8-bit character. wchar_t is supposed to hold a wide character, and then, things get tricky: On Linux, a wchar_t is 4 bytes, while on Windows, it’s 2 bytes. … Read more
The file in question is not using the CP1252 encoding. It’s using another encoding. Which one you have to figure out yourself. Common ones are Latin-1 and UTF-8. Since 0x90 doesn’t actually mean anything in Latin-1, UTF-8 (where 0x90 is a continuation byte) is more likely. You specify the encoding when you open the file: … Read more