Use a subrange of [\u0000-\uFFFF] for what you want. You can also use the re.UNICODE compile flag. The docs say that if UNICODE is set, \w will match the characters [0-9_] plus whatever is classified as alphanumeric in the Unicode character properties database. See also http://coding.derkeiler.com/Archive/Python/comp.lang.python/2004-05/2560.html.
You need to specify the re.UNICODE flag, and input your string as a Unicode string by using the u prefix: >>> re.match(r’^/by_tag/(?P<tag>\w+)/(?P<filename>(\w|[.,!#%{}()@])+)$’, u’/by_tag/påske/øyfjell.jpg’, re.UNICODE).groupdict() {‘tag’: u’p\xe5ske’, ‘filename’: u’\xf8yfjell.jpg’} This is in Python 2; in Python 3 you must leave out the u because all strings are Unicode, and you can leave off the re.UNICODE flag.
You can construct a new character class: [^\W\d_] instead of \w. Translated into English, it means “Any character that is not a non-alphanumeric character ([^\W] is the same as \w), but that is also not a digit and not an underscore”. Therefore, it will only allow Unicode letters.
The regex module (an alternative to the standard re module) supports Unicode codepoint properties with the \p{} syntax.
If your regex flavor supports Unicode blocks ([\p{IsCyrillic}]), you can match Cyrillic characters with: [\p{IsCyrillic}] or [\p{Cyrillic}] Otherwise try using: [U+0400–U+04FF] For PHP use: [\x{0400}-\x{04FF}] Explanation: [\p{IsCyrillic}] Match a character from the Unicode block “Cyrillic” (U+0400–U+04FF) «[\p{IsCyrillic}]» Note: Unicode Characters list and Numeric HTML Entities of [U+0400–U+04FF] .
Are you using python 2.x or 3.0? If you’re using 2.x, try making the regex string a unicode-escape string, with ‘u’. Since it’s regex it’s good practice to make your regex string a raw string, with ‘r’. Also, putting your entire pattern in parentheses is superfluous. re.sub(ur'[\u064B-\u0652\u06D4\u0670\u0674\u06D5-\u06ED]+’, ”, …) http://docs.python.org/tutorial/introduction.html#unicode-strings Edit: It’s also good practice … Read more
Source code The source code for the rewriting functions I discuss below is available here. Update in Java 7 Sun’s updated Pattern class for JDK7 has a marvelous new flag, UNICODE_CHARACTER_CLASS, which makes everything work right again. It’s available as an embeddable (?U) for inside the pattern, so you can use it with the String … Read more
Situation for ES 6 The ECMAScript language specification, edition 6 (also commonly known as ES2015), includes Unicode-aware regular expressions. Support must be enabled with the u modifier on the regex. See Unicode-aware regular expressions in ES6 for a break-down of the feature and some caveats. ES6 is widely adopted in both browsers and stand-alone Javascript … Read more