Let me introduce you to the Levenshtein distance formula. It is awesome: http://en.wikipedia.org/wiki/Levenshtein_distance In information theory and computer science, the Levenshtein distance is a string metric for measuring the amount of difference between two sequences. The term edit distance is often used to refer specifically to Levenshtein distance. Personally I used this in a healthcare … Read more
I suggest you read the articles by Navarro mentioned in the Refences section of the Wikipedia article titled Approximate string matching. Making your decision based on actual research is always better than on suggestions by random strangers.. Especially if performance on a known set of records is important to you.
I wouldn’t use Spark in the first place, but if you are really committed to the particular stack, you can combine a bunch of ml transformers to get best matches. You’ll need Tokenizer (or split): import org.apache.spark.ml.feature.RegexTokenizer val tokenizer = new RegexTokenizer().setPattern(“”).setInputCol(“text”).setMinTokenLength(1).setOutputCol(“tokens”) NGram (for example 3-gram) import org.apache.spark.ml.feature.NGram val ngram = new NGram().setN(3).setInputCol(“tokens”).setOutputCol(“ngrams”) Vectorizer (for … Read more
Use the create_index method where you pass in the keys as an array and TEXT as the index direction : collection.create_index([(‘field_i_want_to_index’, pymongo.TEXT)], name=”search_index”, default_language=”english”)
Possible source for real world typo statistics would be in the Wikipedia’s complete edit history. http://download.wikimedia.org/ Also, you might be interested in the AWB’s RegExTypoFix http://en.wikipedia.org/wiki/Wikipedia:AWB/T
Postgres provides a module with several string comparsion functions such as soundex and metaphone. But you will want to use the levenshtein edit distance function. Example: test=# SELECT levenshtein(‘GUMBO’, ‘GAMBOL’); levenshtein ————- 2 (1 row) The 2 is the edit distance between the two words. When you apply this against a number of words and … Read more
I found the TRE library, which seems to be able to do exactly fuzzy matching of regular expressions. Example: http://hackerboss.com/approximate-regex-matching-in-python/ It only supports insertion, deletion and substitution though. No transposition. But I guess that works ok. I tried the accompanying agrep tool with the regexp on the following file: TV Schedule for 10Jan TVSchedule for … Read more