The scoring contains the Inverse Document Frequency(IDF). If the term “John Smith” is in one partition, 0, 100 times and in partition 1, once. The score for searching for John Smith would be higher search in partition 1 as the term is more scarce. To get round this you would wither have to have your … Read more
Gensim has an efficient tf-idf model and does not need to have everything in memory at once. Your corpus simply needs to be an iterable, so it does not need to have the whole corpus in memory at a time. The make_wiki script runs over Wikipedia in about 50m on a laptop according to the … Read more
Websolr provides a cloud-based Solr with a control panel. It’s in private beta as of this writing, but you can get the service through Heroku. Another hosted Solr service is PowCloud, also in private beta, which seems to offer strong WordPress integration. SolrHQ: another beta service providing a hosted Solr solution, with Joomla and WordPress … Read more
Here’s what I’m using. Is this canonical? Is there a better way? SolrQuery q = new SolrQuery(“*:*”); q.setRows(0); // don’t actually request any data return server.query(q).getResults().getNumFound();
Store.Yes Means that the value of the field will be stored in the index Store.No Means that the value of the field will NOT be stored in the index Store.Yes/No does not affect the indexing or searching with lucene. It just tells lucene if you want it to act as a datastore for the values … Read more
The Lucene index is split into smaller chunks called segments. Each segment is its own index. Lucene searches all of them in sequence. A new segment is created when a new writer is opened and when a writer commits or is closed. The advantages of using this system are that you never have to modify … Read more
Whoosh is a new project which is similar to lucene, but is pure python.
In summary: I’m going to be recommending ElasticSearch, but let’s break the problem down and talk about how to implement it: There are a few parts to this: Extracting the text from your docs to make them indexable Making this text available as full text search Returning highlighted snippets of the doc Knowing where in … Read more
Based on @Alexandre Victoor’s answer, I wrote a little class based on the Lucene Spellchecker in the contrib package (and using the LuceneDictionary included in it) that does exactly what I want. This allows re-indexing from a single source index with a single field, and provides suggestions for terms. Results are sorted by the number … Read more
I had a similar issue, and ifound that “.raw” fixed it – in your example, try url.raw : “http://www.domain_name.com” Or for newer versions of ES(5.x, 6.x): url.keyword : “http://www.domain_name.com”