Are you committed to using NLTK/Python? I ran into the same problems as you, and had much better results using Stanford’s named-entity recognizer: http://nlp.stanford.edu/software/CRF-NER.shtml. The process for training the classifier using your own data is very well-documented in the FAQ. If you really need to use NLTK, I’d hit up the mailing list for some … Read more
Based on an issue and a patch in Clear TK, it seems like BILOU stands for “Beginning, Inside and Last tokens of multi-token chunks, Unit-length chunks and Outside” (emphasis added). For instance, the chunking denoted by brackets (foo foo foo) (bar) no no no (bar bar) can be encoded with BILOU as B-foo, I-foo, L-foo, … Read more
They likely use Information Extraction techniques for this. Here is a demo of Stanford’s SUTime tool: http://nlp.stanford.edu:8080/sutime/process You would extract attributes about n-grams (consecutive words) in a document: numberOfLetters numberOfSymbols length previousWord nextWord nextWordNumberOfSymbols … And then use a classification algorithm, and feed it positive and negative examples: Observation nLetters nSymbols length prevWord nextWord isPartOfDate … Read more