T

tokenizer

Utility for doing lexical parsing of some input. It is optimized for static (string) parsing, not partial input stream parsing, but contains tokenizer supporting either.

chinese-english dictionary based tokenizer for lucene/solr