Add tokenization of words

[mattlockyer]

Adding a tokenization library or making this an optional input could really help.

For instance, what if I wanted to search only top sentences related to "food" for recipes or "locations" for scanning destinations in blog posts?

I realize I can do this using the chain phrase myself but basic NLP entity extraction would be nice to have so we can group by broader categories.

Also, the term "chain phrase" implies that the words will be found in order, like in a "chain". It would be trivial and somewhat useful to add a "preserve order" option so we can proximity search within phrases.
e.g. "captain james kirk" should have preserve order so we can find "captain, james t. kirk" but we don't need "kirk captain james". Suggestion.

Otherwise the library works really well, it's clean and straightforward, thanks!