Tokenizers

A tokenizer does the following:

• Receives a stream of characters.

• Breaks the stream into individual tokens that usually correspond to individual words.

• Returns a stream of tokens.