Creating an RWUTokenizer
An RWUTokenizer stores a deep copy of a string to be tokenized. You can pass the string in the constructor, as shown below:
 
RWUConversionContext ascii("ascii");
RWUString text("This is a string.");
RWUTokenizer tok(text);
The setText() method assigns a new string to a tokenizer, like so:
 
RWUString text2("This is another string.");
tok.setText(text2);
The getText() method returns a copy of the current string associated with a tokenizer.
It is also possible to construct an empty RWUTokenizer, but no tokens can be obtained from such a tokenizer until the setText() method is used to assign a string to the tokenizer. For example:
 
RWUTokenizer tok2();
tok2.setText(text);