Creating an RWUTokenizer
An
RWUTokenizer stores a deep copy of a string to be tokenized. You can pass the string in the constructor, as shown below:
RWUConversionContext ascii("ascii");
RWUString text("This is a string.");
RWUTokenizer tok(text);
The setText() method assigns a new string to a tokenizer, like so:
RWUString text2("This is another string.");
tok.setText(text2);
The getText() method returns a copy of the current string associated with a tokenizer.
It is also possible to construct an empty
RWUTokenizer, but no tokens can be obtained from such a tokenizer until the
setText() method is used to assign a string to the tokenizer. For example:
RWUTokenizer tok2();
tok2.setText(text);
