Using the Function Call Operator
In the tradition of
RWCTokenizer, the function call operator interface scans a string for all occurrences of tokens, consuming all consecutive occurrences of the specified set of delimiter characters. As such, the function call operator does not return empty tokens. For example, this code extracts all tokens from a string using the function call operator and the default delimiters:
RWUConversionContext ascii("ascii");
RWUString text("This is a string.");
RWUString next;
RWUTokenizer tok(text);
for (next = tok(); !next.isNull(); next = tok()) {
// Process each token
}
Note the use of the traditional empty token condition to detect the end of tokenization.
The following tokens are extracted by this code:
This
is
a
string.
