RWWTokenizer

• Data Type and Member Function Indexes
• Synopsis
• Description
• Persistence
• Example
• Public Constructor
• Public Member Function

Member Functions

operator()

#include <rw/wtoken.h>
RWWString str("a string of tokens", RWWString::ascii);
RWWTokenizer(str);  // Lex the above string

Class RWWTokenizer is designed to break a string up into separate tokens, delimited by arbitrary "white space." It can be thought of as an iterator for strings and as an alternative to the C library function wstok() which has the unfortunate side effect of changing the string being tokenized.

None

#include <rw/wtoken.h>
#include <rw/rstream.h>

main(){
  RWWString a(L"Something is rotten in the state of Denmark");

  RWWTokenizer next(a);   // Tokenize the string a

  RWWString token;        // Will receive each token

  // Advance until the null string is returned:
  while (!(token=next()).isNull())
    cout << token << "\n";
}

Program output:

   Something
   is
   rotten
   in
   the
   state
   of
   Denmark

RWWTokenizer(const RWWString& s);

Construct a tokenizer to lex the string s.

RWWSubString
operator();

Advance to the next token and return it as a substring. The tokens are delimited by any of the four wide characters in L" \t\n\0". (space, tab, newline and null).

RWWSubString
operator()(const wchar_t* s);

Advance to the next token and return it as a widesubstring. The tokens are delimited by any wide character in s, or any embedded wide null.

RWWSubString
operator()(const wchar_t* s,size_t num);

Advance to the next token and return it as a substring. The tokens are delimited by any of the first num wide characters in s. Buffer s may contain embedded nulls, and must contain at least num wide characters. Tokens will not be delimited by nulls unless s contains nulls.