RWCTokenizer

• Data Type and Member Function Indexes
• Synopsis
• Description
• Persistence
• Example
• Public Constructor
• Public Member Operators

Member Functions

operator()

#include <rw/ctoken.h>
RWCString str("a string of tokens");
RWCTokenizer(str);  // Lex the above string

Class RWCTokenizer is designed to break a string up into separate tokens, delimited by an arbitrary "white space." It can be thought of as an iterator for strings and as an alternative to the ANSI C function strtok() which has the unfortunate side effect of changing the string being tokenized.

None

#include <rw/ctoken.h>
#include <rw/rstream.h>
main(){
  RWCString a("Something is rotten in the state of Denmark");
  RWCTokenizer next(a);           // Tokenize the string a
  RWCString token;                // Will receive each token
  // Advance until the null string is returned:
  while (!(token=next()).isNull())
    cout << token << "\n";
}

Program output:

   Something
   is
   rotten
   in
   the
   state
   of
   Denmark

RWCTokenizer(const RWCString& s);

Construct a tokenizer to lex the string s.

RWCSubString
operator();

Advance to the next token and return it as a substring. The tokens are delimited by any of the four characters in " \t\n\0". (space, tab, newline and null).

RWCSubString
operator()(const char* s);

Advance to the next token and return it as a substring. The tokens are delimited by any character in s, or any embedded null.

RWCSubString
operator()(const char* s,size_t num);

Advance to the next token and return it as a substring. The tokens are delimited by any of the first num characters in s. Buffer s may contain nulls, and must contain at least num characters. Tokens will not be delimited by nulls unless s contains nulls.