RWCTokenizer Class Reference

Breaks a string into separate tokens, delimited by an arbitrary whitespace. Can be used as an alternative to the C++ Standard Library function std::strtok(). More...

#include <rw/ctoken.h>

Public Member Functions
	RWCTokenizer (const RWCString &s)
	RWCTokenizer (const RWCTokenizer &rhs)
	RWCTokenizer (RWCTokenizer &&rhs)
bool	done () const
RWCSubString	nextToken ()
RWCSubString	nextToken (RWTRegex< char > &regex)
RWCSubString	operator() ()
RWCSubString	operator() (const char *s)
RWCSubString	operator() (const char *s, size_t n)
RWCSubString	operator() (RWTRegex< char > &regex)
RWCTokenizer &	operator= (const RWCTokenizer &rhs)
RWCTokenizer &	operator= (RWCTokenizer &&rhs)
void	swap (RWCTokenizer &rhs)

Detailed Description

Class RWCTokenizer is designed to break a string into separate tokens, delimited by an arbitrary whitespace. Think of it as an iterator for strings and as an alternative to the C++ Standard Library function std::strtok() that has the unfortunate side effect of changing the tokenized string.

Synopsis

#include <rw/ctoken.h>
RWCString str("a string of tokens");
RWCTokenizer(str);  // Lex the above string

Persistence

None

Example

#include <rw/ctoken.h>
#include <iostream>
 
int main() {
  RWCString a(
      "Something is rotten in the state of "
      "Denmark and Hamlet is taking out the trash.");
 
  RWCTokenizer next(a);  // Tokenize the string a
  RWCString token;       // Will receive each token
 
  // Advance until the null string is returned:
  while (!(token = next()).isNull()) {
    std::cout << token << "\n";
  }
}

Constructor & Destructor Documentation

RWCTokenizer() [1/3]

RWCTokenizer::RWCTokenizer ( const RWCString & s )

explicit

Constructs a tokenizer to lex the string s.

RWCTokenizer() [2/3]

RWCTokenizer::RWCTokenizer ( const RWCTokenizer & rhs )

inline

Copy constructor. The created tokenizer copies the data from rhs.

RWCTokenizer() [3/3]

RWCTokenizer::RWCTokenizer ( RWCTokenizer && rhs )

inline

Move constructor. The constructed instance takes ownership of the data owned by rhs.

Condition:

This method is available only on platforms with rvalue reference support.

Member Function Documentation

done()

bool RWCTokenizer::done

(

)

const

Returns true if the last token from the search string has been extracted. Otherwise, returns false. When using the function call operator interface, this is the same as the last non-empty token having been returned.

nextToken() [1/2]

RWCSubString RWCTokenizer::nextToken

(

)

Returns the next token using a specified string of delimiter characters.

This method may return an empty token if there are consecutive occurrences of any delimiter character in the search string.

nextToken() [2/2]

RWCSubString RWCTokenizer::nextToken

(

RWTRegex< char > &

regex

)

Returns the next token using a delimiter pattern represented by a regular expression pattern.

Unlike the other nextToken() overloads, this method allows a single occurrence of a delimiter to span multiple characters.

For example, nextToken(RWCString("ab")) treats either a or b as a delimiter character. Conversely, nextToken(RWTRegex<char>("ab")) treats the two-character pattern ab as a single delimiter.

This method may return an empty token if there are consecutive occurrences of any delimiter character in the search string.

operator()() [1/4]

RWCSubString RWCTokenizer::operator()

(

)

Advances to the next token and returns it as a substring. The tokens are delimited by any of the four characters in " \t\n\0" (space, tab, newline and null).

operator()() [2/4]

RWCSubString RWCTokenizer::operator()

(

const char *

s

)

Advances to the next token and returns it as a substring. The tokens are delimited by any character in s, or any embedded null.

operator()() [3/4]

RWCSubString RWCTokenizer::operator()	(	const char *	s,
		size_t	n )

Advances to the next token and returns it as a substring. The tokens are delimited by any of the first n characters in s. Buffer s may contain nulls, and must contain at least n characters. Tokens will not be delimited by nulls unless s contains nulls.

operator()() [4/4]

RWCSubString RWCTokenizer::operator()

(

RWTRegex< char > &

regex

)

Returns the next token using a delimiter pattern represented by the regular expression pattern regex.

This method, unlike the other operator() overloads, allows a single occurrence of a delimiter to span multiple characters.

For example, consider the RWCTokenizer instance tok. The statement tok(RWCString("ab")) treats either a or b as a delimiter character. On the other hand, tok(RWTRegex<char>("ab")) treats the two-character pattern, ab, as a single delimiter.

This method consumes consecutive occurrences of delimiters and skips over any empty fields present in the string. To obtain empty fields as well as non-empty fields, use the nextToken(RWTRegex<char>&) method.

operator=() [1/2]

RWCTokenizer & RWCTokenizer::operator= ( const RWCTokenizer & rhs )

inline

Assignment operator. The tokenizer copies the data from rhs. Returns a reference to self.

operator=() [2/2]

RWCTokenizer & RWCTokenizer::operator= ( RWCTokenizer && rhs )

inline

Move assignment. Self takes ownership of the data owned by rhs.

Condition:

This method is available only on platforms with rvalue reference support.

swap()

void RWCTokenizer::swap ( RWCTokenizer & rhs )

inline

Swaps the data owned by self with the data owned by rhs.