ibis::text::tokenizer Struct Reference

A tokenizer class to turn a string buffer into tokens. More...

#include <category.h>

Inheritance diagram for ibis::text::tokenizer:

Public Member Functions
virtual int	operator() (std::vector< const char * > &tkns, char *buf)=0
	A tokenizer must implement a two-argument operator(). More...
virtual	~tokenizer ()
	Destructor.

Detailed Description

A tokenizer class to turn a string buffer into tokens.

Used by ibis::keywords to build a term-document index.

Member Function Documentation

virtual int ibis::text::tokenizer::operator() ( std::vector< const char * > &  tkns, char *  buf  )

pure virtual

A tokenizer must implement a two-argument operator().

It takes an input string in buf to produce a list of tokens in tkns. The input buffer may be modified in this function. The return value shall be zero (0) to indicate success, a positive value to carray a warning message, and a negative value to indicate fatal error.

Note

This function is not declared as const because a derived class might want to keep some statistics or otherwise alter its state while processing an incoming text buffer.

Implemented in ibis::keywords::tokenizer.

---

The documentation for this struct was generated from the following file:

• category.h