class Ferret::Analysis::AsciiStandardTokenizer

Summary

The standard tokenizer is an advanced tokenizer which tokenizes most words correctly as well as tokenizing things like email addresses, web addresses, phone numbers, etc.

Example

"Dave's résumé, at http://www.davebalmain.com/ 1234"
  => ["Dave's", "r", "sum", "at", "http://www.davebalmain.com", "1234"]

Public Class Methods

new() → tokenizer click to toggle source

Create a new AsciiStandardTokenizer

static VALUE
frb_a_standard_tokenizer_init(VALUE self, VALUE rstr) 
{
    return get_wrapped_ts(self, rstr, standard_tokenizer_new());
}