AshScylla.Search.Analyzer.Tokenizer (AshScylla v1.6.1)

Copy Markdown View Source

Unicode-aware tokenizer for the search pipeline.

Splits text into individual tokens using Unicode word boundary rules. Handles:

  • Mixed scripts (Latin, CJK, etc.)
  • Punctuation stripping
  • Contiguous alphanumeric sequences as single tokens
  • Emoji filtering

Uses regex patterns that respect Unicode category properties rather than simple whitespace splitting.

Summary

Functions

tokenize(text)

Tokenizes a text string into a list of word tokens.

tokenize(text, opts)

Tokenizes text and returns tokens longer than the minimum length.

tokenize_fields(fields)

Tokenizes multiple text fields, returning a flat list of all tokens.

Functions

tokenize(text)

@spec tokenize(String.t()) :: [String.t()]

Tokenizes a text string into a list of word tokens.

Examples

iex> Tokenizer.tokenize("Learning Elixir Phoenix Framework")
["Learning", "Elixir", "Phoenix", "Framework"]

iex> Tokenizer.tokenize("Phoenix,Framework!测试")
["Phoenix", "Framework", "测试"]

iex> Tokenizer.tokenize("hello_world test-case")
["hello_world", "test", "case"]

tokenize(text, opts)

@spec tokenize(
  String.t(),
  keyword()
) :: [String.t()]

Tokenizes text and returns tokens longer than the minimum length.

tokenize_fields(fields)

@spec tokenize_fields(%{optional(atom()) => String.t()}) :: [String.t()]

Tokenizes multiple text fields, returning a flat list of all tokens.