AshScylla.Search.Analyzer.Normalizer (AshScylla v1.6.1)

Copy Markdown View Source

Text normalization for the search pipeline.

Handles:

  • Lowercasing (Unicode-aware via String.downcase/2)
  • Unicode normalization (NFC form)
  • Punctuation and special character removal

The normalizer runs after tokenization but could also be applied to the full text before tokenization.

Summary

Functions

normalize_term(term)

Normalizes a single term.

normalize_terms(terms)

Normalizes a list of terms, removing any that normalize to nil or empty.

Functions

normalize_term(term)

@spec normalize_term(String.t()) :: String.t() | nil

Normalizes a single term.

Applies lowercase, NFC normalization, and strips surrounding punctuation. Returns nil if the term becomes empty after normalization.

Examples

iex> Normalizer.normalize_term("Phoenix!")
"phoenix"

iex> Normalizer.normalize_term("  ELIXIR  ")
"elixir"

iex> Normalizer.normalize_term("!!!")
nil

normalize_terms(terms)

@spec normalize_terms([String.t()]) :: [String.t()]

Normalizes a list of terms, removing any that normalize to nil or empty.