AshScylla.Search.Analyzer (AshScylla v1.6.1)

Text analysis pipeline coordinator.

Orchestrates the full text analysis pipeline:

Document text
  → Tokenizer (split into words)
  → Normalizer (lowercase, strip punctuation, NFC normalize)
  → Stop Words filter (remove common words)
  → Stemmer (reduce to root form)
  → Unique terms with counts

Usage

iex> Analyzer.analyze("Learning Elixir Phoenix Framework")
[{"phoenix", 1}, {"framework", 1}, {"learn", 1}, {"elixir", 1}]

The result is a keyword list of {term, frequency} pairs ready for indexing or query processing.

Summary

Functions

analyze(text, opts \\ [])

Analyzes text and returns a list of {term, term_frequency} tuples.

analyze_fields(fields, opts \\ [])

Analyzes a map of field names to text values.

analyze_query(query, opts \\ [])

Analyzes a query string for search.

Functions

analyze(text, opts \\ [])

@spec analyze(
  String.t(),
  keyword()
) :: [{String.t(), pos_integer()}]

Analyzes text and returns a list of {term, term_frequency} tuples.

The terms are:

Tokenized from the input text
Normalized (lowercase, punctuation removal, NFC)
Filtered to remove stop words
Stemmed to their root form
Deduplicated with frequency counts

Options

:stem — whether to apply stemming (default: true)
:remove_stop_words — whether to remove stop words (default: true)
:min_length — minimum token length (default: 1)

Examples

iex> Analyzer.analyze("The Phoenix Framework is running fast")
[{"phoenix", 1}, {"framework", 1}, {"run", 1}, {"fast", 1}]

analyze_fields(fields, opts \\ [])

@spec analyze_fields(
  %{optional(atom()) => String.t()},
  keyword()
) :: [{String.t(), pos_integer()}]

Analyzes a map of field names to text values.

Returns a single merged term-frequency list across all fields. Each field is analyzed independently and results are merged.

Examples

iex> Analyzer.analyze_fields(%{
...>   title: "Learning Elixir",
...>   body: "Elixir is great"
...> })
[{"elixir", 2}, {"learn", 1}, {"great", 1}]

analyze_query(query, opts \\ [])

@spec analyze_query(
  String.t(),
  keyword()
) :: [String.t()]

Analyzes a query string for search.

Unlike document analysis, query analysis preserves the term order for phrase search support. Returns a flat list of normalized terms.

Examples

iex> Analyzer.analyze_query("learning phoenix framework")
["learn", "phoenix", "framework"]