# `AshScylla.Search.Analyzer.Tokenizer`
[🔗](https://github.com/ohhi-vn/ash_scylla/blob/main/lib/ash_scylla/search/analyzer/tokenizer.ex#L1)

Unicode-aware tokenizer for the search pipeline.

Splits text into individual tokens using Unicode word boundary rules.
Handles:
  * Mixed scripts (Latin, CJK, etc.)
  * Punctuation stripping
  * Contiguous alphanumeric sequences as single tokens
  * Emoji filtering

Uses regex patterns that respect Unicode category properties rather
than simple whitespace splitting.

# `tokenize`

```elixir
@spec tokenize(String.t()) :: [String.t()]
```

Tokenizes a text string into a list of word tokens.

## Examples

    iex> Tokenizer.tokenize("Learning Elixir Phoenix Framework")
    ["Learning", "Elixir", "Phoenix", "Framework"]

    iex> Tokenizer.tokenize("Phoenix,Framework!测试")
    ["Phoenix", "Framework", "测试"]

    iex> Tokenizer.tokenize("hello_world test-case")
    ["hello_world", "test", "case"]

# `tokenize`

```elixir
@spec tokenize(
  String.t(),
  keyword()
) :: [String.t()]
```

Tokenizes text and returns tokens longer than the minimum length.

# `tokenize_fields`

```elixir
@spec tokenize_fields(%{optional(atom()) => String.t()}) :: [String.t()]
```

Tokenizes multiple text fields, returning a flat list of all tokens.

---

*Consult [api-reference.md](api-reference.md) for complete listing*