A scalable multi-word search engine built on ScyllaDB using an inverted index.
Provides Lucene/OpenSearch-style search capabilities without LIKE,
ALLOW FILTERING, or secondary indexes.
Architecture
The search engine uses an inverted index approach:
Document → Analyzer → Tokenizer → Normalizer → Stemmer
→ Stop Words → Indexer → search_post_terms table
Query → Analyzer → Planner → Boolean Engine → Ranking → ResultsQuick Start
# 1. Create search tables
Search.create_tables(MyApp.Repo, "my_keyspace")
# 2. Index a document
Search.index(MyApp.Repo, "my_keyspace", "post-uuid-here", %{
title: "Learning Elixir Phoenix Framework",
body: "Phoenix is a distributed web framework built on Elixir."
})
# 3. Search
{:ok, results} = Search.search(MyApp.Repo, "my_keyspace", "learning phoenix")
#=> %{entries: [{"post-uuid-here", 2.0}], ...}Features
- Single-word and multi-word AND/OR search
- Pagination with page metadata
- Relevance ranking (TF, TF-IDF, BM25)
- Document updates and deletes
- Sharded partitions to prevent hotspot term partitions
- Unicode-aware tokenization
- Porter stemming
- Stop word filtering
Summary
Functions
Creates the search engine tables in the given keyspace.
Removes a document from the search index.
Drops the search engine tables from the given keyspace.
Indexes a document into the search engine.
Searches the inverted index for documents matching the query.
Same as search/4 but raises on error.
Updates a document's indexed terms.
Types
@type search_result() :: {:ok, AshScylla.Search.Query.Paginator.page()} | {:error, term()}
Functions
Creates the search engine tables in the given keyspace.
This must be called once before indexing or searching.
Returns :ok or {:error, reason}.
Removes a document from the search index.
Deletes all term entries for the given post ID from both
search_post_terms and search_post_fields.
Returns :ok or {:error, reason}.
Drops the search engine tables from the given keyspace.
Indexes a document into the search engine.
Accepts a map of field names to text values. Each field is analyzed and its terms are written to the inverted index.
Returns :ok or {:error, reason}.
Examples
Search.index(MyApp.Repo, "my_keyspace", "abc-123", %{
title: "Learning Elixir",
body: "Elixir is a functional language"
})@spec search(module(), String.t(), String.t(), keyword()) :: search_result()
Searches the inverted index for documents matching the query.
The full search pipeline:
- Parse query string into AST
- Analyze query terms
- Look up posting lists from ScyllaDB
- Apply boolean operations (AND/OR/NOT)
- Rank results by relevance
- Paginate results
Options
:page— page number, starting at 1 (default:1):page_size— results per page (default:20):strategy— ranking strategy::tf(default),:tfidf,:bm25:num_shards— shard count per term partition (default:16):analyzer_opts— options passed to the analyzer (e.g.:stem)
Examples
# Basic search
{:ok, page} = Search.search(repo, keyspace, "elixir phoenix")
# With AND/OR operators
{:ok, page} = Search.search(repo, keyspace, "elixir OR phoenix")
# With BM25 ranking and pagination
{:ok, page} = Search.search(repo, keyspace, "distributed web",
strategy: :bm25, page: 1, page_size: 10)Returns {:ok, page} where page is a map with :entries, :page_number,
:total_count, etc., or {:error, reason}.
@spec search!(module(), String.t(), String.t(), keyword()) :: AshScylla.Search.Query.Paginator.page() | no_return()
Same as search/4 but raises on error.
Updates a document's indexed terms.
Computes the diff between old and new terms for each field, applying only the necessary inserts and deletes. Fields omitted from the map are left unchanged.
Returns :ok or {:error, reason}.