Redis site search makes document structure part of relevance
The redis-sitesearch example turns pages and sections into separate indexed documents. Its scoring rules and scheduled crawler show how application choices determine search relevance and freshness.
A documentation search can find the correct words and still return the wrong level of detail. A section within a deployment guide may match a query even when the reader needs the introductory page for the topic. The redis-sitesearch example addresses that problem by modelling pages and sections separately, then assigning relevance rules that understand the difference.
Andrew Brookins describes the application in the November 27 announcement. The Python implementation powers search on the Redis documentation site and is available as open-source example code. Its architecture is useful because it shows the application work between crawling HTML and returning search results.
I have used Redis since its release, and I like seeing its data structures applied to a concrete service. Search adds a requirement that a key lookup does not answer by itself. The application must decide what constitutes a document and which matching document should appear first.
A page becomes several searchable objects
The application uses a background crawler built with Scrapy. It converts page content into structured search documents, including documents representing sections. That gives a result enough information to identify both the page title and the section where a match occurs.
The announced model includes a title, section title, body and URL. Redis hashes hold the documents, and RediSearch indexes hashes matching the configured prefix. The article describes a prefix containing the environment and site URL, allowing the design to distinguish more than one site even though the current application indexes the documentation site.
This representation has an immediate effect on results. A hit can point to the relevant part of a page while still showing the enclosing page as context. The crawler is therefore doing more than stripping markup into a bag of words. It identifies boundaries that the result presentation later uses.
The choice also produces several candidates from related content. A page document and a section document can both be relevant to the same query. Ranking needs to express how those candidates relate, or the extra granularity can make the result list harder to use.
Ranking contains application policy
The announcement describes giving section documents a lower score than page documents. It also favours pages higher in the site's URL hierarchy over deeper pages when other factors are equal. Those are explicit product choices about which material a documentation reader should encounter first.
The linked scoring implementation separates those choices into scoring functions. One adjusts non-page documents, while another reduces scores according to hierarchy depth. A score floor prevents a document's multiplier from dropping to zero.
RediSearch's document score multiplies the final query relevance value, according to the article. That means the application can encode some facts about a document at indexing time while still allowing the text match to contribute to the final order. The document policy does not need to replace the query's relevance calculation.
This is a useful distinction when evaluating an example. Preferring an introductory page is appropriate to the stated documentation goal. Another application may need the most specific section first. The example demonstrates where that policy can live, without making the ordering universally correct for every collection.
Validation determines what enters the index
The application also filters documents before indexing. Its validators skip release notes and material that looks like a not-found page. This is a separate decision from assigning a low score. A document excluded by validation cannot appear in results regardless of how well its text would match.
That separation keeps two policies understandable. Scoring ranks permitted candidates, while validation decides which candidates exist in the searchable collection. Putting every exclusion into a relevance weight would make it less clear whether an unwanted document could still appear.
The crawler's output therefore defines the search corpus as well as its structure. A successful crawl does not mean every page is meant to be searchable. The example makes those application-specific exclusions visible alongside its relevance rules.
Fast query interaction and fresh content have different clocks
The frontend sends requests while the user types, and a Python API translates them into prefix queries. The article gives the example of turning a partial term into a query ending in an asterisk. Matching prefixes lets results appear before the complete term has been entered.
Indexing follows a different schedule. Redis Queue runs the crawling job every thirty minutes, after which the resulting hashes enter the index. The interface can respond to typing without implying that a newly edited web page is already available to search at that instant.
The deployment places Redis with RediSearch, the API and the indexing worker in one container. The article describes geographically distributed instances behind a global load balancer. That arrangement is part of the example's response-time design, while the scheduled crawler still determines how source-page edits reach each searchable dataset.
I find the separation useful because it names the work an application must supply around a search engine. The example models sections, encodes relevance preferences and filters its corpus before serving prefix queries. Its thirty-minute crawl interval remains a separate freshness condition even when the interactive search response feels immediate.