Search documents without leaking revoked content

Application background

Employees search for words in company documents. A search index stores a processed copy of document text so queries do not need to scan every original file. Results may include a title and a short excerpt called a snippet.

The index can lag behind edits and permission changes. If access is revoked after indexing, even showing a sensitive title or snippet can leak information. The current document permissions must still control the response.

Example walkthrough

An index is derived data, not the authority for access. Your search response must account for the gap between updating the original record and updating its search copy.

Your assignment

Deliver: Build document indexing and search. Recheck current access before returning a result's title, snippet or cached content, including when the index is outdated.

Required behavior: GET /search returns at most 20 authorized hits with document IDs and source versions. Index freshness target is two minutes. The index is a projection. Current authorization is checked before content or snippets are returned.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/document_search.py

Supplied file: examples/architecture-starts/document_search.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for document-search. No AWS resources are created."""
index=[{'id':'public','score':9},{'id':'secret','score':10}]
content={'public':'Engineering handbook','secret':'Private acquisition plan'}
allowed={'public'}
results=[]
for hit in sorted(index,key=lambda h:-h['score']):
    if hit['id'] not in allowed: continue
    results.append({'id':hit['id'],'snippet':content[hit['id']]})
print(results)

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

[{'id': 'public', 'snippet': 'Engineering handbook'}]

Set up your implementation workspace

Create work/document-search/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Index one versioned document

Extract bounded text, normalize fields and send an upsert carrying the source version. Apply deletes as versioned tombstones. An old indexing task must not overwrite newer content or recreate a deleted document.

2. Build a constrained query API

Limit query length, filters, page size and execution timeout. Return a stable pagination token tied to the chosen search snapshot where supported. Keep expensive wildcard and unbounded aggregation features out of the initial public contract.

3. Authorize candidates before snippets

Apply tenant filters in the query as defense in depth, then verify current object access before fetching or returning snippets. Do not return a restricted title or highlight and only check permission when the user clicks. Bound overfetch and allow short pages when many candidates are denied.

4. Reindex without mixing generations

Build a new index from a consistent source boundary and capture subsequent changes. Compare source-version coverage, swap the read alias, and retain a rollback window. Measure indexing lag separately from query latency and relevance quality.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

OpenSearch ranks searchable candidates. The document/ACL authority decides whether the caller can see their content. An index filter alone cannot satisfy immediate revocation when indexing is asynchronous.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Add vector retrieval without bypassing document permissions

An embedding can remain in an index after a document is revoked. Filtering only when indexing grants old derived data more authority than current source permissions.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Use lexical and vector indexes for candidate discovery, then merge bounded candidates by stable document ID and source version. Fetch authorized source chunks through a trusted document service. Recheck permission before returning snippets or generated text if revocation may occur during processing. Rebuild embedding generations separately and preserve deletion markers during backfill. OpenSearch or another vector engine is a retrieval component, not the permission owner.

Walk through the result. Pause the vector indexer, revoke document D, then query text that strongly matches D. It may remain an internal candidate but must not appear in the response or model context. Show the source-version check and denial. Deliver a trace separating candidate recall, authorization filtering and answer generation.

Add semantic retrieval. Keep the same authorization boundary for vector candidates and source chunks. Embeddings are not a substitute for tenant isolation or revocation.