Build evidence-backed answers with permission rechecks

Application background

Ana works in operations and asks an internal assistant what the expense policy allows. The application looks up relevant passages, drafts an answer and shows the passages supporting it. Ana should be able to follow the references and judge whether the answer matches them.

Not every document is available to every employee. Ana may read expenses but not acquisition planning. A source can also become unavailable after lookup but before the response is sent. The system must handle that change without exposing protected text.

A proposed answer response

The interface you add around the supplied assistant workflow should make an unsupported answer distinguishable from a supported one. These are illustrative response shapes:

{"status":"answered","answer":"…","sources":[{"document_id":"expense-policy","version":3,"passage_id":"meals"}]}

{"status":"insufficient_evidence","answer":null,"sources":[]}

A source reference must resolve to a real permitted passage. A plausible-looking document ID does not count as evidence.

An evidence-backed answer is one whose claims can be tied to permitted source material. An abstention is an explicit response that the application cannot answer from such evidence.

Your assignment

Deliver: Extend the supplied assistant workflow with a usable evidence view. Handle permission removal between retrieving a passage and returning an answer that uses it.

Required behavior: Current authorization applies before reading source text and again before returning an answer. Citations identify real document versions and ranges. Retrieved instructions cannot expand tool or document access.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/ai-systems/demo.py assistant

Supplied code: AI workflow implementation, starting at demo.py. These are local workflows with fixture providers and temporary storage. They do not include the user interface or connect to the AWS services in the diagram.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

[
  {
    "request": {
      "action": "document.put",
… (more output follows)

Set up your implementation workspace

Create work/01-evidence-desk/ in your checkout (or use a separate repository). Copy examples/ai-systems/ there so you can change the workflow and its storage/provider boundaries together. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Run the reference and inspect evidence

Execute the assistant demo and identify the authorized sources, citations and revocation result. Read the existing implementation walkthrough below alongside the source. Keep this working local boundary before substituting an embedding index or hosted model.

2. Build a search-only baseline

Add an authenticated query route that returns authorized excerpts without generation. Ingest versioned documents and deletion events. Measure whether the right evidence is found before changing prompts to compensate for missing retrieval.

3. Add bounded generation and final authorization

Generate only from the selected evidence set, keep prompt/model/index versions, and map claims to source ranges. Recheck current source permissions at response finalization. Discard content whose authorization changed during inference.

4. Expose useful uncertainty

Show citations and source dates, abstain when evidence is absent or contradictory, and retain redacted diagnostic evidence. Compare the generated workflow with the search-only baseline on the same supported/unsupported questions.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

This cloud mapping preserves the local reference’s evidence and authorization boundaries. The model is one adapter inside the workflow, not the component that decides what Ana may read.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Revoke a source while an answer is being generated

Alice starts an answer using a payroll passage and loses access while the model is running. Checking only before retrieval would still deliver a new answer containing now-revoked material.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Keep the source IDs, versions and permission scope used in each generation. Before releasing the complete answer, check that the caller remains authorized for every cited or included source. If any decision is unavailable or revoked, discard or regenerate from an authorized subset under a documented policy. For a strict contract, buffer the answer until this check. Already streamed text cannot be recalled. In two regions, either reach the authoritative decision or expose a bounded-revocation contract explicitly.

Walk through the result. Pause after retrieval, revoke source D and resume generation. No text from D reaches the user. Show a safe insufficient-evidence outcome and record the suppressed answer only if your privacy policy permits it. Repeat with the permission service unavailable. Hand over the source manifest, final authorization point and user response.

Add a document revision that contradicts an older policy. Decide which version is authoritative and show the conflict explicitly instead of asking the model to choose without a policy.

if subject not in document["readers"]:
    continue
# Only after authorization do we fetch and consider the source body.

python3.12 examples/ai-systems/demo.py assistant