Collect news feeds with freshness and deduplication

Application background

A news reader regularly downloads lists of new articles from publishers. It stores article links, groups repeated stories and builds the feed a reader sees. Visiting each publisher on a schedule is called polling.

Some publishers update quickly, some stop responding and some ask the reader to slow down. An empty download does not always mean there is no news. Readers need to know when each source was last checked successfully.

Example walkthrough

Freshness describes how recent the information is. Track it separately from whether your own feed endpoint can return a response.

Your assignment

Deliver: Build publisher polling, duplicate grouping and a reader feed. Show when each source was last checked successfully and preserve that evidence during publisher failures.

Required behavior: Ingest RSS/Atom updates, retain source attribution and publish normalized stories with stable IDs. Freshness is measured per reachable source. A publisher outage must not be presented as an empty successful feed.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/news_aggregator.py

Supplied file: examples/architecture-starts/news_aggregator.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for news-aggregator. No AWS resources are created."""
articles={}; etag='v1'
def ingest(source,item,url,title):
    key=(source,item); articles[key]={'url':url,'title':title}
    return 'upserted '+str(key)
print(ingest('publisher-a','42','https://example.invalid/story','First title'))
print(ingest('publisher-a','42','https://example.invalid/story','Corrected title'))
print('304 response: retain existing articles',articles)

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

upserted ('publisher-a', '42')
upserted ('publisher-a', '42')
304 response: retain existing articles {('publisher-a', '42'): {'url': 'https://example.invalid/story', 'title': 'Corrected title'}}

Set up your implementation workspace

Create work/news-aggregator/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Poll one source correctly

Send If-None-Match/If-Modified-Since when supported. Treat 304 as successful unchanged content, not deletion. Bound response size and parsing time, disable unsafe XML entity expansion, and preserve the original feed/item identity.

2. Coordinate host load

Schedule next_poll by source freshness and host allowance. Respect bounded Retry-After and back off failing publishers. Apply outbound destination validation to feed URLs and redirects. A submitted feed must not become an internal-network fetch endpoint.

3. Normalize without losing attribution

Keep publisher item IDs, observed canonical URLs, published/updated times and retrieved time separately. Group likely duplicates while retaining each source record. A corrected headline updates a revision. A mistaken cluster merge must be reversible.

4. Serve an explicit read model

Index published stories and serve bounded pages with cache headers. Show source freshness and partial ingestion status in the editor view. A feed disappearing should trigger a source incident, not mass deletion of historical articles.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

Polling and reader delivery have different scaling patterns. Conditional requests save transfer, while source/item identity preserves corrections without manufacturing duplicate stories.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Rebuild multilingual story groups without losing attribution

Two headlines about an election may describe different events. A similarity score can nominate a pair but cannot safely replace stable article identity. A mistaken merge must be reversible.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Keep immutable article IDs, source URLs and language metadata. Build a separately versioned article-to-story mapping, leaving the old mapping available while the new one is evaluated. Compare known false merges and missed duplicates by language. Switch readers to one mapping version and preserve cursor behavior across the switch. Use S3 for retained source material and a separate index generation for derived groups, with explicit attribution in each result.

Walk through the result. Construct three articles: two translations of the same announcement and one later correction. Show the desired grouping and the evidence that distinguishes the correction. Introduce an incorrect merge, roll back the mapping pointer and demonstrate that source URLs and article identities never changed. Deliver a before/after membership table.

Add multilingual story grouping. Preserve original language and attribution, and evaluate false merges separately from missed duplicates. Similarity is a candidate signal, not identity.