Build a durable crawler with per-host limits

Application background

A research team wants a searchable collection of public pages. A crawler starts with some URLs, downloads a page, extracts more links and schedules those links for later visits. Its frontier is the saved list of URLs waiting to be visited.

The crawler must share work across machines without overwhelming any one website. Some pages generate endless links, and a redirect can point somewhere the crawler should not access. A restart must not lose all pending work.

Example walkthrough

A worker is a process that takes pending URLs and fetches them. Track both its job ownership and the remote host's limits rather than using only a global worker count.

Sizing that affects this decision

One billion waiting URL entries at 200 bytes each is about 200 GB before indexes and history. Fetching 50,000 pages/s at 100 KiB each is about 4.77 GiB/s of incoming page data. A one-fetch-at-a-time rule for each host still applies, so the total rate requires enough different eligible hosts.

These are exercise assumptions. The estimation reference explains the units and approximations. They do not establish the local demo's measured capacity.

Your assignment

Deliver: Build a saved URL frontier, host-aware work assignment and a bounded public-page fetcher. Keep pending work and completed observations understandable after worker restarts.

Required behavior: Fetch only allowed public HTTP(S) destinations, respect the configured robots and host policy, and record a durable frontier state. Fetches may repeat. Canonical URL identity and content/version handling prevent uncontrolled duplicate work.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/web_crawler.py

Supplied file: examples/architecture-starts/web_crawler.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for web-crawler. No AWS resources are created."""
from urllib.parse import urlsplit,urlunsplit
seen=set()
def normalize(url):
    p=urlsplit(url)
    if p.scheme not in ('http','https') or p.username or p.password: raise ValueError('unsupported URL')
    return urlunsplit((p.scheme.lower(),p.netloc.lower(),p.path or '/',p.query,''))
for url in ['https://EXAMPLE.invalid/a#one','https://example.invalid/a#two','https://example.invalid/a?sort=1']:
    key=normalize(url); print(key,'duplicate' if key in seen else 'new'); seen.add(key)
print('Normalization only; network destination validation is a separate required boundary.')

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

https://example.invalid/a new
https://example.invalid/a duplicate
https://example.invalid/a?sort=1 new
Normalization only; network destination validation is a separate required boundary.

Set up your implementation workspace

Create work/web-crawler/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Build a bounded frontier

Normalize scheme/host and remove fragments while preserving query semantics unless a source-specific rule proves equivalence. Version normalization rules. Persist discovered URLs idempotently and cap URL length, crawl depth and per-host discovery budget to contain calendar traps.

2. Coordinate host scheduling

Fetch and cache robots policy under a documented failure rule. A shared host scheduler issues leases respecting concurrency and next_allowed. Workers cannot independently apply one-request-per-host and accidentally multiply the limit across the fleet.

3. Make outbound fetches safe

Resolve and validate all destination addresses, pin the validated connection target, and repeat validation for every redirect. Block private, loopback, link-local and metadata ranges. Bound redirects, body bytes, decompression ratio and total time. Do not execute page scripts in the first milestone.

4. Record outcomes and recover

Store status, final URL, content hash and retry time before completing the lease. Crash recovery can refetch, so content storage is immutable and frontier completion is conditional on the current attempt. Extract links under a bounded parser and preserve provenance.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

The frontier owns work identity. The host scheduler owns permission to fetch now. A distributed queue alone supplies neither robots policy nor fleet-wide per-host fairness.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Migrate URL identity rules without duplicating the crawl

Removing a tracking parameter may correctly merge duplicates, while removing an item parameter can collapse different pages. Rekeying every row in place makes those mistakes difficult to reverse.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Retain raw URLs and version the normalization algorithm. Build a candidate mapping from old keys to new keys, preserving aliases and recording collisions. Classify collision samples before switching scheduling ownership. During transition, use a migration authority to prevent both keys from independently scheduling the same fetch. Keep per-host pacing attached to the actual destination host. Use a shadow frontier generation and a bounded migration worker rather than restarting the entire crawl.

Walk through the result. Compare /item?id=1&utm_source=x with /item?id=1 and /item?id=2. The first two may merge under the stated rule, while the third remains distinct. Revert the candidate mapping and show that all raw URLs and prior fetch records remain available. Deliver collision counts and reviewed examples, not only total row counts.

Change normalization rules after 500 million URLs are stored. Plan identity migration and aliasing so new rules do not duplicate the whole crawl or collapse URLs whose query parameters carry meaning.