Build an SSRF-resistant link preview fetcher

Application background

Alice pastes a public article URL into a reading list. To display its title, the application's server makes an HTTP request to that URL. This is a server-side fetch, separate from the browser request that saved Alice's bookmark.

An attacker can submit a URL too. They might point it at a private company service, or use a public address that redirects there. If the preview server fetches anything it is given, it becomes a way to reach systems the attacker cannot access directly.

A request that exposes the boundary

These are outputs from the supplied address-policy model. It uses controlled resolved-address inputs and makes no network request:

https://example.com True
http://metadata.invalid False
https://mixed.invalid False

The first input resolves only to a public address. The second uses 169.254.169.254, a link-local address. The third includes both a public address and 127.0.0.1, the local machine. Rejecting the mixed result matters too. Your real fetcher still needs to connect to the validated address and recheck redirects.

SSRF means server-side request forgery. The attacker tricks your server into making an unintended request. Validation must apply to the address actually contacted, including each redirect.

Your assignment

Deliver: Replace simulated title lookup with a public-URL fetcher. Validate the address actually contacted, repeat the check for redirects and limit time, response size and other work.

Required behavior: Only explicitly supported public HTTP(S) destinations may be fetched. Validate the resolved addresses used by the actual connection and repeat the policy for every redirect. Bound all response work.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/03_the_fetch_that_cannot_be_aimed_inward.py

Supplied file: examples/architecture-starts/03_the_fetch_that_cannot_be_aimed_inward.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for 03-the-fetch-that-cannot-be-aimed-inward. No AWS resources are created."""
import ipaddress
from urllib.parse import urlsplit
def permitted(url,resolved):
    p=urlsplit(url)
    if p.scheme not in ('http','https') or p.username or p.password or p.port not in (None,80,443): return False
    return bool(resolved) and all(ipaddress.ip_address(a).is_global for a in resolved)
for url,addresses in [('https://example.com',['93.184.216.34']),('http://metadata.invalid',['169.254.169.254']),('https://mixed.invalid',['93.184.216.34','127.0.0.1'])]:
    print(url,permitted(url,addresses))
print('Policy demonstration only; the network adapter must pin the validated IP.')

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

https://example.com True
http://metadata.invalid False
https://mixed.invalid False
Policy demonstration only; the network adapter must pin the validated IP.

Run the application you will extend

The reading-list API setup guide gives you a real local HTTP server, SQLite database, save/list/edit requests and controlled title success/timeout behavior. Start it in one terminal and send the documented curl requests from another. Read that setup before following the implementation steps below. The demo above isolates this lesson's mechanism. The server is where you integrate it.

For a first run, start this in terminal 1 from the repository root:

python3 examples/reading-list-starter/app.py --db /tmp/reading-list.sqlite3

In terminal 2, save one bookmark with a controlled title timeout:

curl -i http://127.0.0.1:8080/bookmarks \
  -H 'X-Demo-User: alice' -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com/docs","title_mode":"timeout"}'

Expect 201 Created, a bookmark id and title_status: "timeout". The URL is persisted despite the title failure. This is the supplied baseline. The assignment adds the behavior described above. The lookup is a fixture, so no external website is contacted. For members Bob or Ben in a scenario, use the starter's second demo identity bob. Alice or Ana corresponds to alice.

Work in your own branch or copy examples/reading-list-starter/ to work/03-the-fetch-that-cannot-be-aimed-inward/. app.py exists in that directory. Add the modules named below there as you separate HTTP, storage and background work. The server has demo membership, not production authentication.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Parse and reject before resolving

Use a standard URL parser, reject embedded credentials, unsupported schemes and unexpected ports, and normalize the host deliberately. Handle IPv4, IPv6 and mapped-address forms consistently. Do not rely on a substring blacklist such as localhost.

2. Bind validation to connection

Resolve the hostname, reject disallowed addresses and connect through an adapter that uses the validated IP while preserving Host and TLS SNI/certificate checks for the original hostname. Disable implicit environment proxies unless explicitly configured under the same policy.

3. Control every redirect and body

Disable automatic redirects in the HTTP client. Parse each Location, resolve and validate again, and stop after three redirects. Enforce a total deadline, decompressed byte limit and bounded content parsing. A tiny compressed response can expand into a large body.

4. Add network defense and diagnostics

Restrict worker egress and metadata access independently of application checks. Return a generic unavailable preview to the user while recording a safe policy reason. Use local controlled public/private-address fixtures to observe denials. Do not probe real internal services.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

Network controls reduce exposure, while the HTTP adapter enforces which public destination was actually approved. A URL regex or a DNS check disconnected from the socket is insufficient.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Allow customer-managed authenticated previews. Keep credentials scoped to approved origins and strip them on redirects. Define whether cross-origin redirects are allowed at all.

Follow-up scenarios and worked designs

Follow-up 1 · DNS returns mixed addresses

One hostname returns both a public IPv4 and private IPv6 address. What does your policy do?

Worked design and implementation

For this exercise reject mixed unsafe answers rather than relying on client selection order. Test IPv4-mapped IPv6 and redirects with the same canonical address policy.

Walk the connection boundary. A lookup returns public address P and private address Q. Reject the destination before choosing either address under this exercise policy. Canonicalize IPv4, IPv6 and IPv4-mapped IPv6 before classification. For an accepted destination, connect using a validated address while preserving the intended HTTP host and TLS identity. An unrestricted second DNS lookup in the HTTP client would reopen the gap.

Your deliverable: record decisions for mixed answers, a public-to-private redirect and a redirect loop. Apply the same destination policy at every redirect and keep one total deadline and byte budget. Do not make real requests to internal metadata addresses to demonstrate rejection.

Follow-up 2 · A future worker reuses fetching

The queued worker gains new credentials and network routes. Is the guard enough?

Worked design and implementation

Reuse the same client and add restricted egress and least-privilege credentials. Application validation and network isolation protect different boundaries. Neither proves the other.

This is an architecture change. A worker with database credentials and private network routes has a different failure impact from a public-only preview process. Split URL fetching into a narrowly privileged service. Let the job worker submit a destination and receive bounded content through a defined interface. Keep application destination validation in that service and enforce network egress independently.

For AWS, an ECS fetching task can be placed behind controlled egress with no unnecessary data-store role. Merely placing it in a private subnet does not block every private destination. Define which outbound paths are actually allowed. Deliver an access matrix for the worker and fetcher, plus a rejected redirect transcript.

Revised flow. These are proposed components to implement, not extra services started by the supplied demo.