Process uploads and publish complete video renditions

Application background

An instructor uploads a large training video. Students need smaller versions suited to different screens and internet speeds. The processing service creates those versions before making the video available to watch.

A rendition is one prepared version of the video. A playback manifest lists the files a player can request. Publishing that list before its files exist would give students a video that starts and then fails.

Example walkthrough

Upload success means the source file arrived. Playback readiness is a later result that the application must track explicitly.

Your assignment

Deliver: Build upload and processing records for each video version. Publish a playback manifest only when all required output files are complete, and allow failed work to resume.

Required behavior: Create an upload session, accept a completed source object, and expose processing status. Publish a versioned playback manifest only after every required rendition is verified. Reprocessing creates a new generation without overwriting the currently published one.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/video_processing.py

Supplied file: examples/architecture-starts/video_processing.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for video-processing. No AWS resources are created."""
required={'360p','720p','1080p'}; outputs={}; published=None
def publish():
    global published
    if not required.issubset(outputs): return 'still processing: '+str(sorted(required-outputs.keys()))
    published=dict(outputs); return 'ready'
outputs.update({'360p':'g2/360.mp4','720p':'g2/720.mp4'})
print(publish()); outputs['1080p']='g2/1080.mp4'; print(publish()); print(published)

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

still processing: ['1080p']
ready
{'360p': 'g2/360.mp4', '720p': 'g2/720.mp4', '1080p': 'g2/1080.mp4'}

Set up your implementation workspace

Create work/video-processing/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Accept and verify a source upload

Issue a short-lived upload authorization scoped to owner, key, size and allowed type. On completion, inspect object metadata and checksum where available. Object notifications may duplicate, so create processing identity from video and source version rather than notification delivery ID.

2. Run versioned processing

Store the required rendition list and generation before submission. Use MediaConvert for supported managed transcoding or a bounded container worker for custom processing. Retry failed work under the same logical job while writing immutable attempt-specific outputs.

3. Publish a complete manifest

Verify that each required output is present, readable and belongs to the current source generation. Commit ready plus the manifest pointer conditionally. A late success from an old generation must not replace newer output.

4. Expose progress and repair

Show queued, processing, failed and ready with a reason and last progress time. Track queue wait separately from encoding duration. Keep original input until the documented reprocessing/retention policy allows removal. Clean orphan output after active attempts expire.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

Transcoding completion and publication are separate decisions. Step Functions tracks workflow progress. DynamoDB’s conditional pointer makes one complete generation visible to readers.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Publish a new video generation while viewers use the old one

Overwriting segment files in place lets a player combine an old manifest with new bytes. Deleting the old generation immediately can break viewers who already loaded its manifest.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. Write segments under immutable generation-specific S3 keys. Record a manifest only after all required outputs are complete and verified. Atomically change the active generation pointer using the expected prior version. Existing sessions keep their old manifest until the documented session and delivery-cache retention bound. Garbage collection checks that bound before deleting old objects. Failed reprocessing leaves the old pointer intact.

Walk through the result. Play generation 4 while generation 5 fails halfway through encoding. New sessions must still receive 4. Complete 5 and switch the pointer once. Show an existing session fetching a generation-4 segment and a new session fetching generation 5. Hand over the publication condition and deletion eligibility rule.

Allow students to keep watching during reprocessing. Keep the previous manifest valid until the new generation is complete, and define when old segments can be removed without breaking active sessions.