Synchronize files with resumable uploads and conflicts

Application background

Ana and Ben share a design folder across their laptops. Each laptop keeps local files and sends changes to a central service. If Ana edits while offline, her changes must be compared with any changes Ben made before she reconnects.

Files may be too large to upload in one uninterrupted request. Deletions also need a record so an old laptop does not mistake a missing file for something it should upload again.

Example walkthrough

A tombstone is a retained record that a file was deleted. A change cursor is a position from which a client can ask for later folder changes.

Your assignment

Deliver: Build file uploads, versioned metadata and a way for clients to retrieve later changes. Preserve conflicting edits and stop old clients from recreating deleted files.

Required behavior: Upload immutable content, then conditionally publish a metadata version against the client’s base version. Conflicts preserve both users’ bytes for resolution. Deletions are versioned tombstones until supported offline clients can no longer replay older state.

The required first milestone is a working local implementation of the behavior above. The numbered implementation steps define the scope. The cloud architecture is a later extension, not something the starter has already provisioned.

Get the code and run the supplied example

The code is in the public junior-to-staff repository. Install Git and Python 3.12+. No AWS account or Python packages are required for this first run. If you already have a checkout, use it and skip cloning.

git clone https://github.com/Soulful-Iris/junior-to-staff.git
cd junior-to-staff
python3 examples/architecture-starts/file_synchronization.py

Supplied file: examples/architecture-starts/file_synchronization.py. You can also read or download the source here (download file, source below).

"""Local mechanism demonstration for file-synchronization. No AWS resources are created."""
file={'version':3,'manifest':'hash-original','deleted':False}
def publish(base,manifest):
    if base!=file['version']: return {'status':409,'keep_local':manifest,'server':dict(file)}
    file.update(version=base+1,manifest=manifest); return dict(file)
print(publish(3,'hash-ana'))
print(publish(3,'hash-ben'))
file.update(version=5,deleted=True,manifest=None)
print(publish(3,'hash-old-laptop'))

This program is a mechanism demonstration: it runs the small scenario in one process and prints the result. It is not an HTTP service, a complete application, or an AWS deployment. A successful run demonstrates this mechanism only. It does not establish the workload or failure guarantees of the application you will build.

Example output from the supplied run:

Generated IDs and timestamps may differ. Compare the state transitions and outcomes.

{'version': 4, 'manifest': 'hash-ana', 'deleted': False}
{'status': 409, 'keep_local': 'hash-ben', 'server': {'version': 4, 'manifest': 'hash-ana', 'deleted': False}}
{'status': 409, 'keep_local': 'hash-old-laptop', 'server': {'version': 5, 'manifest': None, 'deleted': True}}

Set up your implementation workspace

Create work/file-synchronization/ in your checkout (or use a separate repository). Copy the supplied mechanism into that directory as mechanism.py, then extract its state transitions into functions you can call from your implementation. The record and module names below describe what you must implement. They are not a promise that files with those names already exist. Keep a README.md beside your implementation with its exact run commands and observed results.

Local components and state to implement

This table names the records, interfaces or decision inputs for your deliverable. Unless a name is explicitly linked to supplied source above, it is something you create. Implement the local state transitions first, then connect the HTTP, storage or worker boundaries required by the steps.

Implement the assignment

1. Upload content independently

Split files into bounded chunks, hash them and resume only missing chunks. Verify size/checksum before accepting a chunk. Uploading content does not make it visible. Orphan chunks can exist safely until metadata references them.

2. Publish metadata conditionally

Commit a file manifest only if the expected base version still matches. On conflict, preserve the losing local file and return current metadata. For binary documents, an explicit conflict copy is safer than pretending to merge arbitrary bytes.

3. Replay changes and deletions

Return bounded change pages after a folder cursor. Include rename and deletion tombstones with stable file identity. Path strings alone are ambiguous under concurrent renames. Reject stale metadata publication after a deletion generation.

4. Collect storage safely

Trace committed manifests and active uploads before deleting unreferenced chunks. Wait beyond the upload/retry window and respect shared references. Authorize current folder membership before metadata reads or chunk downloads.

Demonstrate the completed local result

Handoff: In your implementation README, include the start command, one successful operation, the failure case above and the resulting stored state or decision. State which dependencies are simulated. Someone with a fresh checkout should be able to reproduce this without your chat history.

Workload assumptions and capacity decisions

These are constructed exercise assumptions. The stated workload is a design target. The local demonstration does not establish that throughput. Use the estimation constants to check units before choosing capacity.

Map the local implementation to AWS

Deployment status: local only. Running the supplied command creates no AWS resources and configures no cloud connections. The diagram is a proposed deployment of the completed application. Each box needs either a deployed runtime, a provisioned service or an explicitly external dependency.

Read the diagram by following the arrows from the entry point: application code accepts the request or event, the state owner commits it, and any worker produces the later result. The table ties those roles to code and adapter work. Multiple boxes do not imply multiple Python files already exist.

S3 holds immutable bytes. DynamoDB decides which manifest is current. That split allows interrupted uploads and conflicting edits without exposing partial files or overwriting the winner’s bytes.

Provision resources, then connect the application

Use one disposable AWS environment for the cloud exercise. Put the named resources in infra/template.yaml or your existing IaC tool, pass resource IDs through configuration, and scope each runtime role to its own tables, buckets and queues. The diagram is a design to implement. It is not a claim that these resources have been deployed. Record the commands you used to deploy and remove the exercise resources.

For concrete provisioning commands, configuration wiring and cleanup, use the AWS foundation guide. It includes a deployable table/queue/object-storage foundation and explains which application and service adapters you still implement.

A provisioned queue or table does not make the local program use it. Configure resource IDs in the deployed runtime, replace the local adapter, and replay the same successful and failing operation against that runtime. Record the deployed commit and observable result, then remove the disposable resources using your infrastructure tool.

Extend the design after the baseline works

Worked follow-up: Move a file between folders with different permissions

Copying bytes and deleting the old entry can leave two visible names or preserve a link that was authorized under the old folder. Object identity, metadata version and download authority need separate treatment.

Revised architecture. Follow the changed responsibility and failure path below. This is a design to implement. The supplied local example does not provision these components.

What to implement. For folders in one metadata store, atomically change parent, metadata version and policy reference after checking permission on both source and destination. Emit one versioned change-feed event for clients. Bytes can stay under the same private object identity. Recheck current metadata for each authorized download, or state the residual lifetime of already issued signed URLs. An offline client with expired feed history must resnapshot without losing its unsynced drafts.

Walk through the result. Move file F from a public team folder into a private folder. A stale client submits an edit using the old metadata version and receives a conflict. A new download through the old path is denied. Keep the old signed-link lifetime visible if you chose that weaker contract. Supply the move transaction and client reconciliation example.

Add cross-folder moves with different permissions. Define the atomic metadata boundary and ensure old download authorization does not silently grant access under the destination’s policy.