Why Re-Architect?
We needed a content delivery pipeline that could keep dunelm.com and our Headless CMS in sync reliably, scale with traffic, and give us clear visibility into module dependencies and cache invalidation. The re-architecture delivered an event-driven, serverless design we could ship in 3 months.
Architecture Overview
The new pipeline is built around user and search interactions, Lambda functions, and SQS queues for decoupled processing:
- User and search: Traffic from dunelm.com and our SaaS search engine triggers the pipeline.
- Send to SaaS search engine: A Lambda sends data to the search engine and triggers cache invalidation via a Cache Invalidator Service.
- Module computation: Messages flow through ModuleComputationSQS to a ModuleComputation Lambda that builds and validates the object (with a DLQ for failed messages).
- Data fetch and dependencies: FetchHeadlessCMSDataSQS feeds a FetchData Lambda that consults a Module relation DB to find modules that depend on the updated one, fetches those (and their dependencies recursively), and sends top-level parents to SQS. The same Lambda updates dependency lists in the Module relation DB.
- Triggering updates: TriggerUpdatesSQS drives a TriggerUpdate Lambda that extracts updated module IDs and forwards them. The pipeline ends at a POST /webhook that we configured our Headless CMS to receive, so content updates are applied consistently.
Key Components
Module Relation DB
We maintain a dedicated store (e.g. DynamoDB) of module dependencies. The FetchData Lambda reads which modules depend on a given updated module and writes back updated dependency lists, enabling correct, recursive propagation of content changes.
Cache Invalidation
A dedicated Cache Invalidator Service is invoked when we send data to the SaaS search engine, so page caches are invalidated in step with content and search index updates.
Fault Tolerance
Dead Letter Queues (ModuleComputationDLQ, TriggerUpdateDLQ) capture failed messages so we can inspect and retry without losing events.
Results
The re-architecture gave us a clear, event-driven content delivery path, better dependency tracking, reliable cache invalidation, and a foundation we can extend. For the full story, diagrams, and implementation details, check out the complete article on Dunelm Engineering.