# Edge worker for machine surfaces
> Serve llms.txt and .md mirrors from a Cloudflare Worker two ways: the CDN path (cf.cacheEverything) honors the origin's stale-while-revalidate; the Cache API does not, so it revalidates in the background via ctx.waitUntil.
## Run this with your coding agent
This prompt works with any coding agent that can fetch a URL. It reads this blueprint, checks your store against it, then implements what you approve. Review everything your agent changes before you ship it. It is your store.
How this works: https://agentmint.net/tools/coding-agent-workflow/
### Implement this blueprint
The agent checks your store against the blueprint, then builds what fits.
```text
Fetch and read the full document at https://agentmint.net/blueprints/edge-worker-machine-surfaces.md before doing anything else. If you cannot fetch URLs, tell me and I will paste the content.
You are implementing this blueprint in this repository's e-commerce store. State which platform you detect (Shopify, WooCommerce, Magento or Adobe Commerce, BigCommerce, or custom/headless) and apply that platform's notes from the document.
Work audit-first. Read the blueprint sections, check what this store already has against each one, and tell me the plan before you change anything. Then, only after I approve, implement the blueprint adapted to the detected platform, in small, reviewable steps. Stop and ask before anything destructive or anything that changes customer-facing behavior.
Report completion against the blueprint sections: done, partial with what remains, or not applicable with a reason. Name the section each change maps to.
Never invent factual product data to satisfy a check. Do not fabricate ratings or review counts, GTINs or other identifiers, delivery windows, prices, or return terms. Where required data is missing, list exactly what I need to supply and stop, rather than guessing.
End with a plain-language summary: what you added, what is partial, and the exact data or decisions you need from me.
```
This blueprint gives you two ready-to-deploy Cloudflare Worker configurations for caching your machine surfaces (llms.txt, .md mirrors, RSS or Atom feeds) at the edge, and the choice between them turns on one fact: the Workers Cache API does not honor `stale-while-revalidate`, so only the CDN path gives you true directive-driven stale serving.
## Key takeaways
- Variant A (CDN path) calls fetch with cf.cacheEverything so Cloudflare honors the origin's stale-while-revalidate directive: it serves stale immediately and revalidates in the background. Prefer it for llms.txt and .md mirrors.
- Variant B (Cache API) is self-contained, but the Cache API does not honor stale-while-revalidate, so it has to orchestrate the background refresh by hand with ctx.waitUntil.
- The reverse-proxy equivalent is nginx proxy_cache_use_stale updating, which serves a stale entry while one request refreshes it upstream.
- Every TTL here is illustrative, not a spec requirement. Keep price and availability on a short TTL so an agent never quotes a stale offer.
## Why there are two variants
A Worker is a small piece of code that runs at Cloudflare's edge in front of your origin, so it is the natural place to cache low-churn machine surfaces and keep agent refetches off your origin. The question is how you get stale-while-revalidate behavior: the edge returns the cached copy instantly and refreshes it out of band. That pattern is what protects an origin from crawler bursts without ever making an agent wait.
A Workers module exports a default object with an `async fetch(request, env, ctx)` handler, and `ctx.waitUntil(promise)` extends the Worker's lifetime so it can finish background work after the response has already been returned to the client.
The load-bearing constraint is what these two Worker APIs do with the `stale-while-revalidate` directive. The Workers Cache API (`caches.default`) does not honor the `stale-while-revalidate` or `stale-if-error` directives. A Worker that caches through that API therefore cannot get automatic stale serving from a header; once its own freshness window lapses, the next `cache.match` is simply a miss. To get real directive-driven stale serving you have to go through Cloudflare's CDN cache instead. Cloudflare's CDN serves stale content during revalidation only when the origin's `Cache-Control` includes `stale-while-revalidate`; the first request after the fresh window triggers an asynchronous background revalidation and immediately returns the stale response with an `UPDATING` cache status.
So the split is clean: Variant A rides the CDN path and lets the origin's header drive real SWR; Variant B stays inside the Cache API and rebuilds the stale-then-refresh behavior by hand.
## Variant A: CDN path (preferred for llms.txt and .md mirrors)
```js
export default {
async fetch(request, env, ctx) {
// Origin sends: Cache-Control: max-age=3600, stale-while-revalidate=86400
// cacheEverything makes Cloudflare's CDN cache honor those directives:
// serve stale + async-revalidate (UPDATING) after the fresh window.
// The TTLs live in the ORIGIN headers (illustrative: 1h fresh / 24h stale).
return fetch(request, { cf: { cacheEverything: true } });
}
};
```
The whole mechanism lives in one line and one origin header. `cf: { cacheEverything: true }` tells Cloudflare to run this response through the CDN cache, and the CDN cache is the layer that reads `stale-while-revalidate` and produces the `UPDATING` flow described above. Because the TTLs live in the origin's `Cache-Control` and not in the Worker, you tune freshness per surface by changing a header, not by redeploying code.
We prefer this variant for llms.txt and .md mirrors because stale serving there is nearly free: those files change rarely, and an agent that reads a link index or a Markdown mirror one revalidation behind loses nothing. Variant A is also the smaller surface to maintain, since the freshness policy is data (a header) rather than logic.
Note the header shape. Variant A pairs `max-age` with `stale-while-revalidate` rather than using `s-maxage`. Cloudflare's cache-control guidance notes that `s-maxage` implies `proxy-revalidate`, so a shared cache must not serve stale without revalidating, and it advises against combining `s-maxage` with `stale-while-revalidate`. That is why the SWR path in Variant A is expressed with `max-age`, while the manual path in Variant B (which has no SWR to combine with) uses `s-maxage` on its own.
## Variant B: Cache API, manual refresh (self-contained)
```js
export default {
async fetch(request, env, ctx) {
if (request.method !== "GET") return fetch(request);
const cache = caches.default;
const cacheKey = new Request(request.url, request);
const cached = await cache.match(cacheKey);
if (cached) {
// HIT: serve now, revalidate in the background. NOTE: the Cache API does
// NOT honor stale-while-revalidate, so we orchestrate the refresh manually.
ctx.waitUntil(refresh(cache, cacheKey, request));
return cached;
}
const response = await fetch(request);
const toStore = new Response(response.body, response);
toStore.headers.set("Cache-Control", "s-maxage=3600"); // illustrative TTL
ctx.waitUntil(cache.put(cacheKey, toStore.clone()));
return toStore;
}
};
async function refresh(cache, cacheKey, request) {
const response = await fetch(request);
const toStore = new Response(response.body, response);
toStore.headers.set("Cache-Control", "s-maxage=3600"); // illustrative TTL
await cache.put(cacheKey, toStore);
}
```
This variant reconstructs the stale-then-refresh pattern that the Cache API will not give you for free. On a hit it returns the cached copy right away and schedules `refresh()` with `ctx.waitUntil`, so the fetch to your origin happens after the agent already has its response. The `waitUntil` handle is what keeps that background fetch from being cut off when the response returns.
Two documented Cache API behaviors shape the code. `cache.match` and `cache.put` respect `Cache-Control` including `s-maxage`, and `cache.put` refuses to store responses to non-GET requests, `206` partial responses, responses with `Vary: *`, or responses carrying a `Set-Cookie` header. The `if (request.method !== "GET")` guard exists because non-GET requests cannot be cached, so there is no reason to route them through the cache lookup. And because the Cache API ignores `stale-while-revalidate` entirely, the code never sets it; `s-maxage` alone defines the freshness window, and the manual `refresh` replaces the SWR the API will not honor.
> **Variant B is an approximation, not native SWR**
>
> Once the `s-maxage` window passes, `cache.match` returns a miss, and that request pays the full origin round trip while the fresh copy is stored. Variant B therefore approximates stale-while-revalidate rather than reproducing it: it revalidates in the background on hits, but it does not serve stale past expiry the way Cloudflare's CDN `UPDATING` flow does. If a warm-refresh-on-every-request guarantee matters, prefer Variant A.
## The reverse-proxy equivalent (nginx)
If you cache on your own reverse proxy instead of a Worker, the same shape of behavior has a documented switch.
```nginx
# Reverse-proxy equivalent of Variant A's stale-while-revalidate behavior:
# serve the stale entry while one request refreshes it upstream, instead of
# stampeding the origin. proxy_cache_valid sets the fresh window (illustrative).
proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=machine:10m inactive=24h;
server {
location ~ \.(txt|md)$ {
proxy_pass http://origin;
proxy_cache machine;
proxy_cache_valid 200 1h; # illustrative fresh window
proxy_cache_use_stale updating; # serve stale while it refreshes
}
}
```
In nginx, `proxy_cache_use_stale` with the `updating` parameter serves a stale cached response while that entry is being refreshed, which minimizes upstream requests (the default is `off`); since version 1.11.10 nginx can also serve stale directly from an upstream's `stale-while-revalidate` or `stale-if-error` extensions, at lower priority than the `proxy_cache_use_stale` parameters. This produces the same practical outcome as Variant A, serve the cached copy now and refresh it out of band, but it is a distinct mechanism from Cloudflare's asynchronous `UPDATING` status rather than an identical one, so treat them as analogous rather than interchangeable.
## Freshness: what you must not serve stale
> Illustrative example, not measured data.
A store puts llms.txt and every .md mirror behind Variant A with a 1 hour fresh window and a 24 hour stale window, and puts the product-detail pages that carry live price and availability on a 60 second TTL with no stale serving. The link index and Markdown copies are cheap to serve slightly behind; the offer facts an agent quotes are not. This is an illustration of the tradeoff, not a measured configuration.
Every TTL in this blueprint (1 hour fresh, 24 hours stale, `s-maxage=3600`, `proxy_cache_valid 200 1h`) is an illustrative editorial choice, not a spec requirement. The directives and their behavior are documented; the specific seconds are yours to set per surface. Stale serving is safe for low-churn surfaces like llms.txt and Markdown mirrors, but price and availability are the opposite: keep them on a short TTL or bypass the cache entirely, because an agent that reads a long-stale offer may quote a price or a stock state you have already changed.
## Which to use, and where this sits
Reach for Variant A on llms.txt, .md mirrors, and feeds, where directive-driven SWR is free and the config is one line plus one origin header. Reach for Variant B when you need caching logic to live inside the Worker, for example when the freshness window depends on request shape rather than a static header, and accept that it approximates SWR rather than reproducing it. On a non-Cloudflare stack, the nginx block above is the equivalent lever.
This is the serving half of making machine surfaces fast for agents. The chapter [serving agent traffic](/handbook/serving-agent-traffic/) covers the surrounding judgment, which crawlers to expect and how caching protects an origin from refetch bursts, and [AI crawlers, robots.txt and llms.txt for stores](/ai-crawlers-robots-llms-txt/) covers the access-policy side. For what these Workers should be caching, see the companion blueprints: [llms.txt for e-commerce catalogs](/blueprints/llms-txt-for-ecommerce-catalogs/), the [product Markdown mirror](/blueprints/product-markdown-mirror/), and [token-efficient product JSON-LD](/blueprints/token-efficient-product-json-ld/). For the format itself, see the [llms.txt](/glossary/#llms-txt) glossary entry.