Caching and Search for a Static Site

This post records two pieces of infrastructure on this site: the Service Worker caching strategy and the Pagefind on-site search.

Caching strategy answers “how resources are stored and when they count as fresh”; on-site search answers “how content is queried, and how you do it with zero runtime services”.

The two work closely together. The search index is a batch of static shards named by content hash — exactly the kind of client a caching strategy worries about least. And the bucket design reserved a place for it from the start.

The first half covers the full Service Worker caching strategy; the second half covers Pagefind’s indexing scheme and one well-hidden pitfall.

First half: Service Worker caching strategy

Once a static blog sits behind a CDN, the first visit is already fast enough; the Service Worker’s (SW) value is mostly in repeat visits.

Skip even the origin revalidation round trip, so switching pages costs zero waiting. It also happens to settle a billing problem specific to static sites: some APIs bill per call, and a cache in the right place is real money.

Bucket first

Cache Storage is one big open floor; mixing everything into it tangles both cleanup and policy.

This site splits resources into four buckets by “how they go stale”:

The direct payoff of bucketing shows up in the activate phase: on a version bump you only change the bucket-name suffix (frosti-v2 → v3), and every bucket not on the new list is deleted wholesale — cleanup is a one-line filter.

If everything sat in one bucket, you would have to judge each entry: “which category is this cache in, and should it be cleared”.

One semantic needs clarifying: “never stale” for the static bucket means the policy never clears it on purpose (hash naming guarantees a hit is correct), not “the browser guarantees it persists”.

Under storage pressure, Cache Storage entries can be evicted by the browser’s own eviction policy; if one is evicted, the next request refills from the network and correctness is unaffected.

It and on-site prefetching (see the first half of the page-switching and first-paint post) divide the labor rather than compete: prefetching fills HTML into the HTTP cache, and navigations the SW has not taken over consume that layer directly. Once the SW does take over, SWR’s background revalidate requests go through the browser network stack and still hit what prefetching left behind.

The two mechanisms keep separate books and are naturally complementary.

Strategy matrix

cache-first: hashed assets, images, fonts, the search index.

A cache hit returns directly; only a miss goes to the network and writes back.

Filenames under /_astro/ carry their own content hash, URL and content map one to one, a hit is by definition correct, and origin revalidation is never needed.

Same for images and fonts (provided deployment renames files when replacing them — otherwise you step on “the image was updated but users see the old one”, which is cache-first’s only price).

stale-while-revalidate (SWR): HTML pages and CDN assets.

HTML is the most delicate category. It cannot be cache-first (after a release users would see the old site forever), and it cannot be network-first (every page switch waits on the network, and the repeat-visit advantage evaporates).

SWR is the compromise: serve from cache instantly, fire a request in the background to update it, and the next visit gets the new version.

The price is that this one visit may show the old version. Entirely acceptable for blog content, and it converges with the version mechanism below.

cache-first + TTL: APIs billed per call.

If the comment system hit its endpoint on every navigation, the Vercel function call volume would look ugly.

The policy: while the cache hits and the TTL (5 minutes) has not expired, not even a network request is sent; only on expiry does a real revalidate happen.

There is a small implementation trick: the Cache API stores the response’s Date header, and comparing it against the current time is a free freshness check.

There is also a fallback sweep in the activate phase: API responses older than 30 minutes are deleted outright, so the bucket cannot balloon.

Version rollover and takeover timing

Three lifecycle details decide the upgrade experience:

1. skipWaiting(): once ready, the new SW takes over immediately instead of waiting for every page under the old SW to close. Combined with SWR for HTML, the user sees at most one stale page, and the next navigation is fresh;
2. clients.claim(): after takeover, even “this page that no SW has controlled yet” comes under management, so the very first install already enjoys the cache;
3. activate cleanup: delete old buckets outside the allowlist plus expired API responses, so leftovers from older versions do not pile up forever.

One honest declaration belongs in the design doc: HTML over SWR means “the user may briefly see an old version”.

For a content site that is the right trade-off; for an app with frequent releases where old and new pages interact, switch to network-first or add a “new version found, click to refresh” prompt.

Another boundary we “did not build” is worth recording too: the offline fallback page.

Under SWR almost every visited page has a cache, but a user whose “first visit is offline” still gets the browser error page.

The thorough PWA approach is to precache a lightweight offline page and fall back to it when fetch fails. The payoff for a blog is tiny (who opens a blog offline for the first time?), so for now it is out of scope.

It is written down here as an explicit record that this was deliberately not built.

What never to cache

This site has a visitor counter: one POST to the stats endpoint per session.

The first line of the SW’s fetch interception is:

1
if (request.method !== 'GET') return;

POST/PUT/DELETE carry side effects; caching them at best loses your statistics (the response gets cached, every later “request” is a cache replay, and the server never sees it), at worst produces duplicate submissions.

Also to be routed around: endpoints carrying Authorization, latency-sensitive WebSockets (which the SW cannot intercept anyway), and any analytics where “the request is the event”.

The first row of a caching policy table is always the exclusion.

Two engineering details

Cache key normalization.

Cache Storage stores and retrieves by URL verbatim, so /blog/foo/ and /blog/foo/?utm_source=twitter for the same article are two independent cache entries.

Tracking params, pagination params, and scroll-restoration params all manufacture redundant entries of “same content, different key”, and the HTML bucket slowly bloats because of it.

The fix is to normalize the URL before the SW looks up the cache: strip known-harmless query params (utm*, fbclid, that family), and read and write only under the normalized key. Hit rate goes up and the bucket stays clean.

Be careful not to strip functional params (search terms, page numbers); a whitelist mindset is steadier than a blacklist.

The update prompt UI.

skipWaiting() means the user sees at most one stale page, and “the next navigation is fresh” is enough for most cases. But for sites where releases interact across versions, an explicit prompt is worth adding: listen with registration.addEventListener('updatefound', ...) for the new SW finishing installation, then pop a small bar on the page — “a new version is available, click to refresh”.

Clicking it calls skipWaiting() and reloads, so the user chooses when to switch. More controllable than a silent auto-replace, and more timely than “always wait for the next navigation”.

A blog can skip this; an app-like site should do it.

Scope and interception order

Two basic mechanisms deserve their own section, because they decide “who the SW governs, and whose place in line it takes”.

Scope: the SW’s jurisdiction is decided by the path of the sw.js file. At the site root (this site: /sw.js), the default scope is / and every page on the site is under its control. Put it in /sub/sw.js and it can only control pages under /sub/**.

“I changed sw.js but some pages behave the same” — check the scope first. Chances are those pages were never under its jurisdiction.

The server’s Service-Worker-Allowed response header can widen the default scope, but most sites do fine with the root path.

Interception order: after the SW takes over, every request the page makes passes through the fetch handler first, not the HTTP cache.

The SW is the request’s first stop; “cache or network” is decided entirely by policy code. Only a fetch() called inside the handler goes on to use the HTTP cache.

This order explains two things: why the SW can implement policies HTTP caching cannot (SWR’s “serve cache, update in the background” cannot be expressed at the pure HTTP layer). And why SW bugs are so destructive: it intercepts at the very front, and an exception in the handler fails the whole request.

So the fetch handler always needs a fallback catch — better to “fall back to a plain fetch” than let a request die halfway.

Registration timing

Registration goes after the window load event:

1
window.addEventListener('load', () => navigator.serviceWorker.register('/sw.js'));

The reason: the SW and its subsequent caching work should not compete for bandwidth with above-the-fold critical resources.

Register after the first screen has loaded, and installation and precaching happen in idle time.

This is standard browser best practice — nothing creative about it, but worth hard-coding in the template so it does not get “optimized” away.

Everyday tools for debugging the SW

SW pitfalls almost all happen when “the browser is following policy and you think it is a bug”. Three tools in DevTools map to three kinds of investigation:

• Update on reload (Application panel): force an SW update on every refresh and skip the wait. Keep it on during development; edit sw.js, refresh, done, no manual skipWaiting() verification;
• Bypass for network: temporarily route every request around the SW straight to the network. Whether “this page’s data is wrong” is the cache’s fault or the code’s becomes obvious with one switch;
• Cache Storage, bucket by bucket: this site has four buckets; expand each one in the panel to see entry URLs and response headers — the Date header TTL decisions rely on, whether hash-named artifacts updated — verified by eye.

Firefox’s equivalent lives in about:debugging, and one console-level tool is worth knowing: navigator.storage.estimate() reports the origin’s total quota and used space.

Cache Storage counts against the storage quota; after running for a while, use it to confirm “whether the static bucket’s growth matches expectations” (content grew, cache grew: normal; content flat and cache exploding: some bucket is leaking).

Cross-origin and opaque responses

The cdn bucket (jsDelivr / unpkg) runs into a hidden response-type problem: requests from a <script> tag default to no-cors mode, and the response is opaque.

The Cache API can still store it, but the content and status code are unreadable (status is 0).

The consequence surfaces in the TTL policy: you want to read the Date header to judge freshness, and what you read is empty.

The workaround is adding the crossorigin attribute when including CDN assets, putting the request in CORS mode so you get a “normal” response.

Public services like jsDelivr and unpkg send CORS headers anyway, so the attribute has no side effects.

Caching opaque responses is not an error (some scenarios leave no alternative), but know the price: you can store it but not read it, and any policy that depends on response content silently fails.

Second half: the Pagefind index

“Pagefind index” keeps appearing in the first half’s bucket table; here is the full account of what it is, why it is built, and one pitfall that nearly made the exclusion switch a no-op.

Why build-time indexing

Full-text search on a static site is usually an unpleasant three-way fork: plug in an external service (a new dependency and a bill), roll your own inverted index (a complexity explosion), or build the index at build time.

Finish the search problem entirely in CI, and at runtime only static files remain.

This site took the third road, Pagefind.

A two-phase model: at build time, pagefind scans the HTML in dist/, chunks and tokenizes each page, builds the inverted index, and emits a self-contained index directory (this site at /pagefind/), sharded aggressively — cut into many small fragments by page chunk, named by content hash. At runtime, the search component loads a bootstrap script of a few KB, and as the user types it pulls on demand only the small shards that contain hits.

The cost model becomes “one search = a few KB transferred”, instead of downloading the whole site’s content table.

Chinese tokenization depends on the build-time chunk granularity; measured against real Chinese content, recall is adequate.

Client-side full indexes (lunr, flexsearch and friends) pack the site content into one JSON: zero search latency, but the first payload grows linearly with content. At a few hundred articles the user downloads several MB of index to search once, and the static site’s “lightness” is gone.

External search services (Algolia and the like) are capable, but bring keys, quotas, and data sync — three burdens a personal site’s search volume cannot amortize.

The build-time sharded index is the middle state between the two: total index size also grows with content, but the user downloads only the shards that hit. The size problem is absorbed by “on demand”.

For a purely static site the benefit is zero runtime: no API key, no quota, no latency spikes; the index follows the build, and deploying is the whole job.

Search engines do not index this directory (Disallow /pagefind/ is already in robots); it exists purely for on-site search.

Scoping the index

By default Pagefind indexes the whole page, including the chrome repeated on every page — navigation, sidebar, footer.

That lets search results be polluted by “the sidebar’s category list”: search “algorithm” and you hit the sidebar links of 500 pages.

The fix is putting data-pagefind-body on <body>, telling Pagefind to collect content with that as the only root. The chrome is excluded naturally, and only the article body reaches the index.

Beyond body/ignore, the same family of data attributes offers a few more ways to raise the signal-to-noise ratio (attribute names evolve between versions; the official docs are authoritative): metadata. Declare the title, publish date, and author on elements, and results can carry “when this was published” instead of body snippets alone. Filters. Declare categories and tags as filterable fields, and the frontend can narrow to “only the Algorithms category”. Weighting. A hit in the title area is worth more than one in the body; with the title container weighted, articles that “have the query in the title” outrank ones that “mention it in passing”.

Together these three pull search from “full-text grep” to the doorstep of “structured retrieval”, at the cost of a few attributes in the template.

The marker is double-edged, and the pitfall is buried right there.

Pitfall: the ignore marker on html is short-circuited

One class of posts on this site must stay out of search (excludeSearch: true in frontmatter — pure log-style notes, say).

The natural implementation: the layout layer decides, and excluded pages get data-pagefind-ignore on <html>.

Per the Pagefind docs, that marker makes the crawl skip the whole page.

The result was that it did not work: excluded articles stayed findable.

The diagnosis is worth walking through in full, because it is the textbook case of “two mechanisms each correct alone, broken in combination”:

1. data-pagefind-ignore on <html>: per the docs, applies to the whole page. Correct on its own;
2. data-pagefind-body on <body>: per the docs, “use this element as the index root”. Also correct on its own;
3. The problem with the combination: once data-pagefind-body exists, Pagefind’s crawl root becomes the body, and the indexing logic no longer walks up to the html level. The ignore marker on html never gets read.

That line sits in an inconspicuous corner of the docs, and since the two markers belong to “exclusion semantics” and “scoping semantics”, intuition says they should stack; in practice the latter short-circuits the former.

The fix: exclusion must be implemented at the body level. Excluded pages also do not emit data-pagefind-body (the attribute is not output at all), and only then does page-level ignoring take effect.

This site’s layout layer merges three conditions (frontmatter’s excludeSearch/exclude, the layout’s noindex) into one boolean that controls whether the body attribute exists.

Index scope and exclusion scope are now decided by the same logic, and one-sided failure is no longer possible.

Language binding on a bilingual site

This site is zh/en bilingual, and pages in both languages land in one index directory; the language boundary is decided by the <html lang> Pagefind reads when the instance is first created, and the instance is pinned to that language afterwards.

A SPA-style trap hides here: if the search page is built as a component and reuses the old instance after a language switch, it keeps searching under the previous language’s binding and the page scrambles.

This site’s solution is to split instance names by language (one set each for zh/en) and swap instances when the language changes — rebuild rather than reuse.

The search page’s own construction is worth a note too: the UI uses the native components of @pagefind/component-ui (input, excerpts, result list), and the core engine is dynamically imported at runtime. The package is loaded by URL from /pagefind/, not bundled into the site.

Index and engine both download only at the moment the user actually opens the search page, so ordinary article browsing costs zero traffic.

For interaction this site offers two result modes: when strict matching returns nothing, it degrades automatically to loose mode for another round — splitting the query apart and relaxing per-word matching, preferring “tangential” results over a blank wall.

In search, “zero results” is the most damaging output: the user cannot tell whether the site lacks it or they typed it wrong.

Loose mode is the site saying “did you mean these” on the user’s behalf.

Result links consistently open in a new tab, so a user mid-search does not lose the search page itself.

Presentation has one more detail worth cultivating: excerpts and highlighting.

Every Pagefind result carries a piece of hit context (the excerpt), with matched terms marked by highlight tags.

Excerpt quality depends on the chunk structure at index time: the closer a chunk is to “one natural paragraph”, the more complete the excerpt; chop the body into fragment-sized tags and the excerpt becomes broken sentences.

The frontend just restyles the excerpt; no need to implement highlighting yourself.

Exclusion granularity has a middle setting too: data-pagefind-ignore is not limited to the page level on html/body; it can sit on a single element inside the page (the sidebar’s “latest posts” widget, say).

When the page should be indexed but one block is noise, do a local exclusion instead of choosing between “whole page in / whole page out”.

Putting search state in the URL

The search page hides an easily overlooked engineering decision: where the query lives.

The laziest implementation keeps it in component state — search on input, leave no trace.

This site chose the more old-fashioned route: the search box is a real form, GET-submitted to /search/?q=keyword, so the query becomes part of the URL.

That buys three things for free: shareable (send someone the results URL and they open the same page of results), back-button-friendly (a keyword change is one step in history, so Back works), and refresh-proof (after F5 the query is still there and the results re-search as they were).

For “looking something up”, these three are worth far more than the implementation cost.

The URL is the browser’s native state management; don’t reinvent it.

Search-as-you-type (results without pressing Enter) needs one companion detail: debouncing.

Typing “javascript” is nine keystrokes; without a debounce that is nine full search requests. A single Pagefind search is fast, but there is no reason to pay nine times.

A debounce of about 300ms merges “typing in bursts” into “search once stopped” — instant feel, one cost.

URL sync and debounce as a combo: debounce the input search, and update ?q= only on Enter (or once results settle), each side keeping its own rhythm.

The index size ledger

The index’s “total size” and “per-search cost” belong in separate columns.

Total size grows linearly with the term diversity of the site’s body text, and the build output shows the shard count rising with it. But at runtime a single search pulls only “the bootstrap script plus the shards that hit”, so the per-search cost is roughly constant.

From 200 articles to 600, the per-search download barely moves. That is the core dividend of sharding.

A periodic checkup is easy: after building, look at the total size and shard count of /pagefind/ (healthy is “small shards, many files”), then search a few representative terms and watch the Network panel for how many shards and how many bytes one search pulls.

Put the two numbers side by side and the index’s health is obvious.

Index freshness and caching

Here we rejoin the first half.

The index follows the build: every deployment generates a fresh set of shards with content hashes in the filenames, content changes mean the hash changes, and the browser’s cached old shards go stale naturally.

“The article updated but search results are old” cannot happen.

The only fixed URL in the whole index is that few-KB bootstrap script; it holds no data itself, so reusing it across deployments is harmless. To be careful, put it under the SW’s network-first policy.

The shards themselves sit in the first half’s static bucket under cache-first: a hit is correct, never revalidated against origin.

That is exactly why “content-hash-named assets” earn the best caching treatment, and the search index is that treatment’s best customer.

Lessons and a checklist

The lesson of this pitfall generalizes into three:

1. “Scoped root” and “ignore marker” are mutually exclusive semantics, not stacking semantics. Whenever a tool’s docs say “use X as the root”, follow up with one question: once X is the root, do rules on X’s ancestors still apply?
2. One boolean needs one owner. If index scope, search exclusion, and noindex are three independent judgments, they will drift apart on one side sooner or later. Converge on one variable, and a single edit is globally consistent. This site later hung “pages that stay out of the sitemap” on the same judgment;
3. Exclusion features need verification cases. A broken exclusion is a silent failure (it “works” as long as you can find the article — who goes out of their way to search for one that should not appear?). At launch, manually search for one excluded article’s title; it takes a minute.

Summary

Seen together: the caching line classifies by staleness behavior, picks strategies by tolerance, and draws red lines by side effects; the search line turns query cost into a one-time build-time expense, then pays only for the shards that hit.

The two mesh at “content-hash naming”. It is at once cache-first’s correctness guarantee and the index shards’ invalidation mechanism.

The full chain after the fix: article frontmatter controls search in/out, the layout layer judges at a single point, the Pagefind index holds only the article body, generated at build time, cached by the SW, distributed from the edge, with median search responses in the tens of milliseconds.

No runtime service anywhere in the chain; for a personal site the maintenance cost is roughly zero.

Thanks for reading! Follow me if you'd like~

Static Site Caching and Search

Sat Sep 19 2026

3824 words · 20 minutes

Frontend EngineeringFrontend

---

© MuxiaoWF | CC BY-NC-SA 4.0

Share

Share this article

X (Twitter)TelegramRedditFacebookEmailQQ

Close

Close