1,343 Pages Scraped, 1,343 Identical Bodies? The `article .entry-content` Cascade Grabbed a Sidebar Widget

There is a failure mode far sneakier than a crash: a process that reports one hundred percent success while every single result is wrong.

I was migrating content from a client site running WordPress into Sanity. The payload was a custom post type holding 1,343 entries. My first plan was the REST API, but that post type had been registered with 'show_in_rest' => false, and changing that on the WordPress side was not an option available to me. That left exactly one route: fetch the rendered HTML and parse it with jsdom.

The first run finished without drama. 1,343 pages fetched, 1,343 reported successful, zero errors. The terminal summary looked like a finished job.

The symptom: thirteen hundred pages, one body

QA was simple enough: open a handful of the imported entries in preview. The first one rendered an empty body plus a single stray link pointing at something completely unrelated to that entry. The second entry: the same stray link. The third: same again.

So I stopped clicking and started counting. Hash each body, then see how many distinct values come back:

import { createHash } from "node:crypto";
 
const buckets = new Map();
for (const item of results) {
  const key = createHash("sha1").update(item.html).digest("hex");
  buckets.set(key, (buckets.get(key) ?? 0) + 1);
}
 
console.log(buckets.size, "unique bodies across", results.length, "pages");
// 1 unique body across 1343 pages

One unique hash for 1,343 pages. Every entry held the exact same block of HTML, 1,405 characters long, identical down to the last byte. The scraper had never failed. It had simply been consistently grabbing the wrong element.

Two dead ends before I stopped guessing

First theory: some cache layer on the WordPress side was serving the same page over and over. Easy to test. I ran curl against several URLs one at a time and read the raw HTML. The real content was right there, correct, and different on every page. So fetching was not the problem.

Second theory: concurrency. Maybe responses were getting crossed between workers. I dropped concurrency to 1 and re-ran a slice. Identical results again. That is the point worth holding on to: if a bug survives fully serial execution, it is not a race condition. It is deterministic, and deterministic bugs live in your own code.

Only then did I do the boring thing I should have done first: dump one page's raw HTML to disk, open it in an editor, and find out which element my selector was actually picking.

The root cause: a cascade that could never fail

The extractor used a pattern that looks defensive, a list of selectors ordered from most specific to loosest:

const CONTENT_SELECTORS = [
  "article .entry-content",
  "article .post-content",
  "article",
];
 
function pickContent(doc) {
  for (const selector of CONTENT_SELECTORS) {
    const el = doc.querySelector(selector);
    if (el) return el.innerHTML;
  }
  return null;
}

The reasoning is sensible: if the first selector misses, fall back to something looser. The problem is that the WordPress theme on this site puts the main content in a top-level .entry-content div with no <article> wrapper at all. And the only <article> tags on the page belonged to a sidebar "popular posts" widget.

So the cascade behaved exactly as written. article .entry-content missed. article .post-content missed. article matched, and what it matched was the first widget card in the sidebar. That sidebar is structurally identical on every page of the site, which is precisely why every result came out identical. The stray link I saw during QA was widget markup, not entry content.

The loose fallback did not save me. The loose fallback is what hid the failure. Had the cascade stopped at the first selector and thrown, I would have known within ten seconds instead of after 1,343 documents landed in the CMS.

The fix

Drop the article prefix entirely, target the content container directly, then strip the theme chrome that rides along with it:

const CONTENT_SELECTORS = [
  ".entry-content",
  ".content-inner",
  ".jeg_post_content",
  "main .single-content",
];
 
const STRIP_SELECTORS = [
  ".jeg_share_button",
  ".jeg_share_float_container",
  ".jeg_sharelist",
  ".jeg_post_meta",
  ".jeg_meta_container",
  ".back-link",
];
 
function extractContent(doc) {
  let el = null;
  for (const selector of CONTENT_SELECTORS) {
    el = doc.querySelector(selector);
    if (el) break;
  }
 
  if (!el) throw new Error("no content container matched");
 
  for (const selector of STRIP_SELECTORS) {
    el.querySelectorAll(selector).forEach((node) => node.remove());
  }
 
  const html = el.innerHTML.trim();
  if (html.length < 200) throw new Error("body suspiciously short, likely wrong element");
  return html;
}

Being honest about it though, neither guard would have caught the bug I just described. That sidebar widget was 1,405 characters long and would clear a minimum length threshold without blinking. Only one check actually catches it, and it lives at the batch level rather than the page level.

const hashes = new Set(results.map((r) => createHash("sha1").update(r.html).digest("hex")));
if (hashes.size < results.length * 0.9) {
  throw new Error(`only ${hashes.size} unique bodies across ${results.length} pages, wrong element`);
}

So there are three layers now. First, no fallback is allowed to settle for whatever it finds: if nothing matches, it throws, and that page lands on the failure list instead of quietly filling with garbage. Second, a minimum length threshold for near empty pages. Third, and this is the one that matters, a uniqueness assertion once every page is done. A wrong page can look perfect on its own; what exposes it is seeing all of them at once.

Re-scrape: 1,343 of 1,343 clean. One timeout showed up midway and recovered once I lowered CONCURRENCY to 2. This time QA on the rendered output showed the real content instead of a sidebar.

What I took away

• A selector cascade can succeed on every page and be wrong on every page. Matched is not a synonym for correct.
• An overly loose fallback is a liability. Bare article will match anything that happens to use the tag, including widgets that are not content at all.
• If scraped results look uniform, suspect your extractor before you suspect the source. Hash each page body and count the distinct values. One hash across thousands of pages is a bug, not a coincidence.
• A bug that survives serial execution is not a concurrency problem. Stop tuning the throttle and go read the raw HTML.
• QA belongs on the rendered output, not on the runner's success report. A green log only tells you the script did not crash, not that the data is right.