Web scraping with Puppeteer: dynamic records and finite pagination
Extract delayed browser-rendered records into plain JSON using inline examples, stable selectors, deduplication and bounded pagination.
Use a browser when the data you are authorized to read is produced by client JavaScript or requires page interactions. If the server already returns the required HTML/JSON, a direct request and parser may be simpler. Puppeteer is not necessary for every extraction task.
Read paginated records step by step
Run npm install puppeteer in your project. The complete script below serves three small pages on a local server, each with delayed records and a Next link. The snippets explain the pagination loop using that server’s origin; no third-party site or separate example files are needed.
1. Prepare the browser and record stores
Keep visited URLs separate from extracted records. The Set detects repeated pages, while the Map collects records by ID. Start with the first local page.
browser = await puppeteer.launch({
headless: true,
});
const page = await browser.newPage();
const visited = new Set();
const records = new Map();
let next = origin + "/";
2. Reject unexpected pages and loops
The complete script caps the loop at five pages. At the start of each iteration, check the next URL’s origin and whether it has already been visited before navigating.
if (
new URL(next).origin !== origin ||
visited.has(next)
) {
throw new Error(
"Unexpected origin or pagination loop",
);
}
visited.add(next);
3. Load the page and wait for its data
Check the document response, then wait for the demo’s explicit ready signal. The HTML can arrive before its delayed records, so document loading alone is not enough.
const response = await page.goto(next, {
waitUntil: "domcontentloaded",
timeout: 5000,
});
if (!response || !response.ok()) {
throw new Error("Document request failed");
}
await page.waitForFunction(
() => document.body.dataset.ready === "true",
{ timeout: 5000 },
);
4. Extract plain records and validate them
Run the DOM query in the browser and return plain data to Node.js. Reject missing IDs or names before adding the batch to the collection.
const batch = await page.$$eval(
"#records li",
(elements) =>
elements.map((item) => ({
id: item.dataset.id,
name: item.textContent,
})),
);
for (const record of batch) {
if (!record.id || !record.name) {
throw new Error("Invalid record");
}
records.set(record.id, record);
}
5. Find the next page and check the end condition
The demo hides its Next link on the last page. Return null there so the loop stops naturally; otherwise, follow the link on the next iteration.
next = await page.$eval(
"#next",
(link) => link.hidden ? null : link.href,
);
After the loop, a remaining next URL means the page limit stopped the work before completion. Fail rather than silently exporting a partial result:
if (next) {
throw new Error(
"Pagination limit reached before the end",
);
}
// Six records: 1-a through 3-b.
console.log([...records.values()]);
Complete script
Here are all the pieces together, including setup and cleanup. Save this as scrape.mjs and run node scrape.mjs.
import { createServer } from "node:http";
import puppeteer from "puppeteer";
const server = createServer((request, response) => {
const url = new URL(request.url, "http://localhost");
const number = Number(
url.searchParams.get("page") ?? 1,
);
if (
url.pathname !== "/" ||
!Number.isInteger(number) ||
number < 1 ||
number > 3
) {
response.writeHead(404);
response.end("Not found");
return;
}
const records = ["a", "b"].map((letter) => ({
id: `${number}-${letter}`,
name: `Book ${number}${letter.toUpperCase()}`,
}));
response.setHeader("Content-Type", "text/html");
response.end(`
<ul id="records"></ul>
<a
id="next"
href="?page=${number + 1}"
${number === 3 ? "hidden" : ""}
>
Next
</a>
<script>
setTimeout(() => {
const records = ${JSON.stringify(records)};
const list =
document.querySelector("#records");
for (const record of records) {
const item =
document.createElement("li");
item.dataset.id = record.id;
item.textContent = record.name;
list.append(item);
}
document.body.dataset.ready = "true";
}, 50);
</script>
`);
});
await new Promise((resolve, reject) => {
server.once("error", reject);
server.listen(0, "127.0.0.1", resolve);
});
const origin = `http://127.0.0.1:${server.address().port}`;
let browser;
try {
browser = await puppeteer.launch({
headless: true,
});
const page = await browser.newPage();
const visited = new Set();
const records = new Map();
let next = origin + "/";
for (let count = 0; next && count < 5; count++) {
if (
new URL(next).origin !== origin ||
visited.has(next)
) {
throw new Error(
"Unexpected origin or pagination loop",
);
}
visited.add(next);
const response = await page.goto(next, {
waitUntil: "domcontentloaded",
timeout: 5000,
});
if (!response || !response.ok()) {
throw new Error("Document request failed");
}
await page.waitForFunction(
() => document.body.dataset.ready === "true",
{ timeout: 5000 },
);
const batch = await page.$$eval(
"#records li",
(elements) =>
elements.map((item) => ({
id: item.dataset.id,
name: item.textContent,
})),
);
for (const record of batch) {
if (!record.id || !record.name) {
throw new Error("Invalid record");
}
records.set(record.id, record);
}
next = await page.$eval(
"#next",
(link) => link.hidden ? null : link.href,
);
}
if (next) {
throw new Error(
"Pagination limit reached before the end",
);
}
// Six records: 1-a through 3-b.
console.log([...records.values()]);
} finally {
try {
if (browser) {
await browser.close();
}
} finally {
server.closeAllConnections();
await new Promise((resolve) => {
server.close(resolve);
});
}
}
The document arrives before the delayed records. The data-ready attribute is this example’s explicit application contract, not an attribute every website exposes. Replace the route/selectors and completion state with those of your authorized source.
Data readiness and response failures
For an API-backed application, register the response wait before the action that triggers it. Match its exact URL/method, check status, and then wait for the rendered result. A failed API must not become a silent partial export. See waiting and navigation failures.
If the application exposes both success and error states, wait for either and inspect which occurred. domcontentloaded is document readiness, not completed fetching/rendering.
Return plain records
$$eval executes in the page and returns serializable data to Node. Do not return DOM nodes expecting to use them as Node-side elements. Use stable application identifiers rather than fragile nth-child selectors. Validate schema and expected completeness before writing final output.
A Map deduplicates IDs, but does not prove a source is complete or that replacing duplicate IDs is always correct. Decide whether conflicting duplicates should instead be an error. See handle/context ownership.
Bound the work
Use explicit page/count/time limits, visited URLs and a real end condition. Infinite scrolling requires its own finite count/height/time strategy. Do not run an unlimited loop until an incidental selector throws.
For a transient authorized read, a finite backoff policy may be appropriate. Do not retry permanent blocks, invalid selectors or unsupported options. Use a bounded worker pool and a separate page/context owner per job; uncontrolled concurrency can increase memory pressure and harm the source.
Ownership, authorization and alternatives
Navigation/replacement destroys references. Reacquire live frames and fresh handles rather than waiting on stale objects. Only access data you are authorized to read; proxies do not confer permission. Honor privacy, site requirements and rate limits.
Read Cheerio versus Puppeteer for static HTML alternatives. This Puppeteer example explains one controlled workflow, not commercial-site compatibility, scraping legality or universal performance rankings.