Free lesson

Extract page text and metadata

Extract page text and metadata — A private PDF knowledge assistant. Install PDF.js deliberately; Save pages.mjs; Promote only reviewed pages. Practical assignment: Lab evidence.

Demo edition. The materials are undergoing editorial review.

Topics covered

  • Install PDF.js deliberately
  • Save pages.mjs
  • Promote only reviewed pages

Install PDF.js deliberately

In the lab directory initialize npm and install pdfjs-dist. Check package engine requirements against your Node version and keep the lockfile.

A PDF is a container for page layout, not a guarantee of readable sentences. A scan may have no text layer; a table may extract in the wrong order; an OCR layer may contain plausible spelling errors. This course therefore treats ingestion quality, source visibility and lifecycle as product requirements. Use only invented workshop documents until every page can be traced back to its original file.

// Terminal:
// npm init -y
// npm install pdfjs-dist
// node --version
// npm view pdfjs-dist engines

Save pages.mjs

Run this extractor with a synthetic policy.pdf. Preserve page numbers even when text is empty. Joining strings is a baseline; inspect reading order manually.

The extractor now accepts an optional file path: node pages.mjs policy.pdf, or the quarantined path printed by intake. Use a fresh output manifest for each document before combining them; the initial fixed docId must be replaced with that document’s stored ID to avoid collisions.

The output-path argument keeps one manifest per document revision, and write flag wx refuses accidental overwrite. If the target already exists, inspect and explicitly choose a new filename; do not delete an unrelated manifest. The finally block destroys the loading task even when page limits, parsing or writing fails. A rejected PDF therefore cannot be mistaken for a newly completed pages file.

Run processing in a separate local directory with synthetic files and explicit limits. Save the raw file under an application-generated identifier rather than a user-supplied path. Preserve a page manifest containing extraction status, revision and processing method. The retrieval project from the prerequisite course consumes approved page text only; it must not receive every uploaded byte merely because upload completed successfully.

import {readFile,writeFile} from 'node:fs/promises';
import {getDocument} from 'pdfjs-dist/legacy/build/pdf.mjs';
const data=new Uint8Array(await readFile(process.argv[2]??'policy.pdf'));
const task=getDocument({data,isEvalSupported:false});
try{
 const pdf=await task.promise;
 if(pdf.numPages>20)throw Error('Training page limit');
 const docId=process.argv[3]??'policy';const revision=Number(process.argv[4]??1);
 if(!/^[a-zA-Z0-9_-]+$/.test(docId)||!Number.isInteger(revision)||revision<1)throw Error('Document identity');
 const pages=[];
 for(let page=1;page<=pdf.numPages;page++){
  const p=await pdf.getPage(page);const content=await p.getTextContent();
  const text=content.items.map(x=>'str'in x?x.str:'').join(' ');
  pages.push({docId,revision,page,text,status:text.trim()?'needs_review':'needs_ocr'});
 }
 await writeFile(process.argv[5]??'pages.json',JSON.stringify(pages,null,2),{flag:'wx'});
}finally{await task.destroy();}
// Usage: node pages.mjs policy.pdf policy 1 policy-r1.pages.json

Promote only reviewed pages

Compare extracted cancellation text with the PDF. After a successful check mark that page ready in the manifest; retain needs_review for broken columns or tables.

Test a normal document, an image-only page and a file that should be rejected. Keep the original visible during review; a fluent answer cannot reveal that OCR changed a number. Access checks apply to both retrieved text and original downloads. Document deletion must also remove active derived chunks and cached answers, while backup retention follows an explicit separate policy. Record the exact behavior you verified.

Practical assignment

Lab evidence

Continue to learning

Course outline

  1. Inspect PDFs before ingestion
  2. Extract page text and metadata
  3. Use OCR without hiding uncertainty
  4. Handle uploads with bounded processing
  5. Ground answers and enforce source access
  6. Maintain, replace and remove documents