Technical Case Study: Resilient Hybrid RAG Engine

• Project Repository: rag(portf)
• Stack: Next.js (App Router + Turbopack), TypeScript, PostgreSQL (pgvector), Drizzle ORM, pdfjs-dist (Legacy ESM build)

Phase-by-Phase Development Log & Breakthroughs

Phase 1: Ingestion API Setup & Compiler Path Hurdles

• The Goal: Build a unified server-side entry point (src/app/api/ingest/route.ts) that streams incoming multipart form binaries directly into an in-memory execution layer, without writing temporary files to disk.
• The Breakage: As soon as the route handler was configured using modular TypeScript conventions, compilation threw explicit module resolution errors: Module not found: Can't resolve './schema.js'.
• The Diagnostic Reason: Under modern bundler resolution models ("moduleResolution": "bundler" or "nodenext" variations in tsconfig.json), Next.js natively handles route mapping paths inside the framework compiler. Forcing rigid .js or .ts extensions in standard project references caused the module layout engine to look for compiled JavaScript files that did not yet exist in .next/server.
• The Code Modification: Removed all hardcoded extension suffixes across route.ts, ingest.ts, and the Drizzle database configuration files, letting the Next.js compiler resolve file nodes directly.

Phase 2: PDF Parsing & The Turbopack Isolated Thread Failure

• The Goal: Run low-overhead, server-side string layout extraction directly from raw file arrays using pdfjs-dist.
• The Breakage: The ingestion handler crashed during evaluation:

Plaintext

TypeError: Invalid `workerSrc` type. Expected string, got object.
at set workerSrc (webpack://pdf.js/src/display/worker_options.js:58:13)

• The Diagnostic Reason: PDF.js offloads binary decryption to a background Web Worker thread. By default, importing the worker bundle via traditional statements (import * as pdfWorker from "pdfjs-dist/build/pdf.worker.mjs") causes Turbopack to interpret the payload as an internal module chunk object instead of a literal, static filesystem string path. This triggers a strict type validation failure inside the PDF.js engine core.
• The Code Modification: Bypassed runtime object imports entirely. Used native Node.js path utilities to locate the raw .mjs worker component inside node_modules, then translated it into a literal file:// string using pathToFileURL:

TypeScript

import path from "path";
import { pathToFileURL } from "url";
import * as pdfjs from "pdfjs-dist/legacy/build/pdf.mjs";

const workerPath = path.join(
  process.cwd(),
  "node_modules",
  "pdfjs-dist",
  "legacy",
  "build",
  "pdf.worker.mjs"
);

pdfjs.GlobalWorkerOptions.workerSrc = pathToFileURL(workerPath).toString();

Phase 3: Database Infrastructure & Vector Storage Connection Caps