Natural Language Search Test

Problem Statement

Course PDFs currently can’t be searched with natural language without manually building an embeddings index and standing up a whole new dedicated app instance per document — that’s how the existing pdf-search-app deployments on the VPS work today (one Docker container per document, each with its own hand-built index). There’s also no way to know, for documents of this size and density, whether retrieval-augmented generation (RAG) actually beats simply sending the whole document to the model on every query (“Stuffing”) — nobody has compared them side by side.

Solution

A single-active-document search tool. An Admin uploads a PDF, which becomes the one “Active Document” (uploading again fully replaces it — there is never more than one). Anyone can then ask natural-language questions about the Active Document without logging in, and get an answer that cites the page/section it came from. Behind the scenes, the Admin can choose which Retrieval Mode (RAG or Stuffing) serves those public answers, and can use an admin-only Compare View to run any query through both modes side by side to judge quality — most usefully right after replacing the Active Document. All processing lives behind a VPS-hosted HTTP API; the existing cPanel install hosts only a thin, logic-free frontend that talks to that API.

User Stories

1. As an Admin, I want to upload a new PDF, so that I can make a new course document searchable.
2. As an Admin, I want uploading a new PDF to fully replace whatever was previously the Active Document, so that there’s never ambiguity about which document is being searched.
3. As an Admin, I want upload/replace to complete synchronously and tell me success or failure before I leave the page, so that I don’t have to guess whether processing finished.
4. As an Admin, I want to authenticate before I can replace the Active Document or change settings, so that random visitors can’t overwrite the course material or its configuration.
5. As an Admin, I want upload to reject files that aren’t valid PDFs or that fail text extraction, so that the Active Document is never left in a broken state.
6. As an Admin, I want to choose which Retrieval Mode (RAG or Stuffing) answers public queries, so that I can put whichever mode performs best into production.
7. As an Admin, I want to see which Retrieval Mode is currently live before I change it, so that I don’t accidentally serve a mode I didn’t intend to.
8. As an Admin, I want a Compare View where I submit one query and see both modes’ answers side by side, so that I can judge quality after replacing the Active Document, without affecting what public Searchers see.
9. As an Admin, I want to see recent search queries, which mode answered each one, and their estimated cost, so that I have visibility into usage and spend.
10. As a Searcher, I want to ask a natural-language question about the Active Document without logging in, so that I can quickly find information in the course material.
11. As a Searcher, I want the answer to cite the page/section of the source PDF it came from, so that I can verify it against the original document.
12. As a Searcher, I want a clear message if there’s no Active Document yet, so that I understand there’s nothing to search.
13. As a Searcher, I want a clear error if I’ve been rate-limited, so that I understand why my query didn’t go through.
14. As the system operator, I want public search requests capped by a per-IP rate limit (10/min), so that the tool can’t be abused into runaway API costs.
15. As the system operator, I want a hard daily spend cap ($5/day) on Claude + Voyage usage, so that a traffic spike or abuse can’t produce a surprise bill; once hit, search returns a clear error until the cap resets.
16. As the system operator, I want all PDF processing, retrieval logic, and LLM/embedding calls to live on the VPS behind HTTP endpoints, so that the cPanel frontend can remain a thin, logic-free presentation layer (ADR-0001).
17. As the system operator, I want replacing the Active Document to discard the old PDF file, extracted text, and embeddings outright, so that there’s no leftover data or version history to manage (ADR-0002).
18. As a developer evaluating the two modes, I want a small set of test questions and reference answers drawn from each of the two example PDFs, so that I can objectively compare RAG vs Stuffing before choosing a production default.
19. As a developer evaluating the two modes, I want Claude used as an LLM judge to score each mode’s answer against the reference answer, so that I get a repeatable, low-effort accuracy signal without hand-grading every case.
20. As a developer, I want the RAG mode to use Voyage’s voyage-law-2 embedding model, so that embeddings are tuned for this legal/costs-law content.
21. As a developer, I want both Retrieval Modes to generate answers with Claude Sonnet 5, so that the comparison isolates retrieval strategy as the only variable.
22. As the system operator, I want the search endpoint open to any visitor while upload/replace and all admin controls require authentication, so that the tool is easy for course participants to use while the content pipeline stays protected.
23. As an Admin, I want the Anthropic API key, Voyage API key, and admin credentials configured via environment variables, so that secrets are never hardcoded or committed to source control.
24. As the system operator, I want this new service isolated on its own port and directory on the VPS, so that it never interferes with the existing pdf-search-app containers or the video-rag stack, which stay running untouched.

Implementation Decisions

• New service (“the API”) built as a Node.js/Express app, containerized with Docker — consistent with the existing proven pattern already running on this VPS, but deployed to a new directory and a new port (8084), fully isolated from the existing pdf-search-app containers (8081-8083) and video-rag (80). No changes to those existing deployments.
• Exactly one Active Document exists at a time (per CONTEXT.md). Storage for its extracted text and, in RAG mode, its chunk+embedding index, is flat files on disk in the container’s data volume — no external database, given the single-document scale (contrast with video-rag’s Postgres/pgvector, which suits a different scale).
• Two Retrieval Modes are both implemented and always available: RAG (chunk + embed via voyage-law-2, retrieve top-k relevant chunks, page-number metadata preserved per chunk) and Stuffing (full extracted text, page-marked, sent on every query). Both use Claude Sonnet 5 for generation and are prompted to cite page/section in every answer.
• Retrieval Mode is a single global admin-set config value that determines which mode answers public search requests. A separate admin-only Compare endpoint/view runs a given query through both modes and returns both answers without changing the global config.
• Two access tiers: public/unauthenticated for search, and an authenticated admin tier (shared credential/token) gating replace, mode changes, the Compare View, and the usage log view.
• Rate limiting: 10 requests/minute per IP on the public search endpoint. A running daily spend counter (reset at UTC midnight) tracks estimated Claude + Voyage cost; once it exceeds $5/day, search requests are rejected with a clear error until reset.
• Usage logging: every search request is logged (timestamp, query text, mode used, estimated cost) to a simple on-disk log, readable via the admin usage view.
• Conceptual endpoints (not literal routes/files, to be decided during implementation): public search, public status (whether an Active Document exists and its title), admin replace (PDF upload), admin set-mode, admin compare, admin usage log.
• The cPanel-hosted frontend is static HTML/JS (optionally light PHP) with no server-side logic; it calls the VPS API directly from the browser. CORS on the API must allow the cPanel origin.
• A standalone evaluation harness (not part of the shipped product) runs the drafted test Q&As from both example PDFs through both Retrieval Modes and uses Claude as judge against reference answers, producing a report that informs which mode the Admin sets as the production default at launch.

Testing Decisions

• Single test seam: HTTP integration tests against the Express app (the API’s external HTTP boundary), with the Anthropic and Voyage network calls stubbed/mocked so tests are deterministic and don’t spend real API credits or money.
• Test only externally observable behavior: e.g., given an Active Document and a given mode, a search request returns an answer containing an expected citation shape; replace discards prior state and subsequent searches reflect the new document; a mode change changes which mode subsequent searches use; requests beyond the rate limit return 429; requests after the daily cost cap is exceeded return the cap-exceeded error; unauthenticated requests to admin endpoints are rejected.
• No isolated unit tests of chunking/embedding math — covered indirectly through the HTTP-level RAG search tests, per the single-seam preference.
• No prior art in this repo — it’s greenfield; this establishes the first test suite and its conventions.

Out of Scope

• Multi-document library / document selection UI — there is exactly one Active Document, per CONTEXT.md.
• Any changes to the existing pdf-search-app containers or the video-rag stack — both stay running untouched.
• User accounts or per-searcher identity — search is anonymous and open to anyone.
• Version history or rollback of a replaced document (ADR-0002).
• OCR or scanned-PDF support — only native-text PDFs are supported, matching the two example PDFs.
• Video transcript search — this spec is PDF-only.
• Production domain/SSL/cPanel deployment mechanics beyond a working integration — treated as a deployment detail, not part of this spec.

Further Notes

• The two example PDFs in pdfs-original/ (“INTEREST ON COSTS COURSE MATERIAL.pdf” and “FINANCIAL MIS-SELLING AND COSTS COURSE MATERIAL.pdf”) are dense UK costs-law course material, roughly 45-50 pages each, heavy with case citations and cross-references — the primary material for both the eval harness and manual QA.
• voyage-law-2 and Claude Sonnet 5 were independently already in use by the existing pdf-search-app deployment on the same VPS, which gives some confidence in the choice.
• VPS facts relevant to implementation: Debian 12, Docker + Compose v5 already installed, passwordless sudo for the debian user, ports 80 and 8081-8083 already occupied by other containers.
• docs/adr/0001-vps-hosts-all-logic.md and docs/adr/0002-discard-on-replace.md record the two most significant architectural trade-offs behind this spec and should be read before implementation.