Most teams building document pipelines make the same mistake: they take a PDF, convert every page to a high-res image, and dump the whole thing into a prompt. It works in a demo. It falls apart the moment someone uploads a 100-page prospectus and the API bill shows up.

I've been building AI-integrated tooling long enough to know that "it works on one PDF" and "it works in production" are two different problems. Here's the architecture that actually holds up: Gemini's Files API, a locked-down JSON schema, DPI tuning, and bounding-box grounding for anything that needs to survive an audit.

Why Your OCR Pipeline Is Losing Data

Tesseract, PDFMiner, and most standard text extractors were never built to understand a page — they were built to read characters off it. That distinction matters more than it sounds like.

Three failure modes show up constantly in enterprise documents:

• Multi-column layouts get scrambled. Text from the left and right columns merges into one string, and the reading order stops matching the visual order.
• Nested tables lose their structure. Merged headers, irregular cell spans, and footnotes disconnect from the rows they belong to.
• Charts and diagrams are invisible. A quarterly revenue chart might carry the single most important number in the report, and a text extractor skips it entirely.

Gemini's multimodal models — 1.5 Pro and Flash, with context windows running 1M to 2M tokens — don't have this problem, because they're not reading text off a page. They're looking at it.

The Architecture: Files API + Structured Schema

The first mistake to avoid is base64-encoding your PDF straight into an HTTP request. That works for a two-page invoice. For anything over 5-10 pages, use the Files API instead — it uploads and caches the document asset so you're not re-transmitting the whole file on every call.

The second piece is a response schema strict enough that the model can't wander. Here's what that looks like for a financial report extraction job:

import { GoogleGenAI, Type, Schema } from '@google/genai';
import fs from 'fs';

const ai = new GoogleGenAI({ apiKey: process.env.GEMINI_API_KEY });

export async function processFinancialReport(pdfPath: string) {
  // 1. Upload PDF file to Gemini File API
  const fileUpload = await ai.files.upload({
    file: fs.createReadStream(pdfPath),
    mimeType: 'application/pdf',
  });

  // 2. Define strict extraction schema for financial tables
  const extractionSchema: Schema = {
    type: Type.OBJECT,
    properties: {
      reportTitle: { type: Type.STRING },
      fiscalQuarter: { type: Type.STRING },
      financialTables: {
        type: Type.ARRAY,
        items: {
          type: Type.OBJECT,
          properties: {
            tableName: { type: Type.STRING },
            headers: { type: Type.ARRAY, items: { type: Type.STRING } },
            rows: {
              type: Type.ARRAY,
              items: {
                type: Type.OBJECT,
                properties: {
                  metric: { type: Type.STRING },
                  value: { type: Type.STRING },
                  unit: { type: Type.STRING },
                  notes: { type: Type.STRING },
                },
                required: ['metric', 'value'],
              },
            },
          },
          required: ['tableName', 'headers', 'rows'],
        },
      },
      keyFindings: { type: Type.ARRAY, items: { type: Type.STRING } },
    },
    required: ['reportTitle', 'financialTables', 'keyFindings'],
  };

  // 3. Extract with structured schema and native vision
  const response = await ai.models.generateContent({
    model: 'gemini-1.5-flash',
    contents: [
      fileUpload,
      {
        text: `Analyze the attached PDF report. Extract all tabular financial figures,
        balance sheet line items, and quarterly summaries according to the provided
        JSON schema. Ensure all footnote annotations are attached to their respective
        row notes.`,
      },
    ],
    config: {
      responseMimeType: 'application/json',
      responseSchema: extractionSchema,
      temperature: 0.1, // low temperature for deterministic extraction
    },
  });

  // 4. Clean up the uploaded file asset
  await ai.files.delete({ name: fileUpload.name });

  return JSON.parse(response.text!);
}

Notice the temperature — 0.1, not the default. Extraction isn't a creative task. You want the model choosing the most likely token every time, not exploring alternatives. And the required fields in the schema aren't decoration; they're what stops the model from silently dropping a table when it's unsure how to structure it.

Token Optimization: DPI Tuning and Smart Chunking

Here's the part most guides skip: a PDF page rendered at 300+ DPI produces an image tensor that costs a lot of tokens without buying you any extra OCR accuracy. High-resolution rendering feels safer. It isn't — it's just more expensive.

Two changes fix most of the cost problem:

Downsample to 150 DPI. This preserves full readability down to 6pt fonts — which covers essentially every footnote you'll encounter in a real financial table — while cutting token and transmission overhead by roughly half compared to 300 DPI.

Filter pages before you send them. Don't push a 100-page prospectus through the model in one shot. Run a lightweight pre-pass — a table-of-contents regex works fine — and send only the 5-10 page range that actually contains what you need.

Neither of these requires touching the model. They're pipeline decisions, and they're the difference between an extraction job that costs cents and one that costs dollars.

Making Extraction Audit-Ready with Bounding Boxes

Structured JSON output is useful. Structured JSON output you can actually verify is what makes it usable in a regulated context.

Ask Gemini to return normalized 2D bounding boxes ([ymin, xmin, ymax, xmax]) alongside every extracted value, and you get a coordinate for exactly where on the page that number came from:

{
  "metric": "Q3 Operating Cash Flow",
  "value": "$14.2B",
  "source_location": {
    "page_number": 4,
    "box_2d": [420, 150, 445, 680]
  }
}

Overlay that box on the rendered PDF in your frontend, and a human auditor can confirm the number in seconds instead of re-reading the whole page. If you're building anything that touches finance, legal, or compliance, I'd treat this as non-negotiable rather than a nice-to-have — a schema-valid JSON blob with no way to trace a number back to its source is a liability, not a feature.

How This Actually Stacks Up on Cost

Traditional OCR is nearly free and nearly useless the moment a document gets complicated. Specialized Document AI APIs solve the accuracy problem but cost 5-10x more than the Gemini route for output that's often no better structured. For most teams, the native multimodal approach sits in the actual sweet spot — closer to Document AI on accuracy, closer to traditional OCR on cost.

The one place I wouldn't cut corners: if the numbers you're extracting feed directly into financial reporting or anything with legal exposure, pay for the bounding-box verification step even if it adds a little latency. Cheap and fast doesn't matter much if nobody trusts the output.