Skip to main content

Vision & OCR Pipeline

Wunderland includes a progressive vision pipeline for extracting text and understanding images. The pipeline uses a 3-tier architecture that automatically selects the best available provider.

Quick Start​

CLI​

# Extract text from an image
wunderland vision ocr ./document.png

# Describe image content
wunderland vision describe ./photo.jpg

# Generate a CLIP embedding vector
wunderland vision embed ./image.png

Programmatic​

import { createVisionPipeline } from '@framers/agentos';

const vision = await createVisionPipeline({ strategy: 'progressive' });
const result = await vision.process(imageBuffer);

console.log(result.text);           // Extracted text
console.log(result.confidence);     // 0.0–1.0 confidence score
console.log(result.contentType);    // 'printed' | 'handwritten' | 'document-layout' | 'photograph'

Three-Tier Architecture​

TierProvidersInstallBest For
Tier 0 — Local OCRPaddleOCR, Tesseract.jsnpm install ppu-paddle-ocrPrinted text, screenshots
Tier 1 — Enhanced LocalTrOCR, Florence-2, CLIPIncluded via @huggingface/transformersHandwriting, document layout, embeddings
Tier 2 — Cloud VisionGoogle Cloud Vision, OpenAI, AnthropicSet OPENAI_API_KEY etc.Complex images, highest accuracy

Processing Strategies​

StrategyDescriptionCost
progressiveStart local, escalate if confidence is low (default)$
local-onlyNever call cloud APIsFree
cloud-onlySkip local, send directly to cloud$$$
parallelRun all tiers simultaneously, merge best results$$$

Content Detection​

The pipeline automatically classifies images into four content types and routes processing to the most appropriate tier:

  • printed — Machine-printed text (PaddleOCR excels)
  • handwritten — Cursive or informal text (TrOCR excels)
  • document-layout — Structured documents with tables and figures (Florence-2 excels)
  • photograph — Natural images without text focus (Cloud Vision excels)

CLIP Embeddings​

Generate 512-dimensional vectors for semantic image search:

const { embedding } = await vision.embed(imageBuffer);
// Use with any vector store for image-to-image or text-to-image search

Installation​

# Best local OCR (optional but recommended)
npm install ppu-paddle-ocr

# Fallback OCR with 100+ languages (optional)
npm install tesseract.js

# Tier 1 models download automatically on first use
# Tier 2 requires cloud API keys (OPENAI_API_KEY, GOOGLE_CLOUD_VISION_KEY, etc.)

Agent Configuration​

Enable vision in your agent config:

{
  "ragMemory": {
    "multimodal": true,
    "vision": {
      "strategy": "progressive",
      "indexImages": true
    }
  }
}

Further Reading​