Multimodal RAG: Chat With Your Manuals and Find Comparable Past Project Photos for Instant Quotes

The problem

Companies sit on a pile of value they can't retrieve: equipment makers and support teams have manuals, SOPs, spec sheets, and wiring diagrams where the real answer lives in a chart or figure — and trades and field-service businesses (roofing, HVAC, remodeling) have years of project photos showing jobs they've already quoted and completed. Plain "chat with PDF" tools silently discard the diagrams, tables, and images that contain the actual answer, so they return confident but ungrounded text. And there's no easy way to upload a photo of a new job and pull up comparable past work to quote from. Building a true multimodal pipeline — ingesting both text and images, embedding them, indexing them, and returning the right figure with its source page and a confidence score — is genuine engineering that's fragile and painful to assemble in tools like n8n.

Who it's for

Equipment and product companies and support teams that need document Q&A grounded in figures and source pages (internal staff or customer-facing), and trades / field-service businesses — roofing, HVAC, remodeling, solar, field installers — that want to upload a photo of a new job and instantly surface comparable completed jobs to build a quote from. Best fit for a business that has a real document or image library it already owns but can't search, and wants a bespoke tool it controls rather than a generic chat-with-PDF subscription.

How it works

1. 1

   Scope the index in plan mode: paste the multimodal embeddings API docs (Gemini Embeddings 2) into Claude Code, and have the agent design and stand up a Pinecone vector index that holds both text chunks and image embeddings in the same space — so a text question and an image query can retrieve from the same library.

2. 2

   Ingest everything with no pre-sorting: drop all media — the full PDFs, manuals, SOPs, and the entire project-photo library — into a single data folder. The agent walks the folder, extracts text and images, generates multimodal embeddings for each asset, and upserts them into Pinecone with metadata (source file, page number, asset type).

3. 3

   Build the front end on localhost with the front-end-design skill: spin up either a chat interface (type a natural-language question) or a photo-upload interface (drop in a new job photo), depending on the use case — or both. Route models through OpenRouter and use Sonnet to synthesize answers from the retrieved chunks.

4. 4

   Query and ground the answer: ask in plain language or upload a photo. The app returns inline images and diagrams, a confidence percentage on each match, and the source page each answer came from — so the user can verify it, not just trust it. For trades, it returns the closest comparable past jobs.

5. 5

   Improve retrieval and synthesize quotes: add per-asset metadata and short descriptions (what the diagram shows, the scope of the past job, materials used) to sharpen matches over time. For the trades use case, the app synthesizes a draft quote from the matched comparable jobs so an estimator starts from real past work instead of a blank page.

Tools

The result

You get a working app — chat, photo-upload visual search, or both — running over your own documents and images, where retrieval is genuinely multimodal: a question can surface the exact diagram that answers it, and a photo can surface the closest comparable past jobs. Each result comes back with the inline image or figure, a confidence percentage, and the source page, so answers are verifiable and grounded rather than hallucinated from text alone. Because ingestion takes the whole folder with no pre-sorting and adding metadata sharpens matches over time, the library improves as you feed it. For trades and field-service teams, the visual-similarity flow turns a years-deep photo archive into a quoting asset: upload a new job, pull up the comparable work you've already priced, and synthesize a draft quote from it. Honest framing: plain "chat with PDF" is saturated by cheap SaaS, so the value here is specifically the multimodal grounding (figures + source pages), the visual-similarity quoting, and the fact that you own a bespoke pipeline tuned to your data. This is a real difficulty-3 build — true text-and-image embeddings, a vector index, confidence scoring, and a custom front end — not a no-code template, which is exactly why it's worth having built rather than wired together by hand.