Natan Oliveira Martins
← Back to index
02 Generative AI · LLMOps In construction

Multimodal RAG with a governance layer

Answers questions from text, tables and charts in PDFs, and audits every prompt and answer for leaked data and hallucination.

  • Unstructured
  • Qdrant
  • LangGraph
  • RabbitMQ
  • vLLM
  • FastAPI

Code and roadmap on GitHub →

Interactive demo

You are

Drag to orbit · hover a shape to read the chunk · ● text · ■ table · ◆ figure · faded = restricted to HR

Answer

Pick a question above. You will see it fall into the vector space, pull its nearest chunks, and pass through the guards.

Retrieved chunks

    Audit log

      Things to try

      1. Ask about Carla Mendes’ salary as an Employee, then as HR. Access control happens at retrieval (the chunk is never even a candidate), not by asking the model to keep a secret.
      2. As HR, notice the CPFs come back as black bars: PII is redacted at the output, even for people allowed to see the document.
      3. Turn on the lying LLM and ask about expense reports. The grounding check catches the changed number because it appears in no source.
      4. Ask the same thing twice: the second answer comes from the cache and costs zero tokens. Click fast enough and you will hit the rate limit.

      The problem

      Corporate knowledge lives in PDFs full of tables and charts that plain text extraction destroys. And once an LLM can read those documents, something has to stop it from repeating a salary or inventing a number.

      How it works

      1. Documents are parsed into typed elements (paragraph, table, figure); tables keep their structure and figures get a generated description.
      2. Elements are embedded and stored in Qdrant with their source page, so every answer can cite.
      3. A LangGraph agent decides whether to search, read a table, or ask for clarification.
      4. A gateway in front of the model caches, rate-limits, and sends every exchange to an async audit queue: PII detection, groundedness against retrieved sources, toxicity.
      PDFs text · tables · charts Parser typed elements Qdrant vector index User question Gateway cache · limits LangGraph agent Audit PII · grounding
      Fig. 02 — How it works

      What is real, what is simulated

      Real

      • Retrieval is real: TF-IDF vectors with cosine similarity over 20 chunks, and the 3D layout is a PCA of those vectors computed in your browser.
      • The ACL filter, the cache, the token-bucket rate limit, the prompt-injection rules, PII redaction and the grounding check are the actual logic.
      • Millisecond timings in the trace are measured on your machine.

      Simulated in this demo

      • The documents belong to a fictional company; names, CPFs and contacts are invented.
      • There is no LLM: answers are extractive (sentences copied from the sources). The “lying LLM” switch alters a number on purpose so the grounding check has something to catch.
      • TF-IDF stands in for dense embeddings, and the audit queue is a list on the page, not RabbitMQ.

      Not built yet

      • PDF parsing of real tables and charts (Unstructured)
      • Qdrant with dense embeddings and payload ACL filters
      • LangGraph agent and a real LLM behind the gateway
      • PII and grounding models (instead of rules) consuming from RabbitMQ