RAG for InterSystems IRIS documentation

Context and problem

At Healthy Reply we work heavily with InterSystems IRIS and its proprietary language ObjectScript. It is a niche language: few examples online, so generic LLMs were trained on a small amount of data. When I asked a model how to define a class, a method or a specific practice, I often got wrong or hallucinated answers.

The problem was not a lack of documentation: InterSystems publishes complete PDF/Markdown docs. The problem was that nobody kept it in the model’s context.

So, in my spare time at work (initiative more than an assigned task), I designed and built a complete RAG system:

given a question about IRIS/ObjectScript, retrieve the relevant documentation passages and answer in a grounded way, with citations pointing to the exact sections.

Architecture

The system has two integrated parts:

1. “Neurons” agent — handles user interaction and executes function calls against the RAG server. It is the interface colleagues actually use.
2. RAG server (this project) — receives queries via API, performs retrieval and generation. It exposes retrieval-only endpoints (for the agent) and an end-to-end /answer endpoint.

flowchart LR
  A["IRIS documentation<br/>(PDF / Markdown)"] --> B["Parsing & cleaning<br/>Markdown"]
  B --> C["Parent-child chunking"]
  C --> D["Embedding<br/>BAAI/bge-m3"]
  D --> E[("Qdrant<br/>vector DB")]
  C --> F[("BM25 index")]
  Q["User query<br/>(even in Italian)"] --> G["Query expansion<br/>LLM"]
  G --> H["Multi-query<br/>RAG-Fusion"]
  E --> I["Hybrid fusion<br/>Reciprocal Rank Fusion"]
  F --> I
  I --> J["Reranking<br/>cross-encoder"]
  J --> K["Ranked parent contexts"]
  K --> L["LCEL answer chain<br/>(LangChain)"]
  L --> M["Grounded answer<br/>+ validated citations"]

Key components

Ingestion. PDFs are converted to Markdown, cleaned and split with a parent-child strategy: small child chunks are indexed for precise retrieval, larger parent chunks are returned to the caller for richer context.

Hybrid retrieval. Dense retrieval with Qdrant and the BAAI/bge-m3 embedding model; sparse retrieval with BM25. Both rankings are merged with Reciprocal Rank Fusion (RRF), so the system does not depend on a single method.

Query expansion. An LLM turns the user’s question (often in Italian) into multiple technical English queries — variants that capture synonyms, class names and APIs. RAG-Fusion runs them all and merges the results.

Reranking. The ms-marco-MiniLM-L-6-v2 cross-encoder compares query-document pairs more accurately (but more expensively), so it is applied only to already-filtered candidates: maximise recall at retrieval time, precision at the end.

Grounded answer. The LCEL pipeline generates an answer with structured citations (deterministic source IDs) and post-generation validation: if a citation does not match a retrieved source, it is flagged.

Hardening. API protected by X-API-Key, context guardrails (characters/tokens), per-stage timings and optional LangSmith tracing.

What I did

I designed and implemented the entire RAG server from scratch: ingestion, indexing, retrieval engine, answer pipeline, evaluation and Docker. I integrated it with the Neurons agent over REST. The project was born from my own initiative: I identified the pain (LLMs that don’t know ObjectScript), proposed the solution and built the whole system.

Design choices (and trade-offs)

• Why BM25 + dense? BM25 is strong on keywords, class/method names and exact terms; dense retrieval captures semantics and paraphrases. RRF fusion reduces dependence on any single method.
• Why the reranker? The initial retriever maximises recall; the reranker improves precision on top-k with a more expensive but more accurate comparison. The chosen model is intentionally light because it runs on CPU.
• Embedded Qdrant with a single worker: simple to manage locally; for multi-worker production you would move to a standalone Qdrant server. Documented choice.
• Evaluation as a regression test: the synthetic dataset (708 LLM-generated queries with ground truth) catches regressions when I change chunking, embedding, query expansion or reranking. Metrics: Recall@K, MRR, Precision@K, NDCG@K.

Results

Baseline over 708 evaluation queries (synthetic, English):

File-level retrieval is very strong; parent-level retrieval is good for a technical RAG. The most useful operational metrics are recall_parent@5, recall_parent@10 and mrr_parent.

Notes

• The eval dataset is synthetic and English-only: treat it as a regression baseline, not a human gold set.
• The code is private (work-related project): happy to walk through it in an interview. GitHub profile: FilippoGalli001.
• Related paper: SEUPD@CLEF 2024 — same topic of query expansion and re-ranking, in a classic Information Retrieval setting.