A model comparison is useful only when both models face the same task. I built an RFP-answering harness that retrieves evidence once, gives the same passages and instructions to Gemini and Claude, and keeps the inputs, outputs, citations and checks available for inspection.
Build the retrieval path
The example corpus is public material from Indic Games and my engineering portfolio. Reingestion opens a fresh Cloudflare Browser Run session, executes the pages’ JavaScript, and collects the rendered content. It also opens the portfolio’s About, project and Toolkit panels so their text is included. If live rendering fails, ingestion fails; it does not silently fall back to an earlier scrape.
For the recorded run, the browser captured 18 views: four Indic Games pages and fourteen portfolio views. The pipeline normalized the text and split each rendered page into passages of up to 1,200 characters with 180 characters of overlap. It embedded those passages with Gemini’s gemini-embedding-001 model at 768 dimensions and indexed them in a dedicated Pinecone namespace. That capture produced 26 chunks.
For each question, the harness embeds the query, retrieves up to five passages with a minimum similarity score of 0.35, and records the passage IDs, text, source URLs, content hashes and similarity scores. A new source generation becomes active only after its chunks are indexed. Retrieval filters against the active content hashes, so an older generation does not leak into a new run.
Hold the comparison steady
The critical control is shared retrieval. The harness retrieves once per question, then sends that same evidence set, system instruction and question to both providers. Both calls run concurrently. Model-specific REST adapters normalize the request settings, including temperature 0.1 and a 4,096-token output limit.
Answers must follow a small output contract: an answer, a source label, and a JSON array containing the exact IDs of cited passages. If retrieved evidence does not support a response, the model must return a canonical abstention and no evidence citations. The instructions also tell the models to treat retrieved text as untrusted data rather than as instructions.
This setup controls the evidence and task presented to the two models. It does not eliminate every source of variation: provider APIs and model implementations differ, and a single response can vary across repeated calls. The experiment is an inspectable comparison, not a claim that every model behavior has been controlled.
Make the questionnaire exercise answering and restraint
The saved sample contains 40 questions: 28 supported fact requests and 12 deliberately unsupported requests. Supported examples ask about public project descriptions and technologies. Unsupported examples probe for facts the sources do not establish, such as contractual uptime, audited revenue, private credentials and healthcare certifications.
Several questions ask about the same facts in different ways. That gives the sample useful regression coverage, but it does not make 40 independent benchmark tasks. The expected facts and abstention labels are used by the checker and are not included in the model’s prompt.
Check the contract, then inspect the answer
Automated checks look at output structure, whether citation IDs belong to the supplied evidence, required phrase coverage, and whether labeled unsupported requests receive the expected abstention. The interface resolves each model-emitted citation ID against that model’s saved retrieval set and displays the full passage supplied on that call.
A valid citation ID only proves that the model selected a passage it received; it does not prove that the passage supports every sentence. Phrase coverage is also a proxy, not a semantic judge. The saved raw outputs, exact evidence, requests and failed checks are therefore part of the result, so a person can review what the automatic checks cannot establish.
One recorded run
On 13 October 2026, the harness completed 80 provider calls across the 40-question sample with no provider errors:
| Model | Checks passed | Unsupported requests declined | Mean inference time |
|---|---|---|---|
| Gemini 2.5 Flash | 37 / 40 | 12 / 12 | 1.39 s |
| Claude Sonnet 4.5 | 39 / 40 | 12 / 12 | 2.07 s |
Four responses were flagged. Both models answered a broad question about documented work with cited evidence, but missed phrases required by its rubric. The question may mix system capabilities with an individual’s contributions, so the rubric needs human review. Gemini’s other two flagged answers were factually correct but used malformed response markup; one also returned citation objects instead of the required array of ID strings. The original outputs and failed checks remain in the record.
The latency figures cover model inference only. They exclude browser rendering, embedding and retrieval. This is one run with repeated facts in the question set, not a statistical model ranking or an accuracy score.
What I wanted the harness to make visible
When an RFP answer is wrong or incomplete, the useful debugging question is not only “which model?” It is also: what did retrieval return, what did the model actually receive, which passage ID did it cite, did it follow the response contract, and what did the checker miss? Keeping those artifacts together makes it possible to trace a result from rendered source to model output and reviewer judgment.
A stronger evaluation would use a reviewed rubric with unambiguous question intent, more independent tasks, and repeated trials. For now, the harness is a small engineering demonstration of shared-context RAG evaluation, live ingestion and inspectable failure analysis.