MIS 752 · Lab 8 · RAG

Answer the Patient: what a reranker changes, measured

webstk3 · MIS 752 Lab 8 · 2026-10-13

The case

A nurse asks about metformin at an eGFR of 38. A bare model recites the general 2000 mg/day maximum, the right number for the wrong patient. This page measures what a bi-encoder alone and a bi-encoder plus cross-encoder reranker do with the same ten-page synthetic handbook, and what happens when the handbook has no answer.

What the reranker changed

promoted on 1 of 5, demoted on 0, unchanged on 4 · biggest move: nephrology_referral (rank 2 to 1) · answer found by stage 1 alone on 5/5, after the cross-encoder on 5/5

Pages surfaced per question

metformin_ckd gold p4 vector top-3: 4,4,4 reranked: 4,4,3 mech_valve_inr gold p6 vector top-3: 6,6,6 reranked: 6,6,6 st_johns_wort gold p7 vector top-3: 7,7,7 reranked: 7,7,6 nephrology_referral gold p5 vector top-3: 5,5,2 reranked: 5,1,1 metformin_titration gold p3 vector top-3: 3,4,4 reranked: 3,4,3 paediatric_amoxicillin gold p- vector top-3: 6,4,3 reranked: 6,4,3

Chunk-size sweep

Before and after, per question

question voice answerable gold_pages pool_rank vector_rank reranked_rank precision_vector precision_reranked rr_vector rr_reranked hit3_vector hit3_reranked gold_in_pool vector_ms rerank_ms
metformin_ckd clinician True 4 1.0 1.0 1.0 0.333333 0.333333 1.0 1.0 1 1 1 58.7 2113.7
mech_valve_inr clinician True 6 1.0 1.0 1.0 0.333333 0.333333 1.0 1.0 1 1 1 27.6 972.9
st_johns_wort patient True 7 1.0 1.0 1.0 0.666667 0.666667 1.0 1.0 1 1 1 30.2 960.7
nephrology_referral clinician True 5 2.0 2.0 1.0 0.333333 0.333333 0.5 1.0 1 1 1 25.2 1110.8
metformin_titration patient True 3 1.0 1.0 1.0 0.333333 0.333333 1.0 1.0 1 1 1 23.8 902.5
paediatric_amoxicillin clinician False - NaN NaN NaN 0.000000 0.000000 0.0 0.0 0 0 0 25.3 898.9

Takeaways

The reranker peaked at 350-character chunks, achieving MRR 1.00 compared to 0.90 from the vector-only stage—a +0.10 improvement on the optimal chunk size. However, across all five answerable questions, the reranker’s rank movements were minimal; questions already ranked first by the bi-encoder remained first after reranking, suggesting the initial retrieval was already precise on this corpus. The real bottleneck was the candidate pool size, not the reranker. At 150-char chunks (108 total), the pool of 20 covered only 18% of pages containing answers; at 1400 chars (11 chunks), coverage reached 100%. The reranker’s ceiling was directly constrained by what the pool could offer.

For the backend, I used numpy exact cosine search (FAISS was unavailable or fell back). At ~46 chunks, this numpy brute-force scan is numerically identical to IndexFlatIP—both compare every chunk to every query exactly. FAISS only earns its place at 10⁵+ vectors.

Outside healthcare, this retrieve-then-rerank architecture appears in customer support (FAQ retrieval + relevance ranking), legal document discovery, scientific literature search, and e-commerce product matching.

The handbook is invented teaching text for a fictional health system. It is not clinical guidance and nothing here should be acted on.