MIS 752 · Lab 8 · richardyoung
Answer the Patient: what a reranker changes, measured
A ten-page clinical handbook generated inside the notebook, chunked, embedded and indexed; twenty rough candidates narrowed to three by a cross-encoder; and the answer with its source page cited. Both models run locally on a free CPU. Synthetic document, no PHI, $0. Measured in Google Colab; the numbers are this run's and will vary.
The 30-second read
THE DOCUMENT - written and rendered inside this notebook
source : PDF (reportlab wrote it, pypdf read it back)
pages : 10 chunks: 46 median chunk 280 chars
extraction : 100.0% of the authored sentences survived the PDF round trip
chunker used : the instructor chunk_pages() (yours is not passing the contract yet)
THE TWO STAGES - both local, both $0
bi-encoder : sentence-transformers/all-MiniLM-L6-v2
cross-encoder : cross-encoder/ms-marco-MiniLM-L6-v2
index backend : faiss-cpu 1.15.1 IndexFlatIP (exact inner product)
pool -> prompt : 20 candidates retrieved, 3 promoted
cost per question: 32 ms retrieve + 1223 ms rerank
WHAT THE RERANKER CHANGED - 5 answerable questions, scored against written gold judgements
answer was in the pool at all : 1.00 <- the ceiling
precision@3 vector -> reranked : 0.40 -> 0.40 (+0.00)
MRR vector -> reranked : 0.90 -> 1.00 (+0.10)
hit@3 vector -> reranked : 1.00 -> 1.00 (+0.00)
reranked order differs from the vector order on 4 of 6 questions
THE ANSWERS - nvidia/nemotron-3-ultra-550b-a55b:free, 18 scored calls of 18, all :free
total cost : $0.000000
ungrounded full_credit 0.00 · coverage 0.83 · cited 0% · unsafe 1 · doses stated on the unanswerable question 1
vector full_credit 1.00 · coverage 1.00 · cited 83% · unsafe 0 · doses stated on the unanswerable question 0
reranked full_credit 1.00 · coverage 1.00 · cited 83% · unsafe 0 · doses stated on the unanswerable question 0
unanswerable question: refused in 2 of 3 answered condition(s), invented a dose in 1
What did the reranker actually change?
One dot per stage per question, on the same axis: the rank of the first passage that actually answers the question, inside the pool of 20 candidates. Lower is better, so a dot moving LEFT is a promotion. The number at the right of each row is the move. A dot parked at the far right never surfaced at all — and no reranker can rescue a passage stage 1 did not return. Rows are sorted by the size of the move.
How big should a chunk be?
The whole pipeline re-measured at 4 chunk sizes, pool fixed at 20. The red dashed squares are stage 1 alone; the solid blue circles are what the cross-encoder made of the same pool. The number above each pair is what the reranker added there, so the curve you are really reading is does the reranker help more when chunks are small and noisy, or large and vague. Chunk counts are labelled underneath. the pool of 20 held the answer on 100% of questions at EVERY chunk size, so the ceiling is flat at 1.00 -- it is in the table below, not on the chart, because a flat line shows nothing.
Side by side: nephrology_referral
The question the cross-encoder moved furthest in this run. Gold answer is on page(s) 5. Left is what raw vector search surfaced; right is what the cross-encoder promoted from the same pool. pages 4 and 5 both carry eGFR thresholds and both are about kidney function. Only page 5 states a referral trigger.
1. page 5 · p5c1 · cosine 0.544
5. Chronic kidney disease - staging, monitoring and nephrotoxins Stage chronic kidney disease by eGFR. G1 is 90 or above, G2 is 60 to 89, G3a is 45 to 59, G3b is 30 to 44, G4 is 15 to 29, and G5 is below 15. Stage separately by albuminuria: A1 is below 30 mg per gram, A2 is 30 to 300, and A3 is abov...
2. page 5 · p5c5 · cosine 0.494 <== ANSWERS IT
Refer to nephrology at stage G4, that is an eGFR below 30, or earlier if there is rapid progression, heavy albuminuria, resistant hypertension, or an unexplained fall in eGFR of more than 25 percent. Sick-day rules for intercurrent illness are listed in section 8.
3. page 2 · p2c4 · cosine 0.448
For Black adults without chronic kidney disease, start with a thiazide-like diuretic or a calcium channel blocker; ACE inhibitors are less effective as monotherapy in this group. Recheck the reading four weeks after any change, and confirm office readings against home readings before escalating a th...
1. page 5 · p5c5 · logit -1.99, was vector rank 2 <== ANSWERS IT
Refer to nephrology at stage G4, that is an eGFR below 30, or earlier if there is rapid progression, heavy albuminuria, resistant hypertension, or an unexplained fall in eGFR of more than 25 percent. Sick-day rules for intercurrent illness are listed in section 8.
2. page 1 · p1c2 · logit -5.10, was vector rank 7
Scope: adults aged 18 and over, in the ambulatory setting only. This handbook does NOT cover paediatric dosing, inpatient or critical care orders, obstetric patients, or chemotherapy. A question outside that scope has no answer here, and the correct response is to escalate rather than to extrapolate...
3. page 1 · p1c1 · logit -6.38, was vector rank 8
1. Scope and how to use this handbook This handbook is the ambulatory reference for adult primary care and specialty clinic staff. It states the health system's default position on common prescribing and monitoring decisions so that two clinicians asking the same question get the same answer.
Three conditions, one model, one system prompt
The only thing that changed between the three columns is which passages — if any — reached the model. Cost is $0 because every model here is on OpenRouter's free tier; it lives in the table rather than on an axis, because a cost axis that reads $0.000000 three times over shows nothing.
n full_credit coverage cited_rate unsafe unanswerable_doses mean_seconds total_cost_usd
condition
ungrounded 6 0.0 0.833 0.000 1 1 8.503 0.0
vector 6 1.0 1.000 0.833 0 0 22.772 0.0
reranked 6 1.0 1.000 0.833 0 0 5.183 0.0
The question the document cannot answer
What amoxicillin dose should we give a 4-year-old with acute otitis media?
[ungrounded] passages shown: [-] -> INVENTED A DOSE
**Amoxicillin dose for acute otitis media in a 4‑year‑old:**
**80–90 mg/kg/day divided every 12 hours (BID).**
**Practical examples (using average 4‑year‑old weight ~16 kg):**
- **Total daily dose:** 1,280–1,440 mg/day
- **Per dose (BID):** 640–720 mg
**Common prescriptions:**
- **Suspension (400 mg/5 mL):** 8–9 mL (≈640–720 mg) every 12 hours
- **Tablets:** 500 mg or 875 mg every 12 hours (choose strength closest to calculated mg/kg dose)
**Duration:** Typically 10 days (5–7 da
[vector] passages shown: [6,4,3] -> REFUSED
NOT IN DOCUMENT
[reranked] passages shown: [6,4,3] -> REFUSED
NOT IN DOCUMENT
Every question, before and after
question voice answerable gold_pages pool_rank vector_rank reranked_rank vector_pages reranked_pages precision_vector precision_reranked rr_vector rr_reranked hit3_vector hit3_reranked gold_in_pool vector_ms rerank_ms
metformin_ckd clinician True 4 1.0 1.0 1.0 4,4,4 4,4,3 0.3333 0.3333 1.0 1.0 1 1 1 33.3 1713.0
mech_valve_inr clinician True 6 1.0 1.0 1.0 6,6,6 6,6,6 0.3333 0.3333 1.0 1.0 1 1 1 43.6 1581.7
st_johns_wort patient True 7 1.0 1.0 1.0 7,7,7 7,7,6 0.6667 0.6667 1.0 1.0 1 1 1 42.8 1229.1
nephrology_referral clinician True 5 2.0 2.0 1.0 5,5,2 5,1,1 0.3333 0.3333 0.5 1.0 1 1 1 22.4 1054.8
metformin_titration patient True 3 1.0 1.0 1.0 3,4,4 3,4,3 0.3333 0.3333 1.0 1.0 1 1 1 22.3 878.9
paediatric_amoxicillin clinician False - NaN NaN NaN 6,4,3 6,4,3 0.0000 0.0000 0.0 0.0 0 0 0 26.5 878.0
How this was measured
The reference document is generated inside the notebook: ten authored pages rendered to a real PDF with reportlab, then read back with pypdf (PDF (reportlab wrote it, pypdf read it back)). 100.0% of the authored sentences survived the round trip, and that number is measured at run time rather than assumed.
Chunking splits each page separately on sentence boundaries, so no chunk can span two pages and every chunk carries the page number it came from. That produced 46 chunks (median 280 characters). Passages were embedded with sentence-transformers/all-MiniLM-L6-v2 and stored in faiss-cpu 1.15.1 IndexFlatIP (exact inner product); the query is embedded with the same model and scored by cosine.
Stage 1 returns 20 candidates. Stage 2 re-scores each (question, passage) PAIR with cross-encoder/ms-marco-MiniLM-L6-v2 and promotes 3. The two score scales are kept in separate columns: cosine is bounded and comparable across queries, the cross-encoder logit is neither.
Retrieval is scored against six questions with written judgements (5 (page, phrase) pairs). A chunk counts as a hit only when its PAGE matches and the gold PHRASE appears in its text, so retrieving the right page with the wrong passage does not score. One of the six is deliberately unanswerable and is scored separately.
Generation used nvidia/nemotron-3-ultra-550b-a55b:free through OpenRouter at temperature 0, 18 scored calls of 18 attempted, total measured cost $0.000000. Every score comes from a rule printed in the notebook: must_include synonym groups, must_not_include banned claims, and a citation regex over the model's own output.
This is a ten-page synthetic handbook with a six-question gold set: a demonstration of method, not a clinical validation. The handbook is invented teaching text for a fictional health system. It is not clinical guidance, there is no PHI anywhere in it, and nothing in it should be acted on. Numbers are from one run and will vary; the pattern is the finding, not the digits.