Paper

{
  "raw_xml": "<entry>\n    <id>http://arxiv.org/abs/2603.06183v1</id>\n    <title>CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation</title>\n    <updated>2026-03-06T11:43:42Z</updated>\n    <link href='https://arxiv.org/abs/2603.06183v1' rel='alternate' type='text/html'/>\n    <link href='https://arxiv.org/pdf/2603.06183v1' rel='related' title='pdf' type='application/pdf'/>\n    <summary>We introduce CRIMSON, a clinically grounded evaluation framework for chest X-ray report generation that assesses reports based on diagnostic correctness, contextual relevance, and patient safety. Unlike prior metrics, CRIMSON incorporates full clinical context, including patient age, indication, and guideline-based decision rules, and prevents normal or clinically insignificant findings from exerting disproportionate influence on the overall score. The framework categorizes errors into a comprehensive taxonomy covering false findings, missing findings, and eight attribute-level errors (e.g., location, severity, measurement, and diagnostic overinterpretation). Each finding is assigned a clinical significance level (urgent, actionable non-urgent, non-actionable, or expected/benign), based on a guideline developed in collaboration with attending cardiothoracic radiologists, enabling severity-aware weighting that prioritizes clinically consequential mistakes over benign discrepancies. CRIMSON is validated through strong alignment with clinically significant error counts annotated by six board-certified radiologists in ReXVal (Kendalls tau = 0.61-0.71; Pearsons r = 0.71-0.84), and through two additional benchmarks that we introduce. In RadJudge, a targeted suite of clinically challenging pass-fail scenarios, CRIMSON shows consistent agreement with expert judgment. In RadPref, a larger radiologist preference benchmark of over 100 pairwise cases with structured error categorization, severity modeling, and 1-5 overall quality ratings from three cardiothoracic radiologists, CRIMSON achieves the strongest alignment with radiologist preferences. We release the metric, the evaluation benchmarks, RadJudge and RadPref, and a fine-tuned MedGemma model to enable reproducible evaluation of report generation, all available at https://github.com/rajpurkarlab/CRIMSON.</summary>\n    <category scheme='http://arxiv.org/schemas/atom' term='cs.CL'/>\n    <category scheme='http://arxiv.org/schemas/atom' term='cs.AI'/>\n    <category scheme='http://arxiv.org/schemas/atom' term='cs.CV'/>\n    <published>2026-03-06T11:43:42Z</published>\n    <arxiv:primary_category term='cs.CL'/>\n    <author>\n      <name>Mohammed Baharoon</name>\n    </author>\n    <author>\n      <name>Thibault Heintz</name>\n    </author>\n    <author>\n      <name>Siavash Raissi</name>\n    </author>\n    <author>\n      <name>Mahmoud Alabbad</name>\n    </author>\n    <author>\n      <name>Mona Alhammad</name>\n    </author>\n    <author>\n      <name>Hassan AlOmaish</name>\n    </author>\n    <author>\n      <name>Sung Eun Kim</name>\n    </author>\n    <author>\n      <name>Oishi Banerjee</name>\n    </author>\n    <author>\n      <name>Pranav Rajpurkar</name>\n    </author>\n  </entry>"
}