Paper

{
  "raw_xml": "<entry>\n    <id>http://arxiv.org/abs/2603.04598v1</id>\n    <title>PinPoint: Evaluation of Composed Image Retrieval with Explicit Negatives, Multi-Image Queries, and Paraphrase Testing</title>\n    <updated>2026-03-04T20:55:30Z</updated>\n    <link href='https://arxiv.org/abs/2603.04598v1' rel='alternate' type='text/html'/>\n    <link href='https://arxiv.org/pdf/2603.04598v1' rel='related' title='pdf' type='application/pdf'/>\n    <summary>Composed Image Retrieval (CIR) has made significant progress, yet current benchmarks are limited to single ground-truth answers and lack the annotations needed to evaluate false positive avoidance, robustness and multi-image reasoning. We present PinPoint, a comprehensive real world benchmark with 7,635 queries and 329K relevance judgments across 23 query categories. PinPoint advances the field by providing: (1) multiple correct answers (averaging 9.1 per query) (2) explicit hard negatives, (3) six instruction paraphrases per query for robustness testing, (4) multi-image composition support (13.4% of queries), and (5) demographic metadata for fairness evaluation. Based on our analysis of 20+ methods across 4 different major paradigms, we uncover three significant drawbacks: The best methods while achieving mAP@10 of 28.5%, still retrieves irrelevant results (hard negatives) 9% of the time. The best models also exhibit 25.1% performance variation across paraphrases, indicating significant potential for enhancing current CIR techniques. Multi-image queries performs 40 to 70% worse across different methods. To overcome these new issues uncovered by our evaluation framework, we propose a training-free reranking method based on an off-the-shelf MLLM that can be applied to any existing system to bridge the gap. We release the complete dataset, including all images, queries, annotations, retrieval index, and benchmarking code.</summary>\n    <category scheme='http://arxiv.org/schemas/atom' term='cs.CV'/>\n    <published>2026-03-04T20:55:30Z</published>\n    <arxiv:comment>Accepted for CVPR 2026</arxiv:comment>\n    <arxiv:primary_category term='cs.CV'/>\n    <author>\n      <name>Rohan Mahadev</name>\n    </author>\n    <author>\n      <name>Joyce Yuan</name>\n    </author>\n    <author>\n      <name>Patrick Poirson</name>\n    </author>\n    <author>\n      <name>David Xue</name>\n    </author>\n    <author>\n      <name>Hao-Yu Wu</name>\n    </author>\n    <author>\n      <name>Dmitry Kislyuk</name>\n    </author>\n  </entry>"
}