Paper

{
  "raw_xml": "<entry>\n    <id>http://arxiv.org/abs/2603.12117v1</id>\n    <title>SommBench: Assessing Sommelier Expertise of Language Models</title>\n    <updated>2026-03-12T16:19:04Z</updated>\n    <link href='https://arxiv.org/abs/2603.12117v1' rel='alternate' type='text/html'/>\n    <link href='https://arxiv.org/pdf/2603.12117v1' rel='related' title='pdf' type='application/pdf'/>\n    <summary>With the rapid advances of large language models, it becomes increasingly important to systematically evaluate their multilingual and multicultural capabilities. Previous cultural evaluation benchmarks focus mainly on basic cultural knowledge that can be encoded in linguistic form. Here, we propose SommBench, a multilingual benchmark to assess sommelier expertise, a domain deeply grounded in the senses of smell and taste. While language models learn about sensory properties exclusively through textual descriptions, SommBench tests whether this textual grounding is sufficient to emulate expert-level sensory judgment. SommBench comprises three main tasks: Wine Theory Question Answering (WTQA), Wine Feature Completion (WFC), and Food-Wine Pairing (FWP). SommBench is available in multiple languages: English, Slovak, Swedish, Finnish, German, Danish, Italian, and Spanish. This helps separate a language model's wine expertise from its language skills. The benchmark datasets were developed in close collaboration with a professional sommelier and native speakers of the respective languages, resulting in 1,024 wine theory question-answering questions, 1,000 wine feature-completion examples, and 1,000 food-wine pairing examples. We provide results for the most popular language models, including closed-weights models such as Gemini 2.5, and open-weights models, such as GPT-OSS and Qwen 3. Our results show that the most capable models perform well on wine theory question answering (up to 97% correct with a closed-weights model), yet feature completion (peaking at 65%) and food-wine pairing show (MCC ranging between 0 and 0.39) turn out to be more challenging. These results position SommBench as an interesting and challenging benchmark for evaluating the sommelier expertise of language models. The benchmark is publicly available at https://github.com/sommify/sommbench.</summary>\n    <category scheme='http://arxiv.org/schemas/atom' term='cs.CL'/>\n    <category scheme='http://arxiv.org/schemas/atom' term='cs.AI'/>\n    <published>2026-03-12T16:19:04Z</published>\n    <arxiv:primary_category term='cs.CL'/>\n    <author>\n      <name>William Brach</name>\n    </author>\n    <author>\n      <name>Tomas Bedej</name>\n    </author>\n    <author>\n      <name>Jacob Nielsen</name>\n    </author>\n    <author>\n      <name>Jacob Pichna</name>\n    </author>\n    <author>\n      <name>Juraj Bedej</name>\n    </author>\n    <author>\n      <name>Eemeli Saarensilta</name>\n    </author>\n    <author>\n      <name>Julie Dupouy</name>\n    </author>\n    <author>\n      <name>Gianluca Barmina</name>\n    </author>\n    <author>\n      <name>Andrea Blasi Núñez</name>\n    </author>\n    <author>\n      <name>Peter Schneider-Kamp</name>\n    </author>\n    <author>\n      <name>Kristian Košťál</name>\n    </author>\n    <author>\n      <name>Michal Ries</name>\n    </author>\n    <author>\n      <name>Lukas Galke Poech</name>\n    </author>\n  </entry>"
}