Paper
Developing the PsyCogMetrics AI Lab to Evaluate Large Language Models and Advance Cognitive Science -- A Three-Cycle Action Design Science Study
Authors
Zhiye Jin, Yibai Li, K. D. Joshi, Xuefei, Deng, Xiaobing, Li
Abstract
This study presents the development of the PsyCogMetrics AI Lab (psycogmetrics.ai), an integrated, cloud-based platform that operationalizes psychometric and cognitive-science methodologies for Large Language Model (LLM) evaluation. Framed as a three-cycle Action Design Science study, the Relevance Cycle identifies key limitations in current evaluation methods and unfulfilled stakeholder needs. The Rigor Cycle draws on kernel theories such as Popperian falsifiability, Classical Test Theory, and Cognitive Load Theory to derive deductive design objectives. The Design Cycle operationalizes these objectives through nested Build-Intervene-Evaluate loops. The study contributes a novel IT artifact, a validated design for LLM evaluation, benefiting research at the intersection of AI, psychology, cognitive science, and the social and behavioral sciences.
Metadata
Related papers
Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini
Ruofei Du, Benjamin Hersh, David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongy... • 2026-03-25
Comparing Developer and LLM Biases in Code Evaluation
Aditya Mittal, Ryan Shar, Zichu Wu, Shyam Agarwal, Tongshuang Wu, Chris Donah... • 2026-03-25
The Stochastic Gap: A Markovian Framework for Pre-Deployment Reliability and Oversight-Cost Auditing in Agentic Artificial Intelligence
Biplab Pal, Santanu Bhattacharya • 2026-03-25
Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA
Saahil Mathur, Ryan David Rittner, Vedant Ajit Thakur, Daniel Stuart Schiff, ... • 2026-03-25
MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucination
Zhuo Li, Yupeng Zhang, Pengyu Cheng, Jiajun Song, Mengyu Zhou, Hao Li, Shujie... • 2026-03-25
Raw Data (Debug)
{
"raw_xml": "<entry>\n <id>http://arxiv.org/abs/2603.13126v1</id>\n <title>Developing the PsyCogMetrics AI Lab to Evaluate Large Language Models and Advance Cognitive Science -- A Three-Cycle Action Design Science Study</title>\n <updated>2026-03-13T16:17:45Z</updated>\n <link href='https://arxiv.org/abs/2603.13126v1' rel='alternate' type='text/html'/>\n <link href='https://arxiv.org/pdf/2603.13126v1' rel='related' title='pdf' type='application/pdf'/>\n <summary>This study presents the development of the PsyCogMetrics AI Lab (psycogmetrics.ai), an integrated, cloud-based platform that operationalizes psychometric and cognitive-science methodologies for Large Language Model (LLM) evaluation. Framed as a three-cycle Action Design Science study, the Relevance Cycle identifies key limitations in current evaluation methods and unfulfilled stakeholder needs. The Rigor Cycle draws on kernel theories such as Popperian falsifiability, Classical Test Theory, and Cognitive Load Theory to derive deductive design objectives. The Design Cycle operationalizes these objectives through nested Build-Intervene-Evaluate loops. The study contributes a novel IT artifact, a validated design for LLM evaluation, benefiting research at the intersection of AI, psychology, cognitive science, and the social and behavioral sciences.</summary>\n <category scheme='http://arxiv.org/schemas/atom' term='q-bio.NC'/>\n <category scheme='http://arxiv.org/schemas/atom' term='cs.AI'/>\n <published>2026-03-13T16:17:45Z</published>\n <arxiv:comment>10 pages. Prepared: April 2025; submitted: June 15, 2025; accepted: August 2025. In: Proceedings of the 59th Hawaii International Conference on System Sciences (HICSS 2026), January 2026</arxiv:comment>\n <arxiv:primary_category term='q-bio.NC'/>\n <arxiv:journal_ref>Proceedings of the 59th Hawaii International Conference on System Sciences (HICSS), January 2026, pp. 6952-6961</arxiv:journal_ref>\n <author>\n <name>Zhiye Jin</name>\n <arxiv:affiliation>Nancy</arxiv:affiliation>\n </author>\n <author>\n <name>Yibai Li</name>\n <arxiv:affiliation>Nancy</arxiv:affiliation>\n </author>\n <author>\n <name>K. D. Joshi</name>\n <arxiv:affiliation>Nancy</arxiv:affiliation>\n </author>\n <author>\n <name> Xuefei</name>\n <arxiv:affiliation>Nancy</arxiv:affiliation>\n </author>\n <author>\n <name> Deng</name>\n <arxiv:affiliation>Emily</arxiv:affiliation>\n </author>\n <author>\n <name> Xiaobing</name>\n <arxiv:affiliation>Emily</arxiv:affiliation>\n </author>\n <author>\n <name> Li</name>\n </author>\n </entry>"
}