Paper
Procedural Generation of Algorithm Discovery Tasks in Machine Learning
Authors
Alexander D. Goldie, Zilin Wang, Adrian Hayler, Deepak Nathani, Edan Toledo, Ken Thampiratwong, Aleksandra Kalisz, Michael Beukman, Alistair Letcher, Shashank Reddy, Clarisse Wibault, Theo Wolf, Charles O'Neill, Uljad Berdica, Nicholas Roberts, Saeed Rahmani, Hannah Erlebach, Roberta Raileanu, Shimon Whiteson, Jakob N. Foerster
Abstract
Automating the development of machine learning algorithms has the potential to unlock new breakthroughs. However, our ability to improve and evaluate algorithm discovery systems has thus far been limited by existing task suites. They suffer from many issues, such as: poor evaluation methodologies; data contamination; and containing saturated or very similar problems. Here, we introduce DiscoGen, a procedural generator of algorithm discovery tasks for machine learning, such as developing optimisers for reinforcement learning or loss functions for image classification. Motivated by the success of procedural generation in reinforcement learning, DiscoGen spans millions of tasks of varying difficulty and complexity from a range of machine learning fields. These tasks are specified by a small number of configuration parameters and can be used to optimise algorithm discovery agents (ADAs). We present DiscoBench, a benchmark consisting of a fixed, small subset of DiscoGen tasks for principled evaluation of ADAs. Finally, we propose a number of ambitious, impactful research directions enabled by DiscoGen, in addition to experiments demonstrating its use for prompt optimisation of an ADA. DiscoGen is released open-source at https://github.com/AlexGoldie/discogen.
Metadata
Related papers
Vibe Coding XR: Accelerating AI + XR Prototyping with XR Blocks and Gemini
Ruofei Du, Benjamin Hersh, David Li, Nels Numan, Xun Qian, Yanhe Chen, Zhongy... • 2026-03-25
Comparing Developer and LLM Biases in Code Evaluation
Aditya Mittal, Ryan Shar, Zichu Wu, Shyam Agarwal, Tongshuang Wu, Chris Donah... • 2026-03-25
The Stochastic Gap: A Markovian Framework for Pre-Deployment Reliability and Oversight-Cost Auditing in Agentic Artificial Intelligence
Biplab Pal, Santanu Bhattacharya • 2026-03-25
Retrieval Improvements Do Not Guarantee Better Answers: A Study of RAG for AI Policy QA
Saahil Mathur, Ryan David Rittner, Vedant Ajit Thakur, Daniel Stuart Schiff, ... • 2026-03-25
MARCH: Multi-Agent Reinforced Self-Check for LLM Hallucination
Zhuo Li, Yupeng Zhang, Pengyu Cheng, Jiajun Song, Mengyu Zhou, Hao Li, Shujie... • 2026-03-25
Raw Data (Debug)
{
"raw_xml": "<entry>\n <id>http://arxiv.org/abs/2603.17863v1</id>\n <title>Procedural Generation of Algorithm Discovery Tasks in Machine Learning</title>\n <updated>2026-03-18T15:49:32Z</updated>\n <link href='https://arxiv.org/abs/2603.17863v1' rel='alternate' type='text/html'/>\n <link href='https://arxiv.org/pdf/2603.17863v1' rel='related' title='pdf' type='application/pdf'/>\n <summary>Automating the development of machine learning algorithms has the potential to unlock new breakthroughs. However, our ability to improve and evaluate algorithm discovery systems has thus far been limited by existing task suites. They suffer from many issues, such as: poor evaluation methodologies; data contamination; and containing saturated or very similar problems. Here, we introduce DiscoGen, a procedural generator of algorithm discovery tasks for machine learning, such as developing optimisers for reinforcement learning or loss functions for image classification. Motivated by the success of procedural generation in reinforcement learning, DiscoGen spans millions of tasks of varying difficulty and complexity from a range of machine learning fields. These tasks are specified by a small number of configuration parameters and can be used to optimise algorithm discovery agents (ADAs). We present DiscoBench, a benchmark consisting of a fixed, small subset of DiscoGen tasks for principled evaluation of ADAs. Finally, we propose a number of ambitious, impactful research directions enabled by DiscoGen, in addition to experiments demonstrating its use for prompt optimisation of an ADA. DiscoGen is released open-source at https://github.com/AlexGoldie/discogen.</summary>\n <category scheme='http://arxiv.org/schemas/atom' term='cs.LG'/>\n <category scheme='http://arxiv.org/schemas/atom' term='cs.AI'/>\n <published>2026-03-18T15:49:32Z</published>\n <arxiv:primary_category term='cs.LG'/>\n <author>\n <name>Alexander D. Goldie</name>\n </author>\n <author>\n <name>Zilin Wang</name>\n </author>\n <author>\n <name>Adrian Hayler</name>\n </author>\n <author>\n <name>Deepak Nathani</name>\n </author>\n <author>\n <name>Edan Toledo</name>\n </author>\n <author>\n <name>Ken Thampiratwong</name>\n </author>\n <author>\n <name>Aleksandra Kalisz</name>\n </author>\n <author>\n <name>Michael Beukman</name>\n </author>\n <author>\n <name>Alistair Letcher</name>\n </author>\n <author>\n <name>Shashank Reddy</name>\n </author>\n <author>\n <name>Clarisse Wibault</name>\n </author>\n <author>\n <name>Theo Wolf</name>\n </author>\n <author>\n <name>Charles O'Neill</name>\n </author>\n <author>\n <name>Uljad Berdica</name>\n </author>\n <author>\n <name>Nicholas Roberts</name>\n </author>\n <author>\n <name>Saeed Rahmani</name>\n </author>\n <author>\n <name>Hannah Erlebach</name>\n </author>\n <author>\n <name>Roberta Raileanu</name>\n </author>\n <author>\n <name>Shimon Whiteson</name>\n </author>\n <author>\n <name>Jakob N. Foerster</name>\n </author>\n </entry>"
}