迈向执行接地的自动化 AI 研究
"Towards Execution-Grounded Automated AI Research"
推荐全译对照查看原文(PDF) ↗
导读
这篇论文来自斯坦福大学(Chenglei Si、Zitong Yang 等,Yejin Choi、Emmanuel Candès、Diyi Yang、Tatsunori Hashimoto 团队),安排在第 1 周课程导论,是「Agent 能否真正自动化 AI 研究本身」这一课程主线的代表性实证工作。此前的研究(如对 LLM 生成研究想法的大规模专家评审)表明:LLM 生成的想法常常看起来很有说服力,但被人执行后并无效。本文提出用「执行接地(execution grounding)」来解决这一问题:构建一个高吞吐自动化执行器,把自然语言研究想法变成代码、在 GPU 集群上真正跑实验,再以实验结果作为反馈信号去改进想法生成。作者把两个真实的 GPU 密集型研究问题——LLM 预训练(改进 nanoGPT)与后训练(改进 GRPO)——改造成执行环境,系统比较了进化搜索与强化学习两种学习算法:前者样本高效、10 个搜索轮次内即超越基线甚至人类专家,后者虽能提高平均奖励却因模式坍缩(mode collapse)无法提升上界。论文对「自动化 AI 研究员」的可行性与局限都给出了扎实的量化证据。
全文对照翻译
译注:以下自摘要(Abstract)起逐段给出英文原文与中文全译;参考文献(References)从略;附录中纯代码 diff 部分从略(以译注标明),其余实质内容均收录。
摘要(Abstract)
Automated AI research holds great potential to accelerate scientific discovery. However, current LLMs often generate plausible-looking but ineffective ideas. Execution grounding may help, but it is unclear whether automated execution is feasible and whether LLMs can learn from the execution feedback. To investigate these, we first build an automated executor to implement ideas and launch large-scale parallel GPU experiments to verify their effectiveness. We then convert two realistic research problems – LLM pre-training and post-training – into execution environments and demonstrate that our automated executor can implement a large fraction of the ideas sampled from frontier LLMs. We analyze two methods to learn from the execution feedback: evolutionary search and reinforcement learning. Execution-guided evolutionary search is sample-efficient: it finds a method that significantly outperforms the GRPO baseline (69.4% vs 48.0%) on post-training, and finds a pre-training recipe that outperforms the nanoGPT baseline (19.7 minutes vs 35.9 minutes) on pre-training, all within just ten search epochs. Frontier LLMs often generate meaningful algorithmic ideas during search, but they tend to saturate early and only occasionally exhibit scaling trends. Reinforcement learning from execution reward, on the other hand, suffers from mode collapse. It successfully improves the average reward of the ideator model but not the upper-bound, due to models converging on simple ideas. We thoroughly analyze the executed ideas and training dynamics to facilitate future efforts towards execution-grounded automated AI research.
自动化 AI 研究(automated AI research)在加速科学发现方面潜力巨大。然而,当前的 LLM 常常生成看似合理(plausible-looking)却无效的想法。执行接地(execution grounding)或许能帮上忙,但自动化执行是否可行、LLM 能否从执行反馈(execution feedback)中学习,仍不清楚。为研究这些问题,我们首先构建一个自动化执行器(automated executor)来实现想法,并发起大规模并行 GPU 实验验证其有效性;随后,我们把两个现实的研究问题——LLM 预训练(pre-training)与后训练(post-training)——转化为执行环境,并证明该自动化执行器能够实现很大比例的来自前沿 LLM 采样的想法。我们分析了两种从执行反馈中学习的方法:进化搜索(evolutionary search)与强化学习(reinforcement learning,RL)。执行引导的进化搜索具有样本效率(sample-efficient):它在后训练任务上找到了显著优于 GRPO 基线的方法(69.4% 对 48.0%),在预训练任务上找到了优于 nanoGPT 基线的配方(19.7 分钟对 35.9 分钟),而这一切只用了十个搜索轮次(epoch)。前沿 LLM 在搜索过程中常能生成有意义的算法性(algorithmic)想法,但它们往往早早饱和,只偶尔展现扩展(scaling)趋势。相比之下,从执行奖励出发的强化学习则遭受模式坍缩(mode collapse):它成功提升了想法生成(ideator)模型的平均奖励,却因模型收敛到简单想法而无法提升上界(upper-bound)。我们对被执行的想法与训练动态做了透彻分析,以促进未来迈向执行接地的自动化 AI 研究的努力。
[图 1: We build an automated idea executor involving Implementer, Scheduler, and Worker. We then use this automated executor as a reward function to teach LLMs to generate more effective ideas through evolutionary search and RL. We only update the ideator in the learning process.]
图 1 说明:我们构建了一个由实现者(Implementer)、调度器(Scheduler)与工作者(Worker)组成的自动化想法执行器;随后将该自动执行器用作奖励函数,通过进化搜索与 RL 教 LLM 生成更有效的想法。学习过程中只更新想法生成器(ideator)。
1 引言(Introduction)
We envision automated AI research: LLMs generate research ideas to tackle important research problems, implement the ideas as code, run experiments to verify the effectiveness, and continuously learn from the execution results. If successful, these automated AI researchers could automatically develop and identify effective research ideas in a massive search space, thereby scalably converting compute into scientific discovery; the discovered ideas could, in turn, improve frontier AI models themselves, enabling recursive self-improvement. Despite the promise, automated AI research is bottlenecked by the ability of LLMs to generate effective ideas. Si et al. (2025b) and Si et al. (2025a) evaluated the quality of LLM-generated research ideas through large-scale expert review and found that LLM ideas often look convincing but are ineffective after being executed by human researchers.
我们设想这样的自动化 AI 研究:LLM 生成针对重要研究问题的研究想法、把想法实现为代码、运行实验验证其有效性、并持续从执行结果中学习。若能成功,这些「自动化 AI 研究员」就能在巨大的搜索空间中自动开发并识别有效的研究想法,从而可扩展地把算力转化为科学发现;被发现的想法又能反过来改进前沿 AI 模型自身,实现递归自我改进(recursive self-improvement)。尽管前景可观,自动化 AI 研究的瓶颈在于 LLM 生成有效想法的能力。Si et al. (2025b) 与 Si et al. (2025a) 通过大规模专家评审评估了 LLM 生成研究想法的质量,发现 LLM 想法常常看起来令人信服,但被人类研究者执行后并无效。
This highlights the need to ground idea generation in execution. However, obtaining execution results of ideas in an automated and scalable manner is challenging, especially since we are targeting open-ended AI research where any ideas expressible in natural language are within our action space. To tackle this, we design and build a high-throughput automated idea executor that can implement hundreds of model-generated ideas and execute them in parallel to obtain the experiment results as execution feedback.
这凸显了把想法生成接地在执行上的必要性。然而,以自动化、可扩展的方式获得想法的执行结果并不容易,尤其因为我们瞄准的是开放式(open-ended)AI 研究——任何能用自然语言表达的想法都在动作空间之内。为此,我们设计并构建了一个高吞吐的自动化想法执行器,它能实现数百个模型生成的想法并并行执行,以获得实验结果作为执行反馈。
To study the extent to which we can automate realistic LLM research, we chose two GPU-intensive research problems (LLM pre-training and post-training) that are critical for improving the capabilities of LLMs as the research environments for our automated AI researchers. For the first time, we demonstrate that our automated executor can implement a large fraction of LLM-generated ideas on these challenging open-ended research problems, with over 90% execution rates on the pre-training environment with Claude-4.5-Sonnet and Claude-4.5-Opus.
为研究能在多大程度上自动化现实的 LLM 研究,我们选择了两个 GPU 密集、对提升 LLM 能力至关重要的研究问题(LLM 预训练与后训练),作为自动化 AI 研究员的研究环境。我们首次证明,我们的自动执行器能在这些具有挑战性的开放式研究问题上实现很大比例的 LLM 生成想法——在预训练环境中,配合 Claude-4.5-Sonnet 与 Claude-4.5-Opus 的执行率超过 90%。
To analyze whether grounding on execution-feedback can improve LLM idea generation, we define objective performance metrics for both environments and analyze the strengths and weaknesses of two popular learning algorithms: evolutionary search and reinforcement learning. We use our automated executor to guide evolutionary search. Within ten search epochs, this execution-guided search finds a post-training recipe that outperforms the GRPO baseline (69.4% vs 48.0%) on the task of post-training a 1.5B model for math reasoning, and a pre-training recipe that outperforms the nanoGPT baseline (19.7 minutes vs 35.9 minutes) on the task of minimizing the training wall-clock time to reach the target validation loss (Table 1). Our analysis shows that models are often generating algorithmic ideas apart from tuning hyper-parameters, and evolutionary search significantly outperforms best-of-N under the same sampling budget. However, when analyzing the scaling trend, only Claude-4.5-Opus shows a clear scaling curve, while both Claude-4.5-Sonnet and GPT-5 tend to saturate early.
为分析基于执行反馈的接地能否改进 LLM 的想法生成,我们为两个环境定义了客观的性能指标,并分析了两种流行学习算法的强项与弱项:进化搜索与强化学习。我们用自动执行器来引导进化搜索。在十个搜索轮次内,这一执行引导的搜索在「后训练 1.5B 模型做数学推理」的任务上找到了优于 GRPO 基线的后训练配方(69.4% vs 48.0%),并在「最小化达到目标验证损失所需训练墙钟时间(wall-clock time)」的任务上找到了优于 nanoGPT 基线的预训练配方(19.7 分钟 vs 35.9 分钟)(表 1)。我们的分析显示,模型除调整超参数(hyper-parameter)外还常生成算法性想法,并且在同等采样预算下进化搜索显著优于 best-of-N。然而,在分析扩展趋势时,只有 Claude-4.5-Opus 呈现清晰的 scaling 曲线,而 Claude-4.5-Sonnet 与 GPT-5 都倾向于早早饱和。
We then use the automated executor as the reward function in an RL loop to finetune Qwen3-30B. We show that RL with execution reward can successfully improve the average reward of the ideator model, similar to typical RL from verifiable rewards. However, RL does not improve the max reward, which is the more important metric for scientific discovery. In fact, we reveal that RL causes the ideator model to converge on a few easy-to-implement ideas, resulting in a collapse in thinking length and idea diversity.
接着,我们把自动执行器用作 RL 循环中的奖励函数来微调 Qwen3-30B。我们表明,带执行奖励的 RL 能像典型的可验证奖励 RL(RL from verifiable rewards)那样成功提升想法生成模型的平均奖励。然而,RL 并未提升最大奖励(max reward)——对科学发现而言更重要的指标。事实上,我们揭示出 RL 会使想法生成模型收敛到少数易于实现的简单想法,导致思考长度与想法多样性的坍缩。
In summary, we develop a large-scale automated idea executor system that can implement research ideas for open-ended and realistic research problems. Using this automated executor, we conduct an in-depth analysis of how well LLM ideators can learn from execution feedback to improve effectiveness through evolutionary search and RL. Execution-guided evolutionary search is sample-efficient and effective, but shows limited scaling. RL from execution reward suffers from diversity collapse and does not improve the upper-bound. We additionally provide extensive analysis on the executed ideas and suggest promising directions to improve the existing learning algorithms. Altogether, we demonstrate the feasibility and potential of grounding LLM ideation in automated execution and uncover important limitations for future improvement.
总之,我们开发了一个大规模的自动化想法执行器系统,能为开放式、现实的研究问题实现研究想法。利用这一自动执行器,我们深入分析了 LLM 想法生成器能多好地通过进化搜索与 RL 从执行反馈中学习、以提升有效性。执行引导的进化搜索样本高效且有效,但扩展性有限;执行奖励 RL 则遭受多样性坍缩、无法提升上界。我们还对被执行的想法提供了大量分析,并提出了改进现有学习算法的有前景方向。总而言之,我们证明了把 LLM 想法生成接地于自动执行的可行性与潜力,也揭示了有待未来改进的重要局限。
2 自动化想法执行器(Automated Idea Executor)
To measure the effectiveness of model-generated ideas, we build an automated executor that takes natural language research ideas as input, generates code implementations, runs the experiments on the backend, and returns the idea's benchmark performance as the final output.
为衡量模型生成想法的有效性,我们构建了一个自动执行器:它以自然语言研究想法为输入,生成代码实现,在后端运行实验,并把该想法的基准性能(benchmark performance)作为最终输出返回。
表 1:Performance of our execution-guided search in comparison with the provided baselines and best human experts. The post-training task is to finetune a 1.5B model for math reasoning, and the metric is validation accuracy. The pre-training task is to train a 124M Transformer on FineWeb, and the metric is the training time to reach 3.28 validation loss.
| 方法 | Post-Training(验证准确率↑) | Pre-Training(达标训练时间↓) |
|---|---|---|
| Baseline(基线) | 48.0% | 35.9 min |
| Execution-Guided Search(执行引导搜索) | 69.4% | 19.7 min |
| Best Human Expert(最佳人类专家) | 68.8% | 2.1 min |
表 1 说明:执行引导搜索与所给基线及最佳人类专家的对比。后训练任务是为数学推理微调一个 1.5B 模型,指标为验证准确率;预训练任务是在 FineWeb 上训练 124M Transformer,指标为达到 3.28 验证损失所需的训练时间。
2.1 用于想法生成的研究环境(Research Environments for Ideation)
Our automated idea executor is grounded in specific research environments, where each environment consists of a research problem, a baseline codebase, a benchmark to measure performance on, fixed training and evaluation data, and evaluation metrics. When constructing the research environments, we aim to select research problems that are open-ended, so that there is ample room for new algorithmic innovations, and at the same time have well-established baselines and benchmarking metrics so that measuring effectiveness is straightforward. In this work, we construct both a pre-training environment and a post-training environment for the automated AI researchers to work on.
Footnote 1: We open-source our environments and idea execution trajectories in https://github.com/NoviScl/Automated-AI-Researcher.
我们的自动想法执行器接地于具体的研究环境。每个环境由以下要素构成:一个研究问题、一份基线代码库(baseline codebase)、一个用于衡量性能的基准(benchmark)、固定的训练与评估数据,以及评估指标。在构建研究环境时,我们力求选择开放式的研究问题——给新的算法创新留出充足空间;同时要有成熟的基线与基准测试指标,使有效性衡量直接明了。本文中,我们构建了一个预训练环境与一个后训练环境,供自动化 AI 研究员工作。
脚注 1:我们在 https://github.com/NoviScl/Automated-AI-Researcher 开源了我们的环境与想法执行轨迹。
**Pre-Training Task: Improving nanoGPT** In the nanoGPT environment, we provide a baseline codebase adapted from the nanoGPT speedrun (Jordan et al., 2024) and ask the ideator model to brainstorm possible improvements. The original speedrun task is to minimize the time to pre-train a 124M GPT-2 model (Radford et al., 2019) on the FineWeb corpus (Penedo et al., 2024) to reach a validation loss of 3.28 on the validation set on 8 H100 GPUs. We did several modifications to the original speedrun setting. First, we introduce a proxy reward equal to the reciprocal of the validation loss ($\frac{1}{\text{loss}}$) when performing the search and RL experiments in later sections of the paper. This way, we can fix the training wall-clock time to be 25 minutes and ask the model to directly optimize the proxy reward under this fixed budget, so that we can avoid different runs having vastly different runtimes. We report the validation loss or the proxy reward metric in most plots, and only measure and report the training time metric for the top solution in order to directly compare it with the human experts' solutions on the original nanoGPT speedrun leaderboard. Second, to avoid any possible reward hacking, we freeze all evaluation hyper-parameters and implement an inference function that predicts one future token at a time to prevent models from changing the attention mechanism in a way that leaks future tokens (which happened multiple times during our initial development). We use this inference function during the final validation after each training run.
预训练任务:改进 nanoGPT。 在 nanoGPT 环境中,我们提供一份改编自 nanoGPT speedrun(Jordan et al., 2024)的基线代码库,并让想法生成模型构思可能的改进。原始 speedrun 任务是:在 8 张 H100 GPU 上、于 FineWeb 语料(Penedo et al., 2024)上预训练 124M GPT-2 模型(Radford et al., 2019),使其在验证集上达到 3.28 的验证损失,并最小化所需时间。我们对原始 speedrun 设定做了若干修改。其一,在论文后续章节的搜索与 RL 实验中,我们引入等于验证损失倒数的代理奖励(proxy reward,$\frac{1}{\text{loss}}$)。这样就能把训练墙钟时间固定为 25 分钟,让模型在该固定预算下直接优化代理奖励,从而避免不同运行之间的时长差异过大。我们在大多数图中报告验证损失或代理奖励指标,只为最优解测量并报告训练时间指标,以便与原 nanoGPT speedrun 排行榜上人类专家的解直接对比。其二,为杜绝任何可能的奖励欺骗(reward hacking),我们冻结所有评估超参数,并实现了一个一次只预测一个未来 token 的推理函数,防止模型以泄漏未来 token 的方式改动注意力机制(在我们初期开发中,此类问题多次出现)。我们在每次训练运行后的最终验证中使用该推理函数。
**Post-Training Task: Improving GRPO** In the GRPO environment, the baseline is an implementation of the GRPO algorithm (Shao et al., 2024) that finetunes a Qwen2.5-Math-1.5B checkpoint (Yang et al., 2024) on the MATH dataset (Hendrycks et al., 2021). The ideator model needs to brainstorm post-training algorithms that are more effective than the baseline. We specify a fixed training wall-clock time budget and use the max accuracy on the MATH validation set during training as the metric. To prevent reward hacking, we keep all validation-related code in a separate file and do not allow the automatic executor to access or modify it.
后训练任务:改进 GRPO。 在 GRPO 环境中,基线是 GRPO 算法(Shao et al., 2024)的一个实现:在 MATH 数据集(Hendrycks et al., 2021)上微调 Qwen2.5-Math-1.5B 检查点(Yang et al., 2024)。想法生成模型需要构思比基线更有效的后训练算法。我们指定固定的训练墙钟时间预算,并以训练期间 MATH 验证集上的最高准确率作为指标。为防止奖励欺骗,所有验证相关代码都保存在单独文件中,不允许自动执行器访问或修改。
In both environments, we do not set any constraints on the ideation scope, so anything between extensive hyper-parameter tuning and novel model architecture or training algorithms is within scope.
在这两个环境中,我们都不对想法范围设任何约束,因此从大规模超参数调优到全新的模型架构或训练算法,一切都在范围之内。
2.2 系统设计(System Design)
The automated idea executor can be viewed as a high-level API whose input is a batch of natural language ideas, and the output is the benchmark performance of each idea. There are three core building blocks of this API (Figure 1): Implementer – the server that generates the code diff for the idea and applies those changes; Scheduler – a middle layer that receives the list of codebases and allocates resources to run experiments; Worker – the cluster with GPU available that runs the experiments and uploads the experiment results.
自动想法执行器可视为一个高层 API:输入是一批自然语言想法,输出是每个想法的基准性能。该 API 有三个核心构建模块(图 1):实现者(Implementer)——为想法生成代码 diff 并应用这些改动的服务器;调度器(Scheduler)——接收代码库列表并分配资源运行实验的中间层;工作者(Worker)——拥有可用 GPU、运行实验并上传实验结果的集群。
**Implementer** The implementer is hosted on a CPU machine with high IO capacity. First, the user submits a batch of natural language ideas. Then, for each idea, the implementer makes parallelized API calls to the code execution LLM to obtain a diff file that can be patched into the corresponding baseline codebase. To optimize for efficiency, we prompt the code execution LLM with both the idea and the baseline codebase to sample 10 code diff files in parallel. For each sample, if the generated diff file cannot be patched into the original codebase, we provide the patch log and ask the model to revise the original generation. We repeat this sequential self-revision for a maximum of 2 times. In the end, we return the first code diff file that can be successfully patched into the baseline codebase. The patched codebase is then submitted to a cloud bucket as a .zip file.
实现者(Implementer)。 实现者部署在一台高 IO 能力的 CPU 机器上。首先,用户提交一批自然语言想法;随后,对每个想法,实现者并行调用代码执行 LLM 的 API,获得可打入对应基线代码库的 diff 文件。为优化效率,我们同时以想法与基线代码库为提示,让代码执行 LLM 并行采样 10 个 diff 文件。对每个采样,若生成的 diff 无法打入原代码库,我们就把补丁日志提供给模型,要求其修正最初的生成;这种顺序式自我修正最多重复 2 次。最终,我们返回第一个能成功打入基线代码库的 diff。打好补丁的代码库随后以 .zip 文件形式提交到云桶。
**Scheduler** Under a set clock frequency, the scheduler downloads the new codebases from the cloud. If the codebase has not been executed, the scheduler examines the resource requirement of the given research environment and prepares a job configuration to be submitted.
调度器(Scheduler)。 调度器按固定时钟频率从云端下载新代码库。若某代码库尚未被执行,调度器会检查给定研究环境的资源需求,并准备待提交的作业配置。
**Worker** Once the scheduler finds available resources, it connects the prepared job configuration with the GPU resource and initializes the worker to run the experiment. If the execution of the experiment is successful, the worker will upload the experiment logs including all performance metrics to another cloud bucket (wandb) along with the complete metadata: idea content, code change, execution log, etc. If the execution fails (e.g., due to bugs in code implementation), the worker halts. The user (i.e., the ideator model) can then download the execution results and see the performance of the batch of ideas they submitted with full training logs.
工作者(Worker)。 一旦调度器找到可用资源,便把准备好的作业配置与 GPU 资源连接,并初始化 worker 运行实验。若实验执行成功,worker 会把包含全部性能指标的实验日志连同完整元数据(想法内容、代码改动、执行日志等)上传到另一个云桶(wandb)。若执行失败(例如因代码实现中的 bug),worker 停止运行。用户(即想法生成模型)随后可下载执行结果,查看其提交的这批想法的性能与完整训练日志。
3 基准测试:LLM 想法生成器与执行器(Benchmarking LLM Ideators and Executors)
The prerequisite for an execution-grounded feedback loop is that current LLMs can serve as both ideators and executors, so that we can get meaningful reward signals for the models to learn from. To examine this prerequisite, we first benchmark various frontier LLMs as both the ideator and the executor.
执行接地反馈回路的前提是:当前 LLM 既能当想法生成器(ideator),又能当执行器(executor),这样我们才能获得有意义的奖励信号供模型学习。为检验这一前提,我们首先把多个前沿 LLM 同时当作想法生成器与执行器进行基准测试。
[图 2: Model performance comparison with self-execution (top row) vs GPT-5 execution (bottom row) on the GRPO and nanoGPT environments. The baseline accuracy for GRPO is 0.480, and the baseline loss for nanoGPT is 3.255. The completion rate is high for most models, especially under self-execution.]
图 2 说明:各模型在 GRPO 与 nanoGPT 环境上的性能对比,上排为自执行(self-execution),下排为 GPT-5 执行。GRPO 的基线准确率为 0.480,nanoGPT 的基线损失为 3.255。大多数模型的完成率(completion rate)都很高,自执行设定下尤其如此。
3.1 端到端的想法生成与执行(End-to-End Ideation and Execution)
In the first setting, we sample ideas from an LLM, and use the same LLM as the code execution model to execute its own ideas. We sample and execute 50 ideas from Claude-4.5-Opus, Claude-4.5-Sonnet, and GPT-5, and measure several metrics: (1) completion rate: the percentage of ideas that are successfully executed with a valid (non-zero) experiment result after execution; (2) average performance: the average validation accuracy or loss for all the successfully executed ideas among the 50 samples; (3) best performance: the highest validation accuracy or lowest validation loss among all executed ideas. We present results in the top row of Figure 2. Notably, a large fraction of the sampled ideas can indeed be executed successfully, with Claude-4.5-Opus and Claude-4.5-Sonnet having a significantly higher execution rate than GPT-5. Moreover, the best-of-N performance (N= 50) of these models can already beat the original baseline solutions. For example, on the GRPO environment, Claude-4.5-Sonnet gets a max accuracy of 60.4% as compared to the baseline of 48.0%; on nanoGPT, Claude-4.5-Opus gets a lowest loss of 3.237 as compared to the baseline of 3.255.
第一种设定是:从某个 LLM 采样想法,并用同一个 LLM 作为代码执行模型来执行它自己的想法。我们从 Claude-4.5-Opus、Claude-4.5-Sonnet 与 GPT-5 各采样并执行 50 个想法,并测量若干指标:(1)完成率(completion rate)——执行后获得有效(非零)实验结果的想法占比;(2)平均性能——50 个样本中所有成功执行想法的平均验证准确率或损失;(3)最佳性能——所有被执行想法中的最高验证准确率或最低验证损失。结果见图 2 上排。值得注意的是,大部分采样想法确实能被成功执行,其中 Claude-4.5-Opus 与 Claude-4.5-Sonnet 的执行率显著高于 GPT-5。此外,这些模型的 best-of-N(N=50)性能已能击败原始基线方案。例如,在 GRPO 环境上,Claude-4.5-Sonnet 取得 60.4% 的最高准确率,而基线为 48.0%;在 nanoGPT 上,Claude-4.5-Opus 取得 3.237 的最低损失,而基线为 3.255。
3.2 固定执行器、比较想法生成器(Comparing Ideators with the Same Executor)
In the second setting, we fix the executor model to be GPT-5 and use different ideator models to sample ideas. As shown in the bottom row of Figure 2, even when the ideator and executor are different models, the execution rate is still decent (ranging from 42% to 78%), although we do notice that the same ideas from Claude-4.5-Sonnet get a lower execution rate when executed by GPT-5 instead of itself (84% vs 42% on GRPO and 90% vs 78% on nanoGPT). Moreover, frontier open-weight models like Kimi-K2-Thinking (Kimi Team, 2025) and Qwen3-235B-A22B (Yang et al., 2025a) can also get non-trivial completion rates and achieve best-of-N performance that outperforms the baseline solutions in this setting. For example, Qwen3-235B achieves a max accuracy of 50.2% on GRPO and min loss of 3.238 on nanoGPT with N= 50, both better than the baselines. These benchmarking results demonstrate the feasibility of the automated ideation and execution loop. Next, we build search scaffolds and RL training loops to examine whether models can learn from the execution feedback.
第二种设定是:把执行器模型固定为 GPT-5,用不同的想法生成模型采样想法。如图 2 下排所示,即使想法生成器与执行器是不同模型,执行率仍然可观(42% 至 78%),不过我们确实注意到,Claude-4.5-Sonnet 的同一批想法由 GPT-5 执行时的执行率低于由其自身执行(GRPO 上 84% vs 42%,nanoGPT 上 90% vs 78%)。此外,前沿开源权重模型如 Kimi-K2-Thinking(Kimi Team, 2025)与 Qwen3-235B-A22B(Yang et al., 2025a)也能获得不俗的完成率,并在该设定下取得超越基线方案的 best-of-N 表现。例如,Qwen3-235B 在 N=50 时于 GRPO 上达到 50.2% 的最高准确率、于 nanoGPT 上达到 3.238 的最低损失,两者均优于基线。这些基准测试结果证明了自动化「想法生成—执行」回路的可行性。接下来,我们构建搜索脚手架与 RL 训练循环,检验模型能否从执行反馈中学习。
4 执行引导的进化搜索(Execution-Guided Evolutionary Search)
Evolutionary search (Koza, 1994; Lehman et al., 2023) is a traditional optimization method without the need for gradient updates. We develop an evolutionary search scaffold on top of frontier LLMs to optimize for effective ideas based on execution feedback. We introduce our search method that blends exploration and exploitation, its effectiveness on our two research environments, and various analyses of the generated ideas throughout the evolutionary search process.
进化搜索(evolutionary search)(Koza, 1994; Lehman et al., 2023)是一种无需梯度更新的传统优化方法。我们在前沿 LLM 之上开发了一个进化搜索脚手架(search scaffold),基于执行反馈来优化有效想法。我们将介绍这一融合探索(exploration)与利用(exploitation)的搜索方法、它在两个研究环境上的有效性,以及贯穿整个进化搜索过程对生成想法的多方面分析。
算法 1:Execution-Guided Search(执行引导搜索)
Require: batch size N, epochs T, baseline performance β
Require: initial exploitation rate a₁ ∈ [0, 100], annealing schedule a(t) for t ∈ {1, …, T}
1: Sample initial batch of ideas I₀ ← SAMPLEIDEAS(N)
2: Execute I₀ to obtain trajectories D₀ ← {(idea, reward)}
3: for t = 1 to T do
4: a ← a(t) ▷ (100 − a)% exploration rate
5: D₍₍<t₎₎ ← ⋃(k=0..t−1) Dₖ
6: D₊ ← {(i, r) ∈ D₍₍<t₎₎ : r > β} ▷ positive trajectories
7: Nₑₓₚ ← (a/100)·N, Nₑₓₚₗ ← N − Nₑₓₚ
8: Iₜᵉˣᵖ ← EXPLOITVARIANTS(D₊, Nₑₓₚ)
9: D̃₍₍<t₎₎ ← SUBSAMPLETOCONTEXT(D₍₍<t₎₎)
10: Iₜᵉˣᵖˡ ← EXPLORENOVEL(D̃₍₍<t₎₎, Nₑₓₚₗ)
11: Iₜ ← Iₜᵉˣᵖ ∪ Iₜᵉˣᵖˡ
12: Execute Iₜ to obtain trajectories Dₜ ← {(idea, reward)}
13: end for
14: return ⋃(t=0..T) Dₜ
算法 1 说明:执行引导搜索。输入为批大小 N、轮数 T 与基线性能 β。第 1–2 行先采样并执行一整批初始想法;之后每一轮按退火日程确定利用率 a(探索率为 100−a%):把此前所有轮次轨迹中奖励高于基线的正向轨迹交给「利用」子程序生成变体,再对历史轨迹下采样到上下文长度上限、交给「探索」子程序生成全新想法;合并两类想法执行,最终返回全部轨迹。
4.1 搜索脚手架(Search Scaffold)
Our search method is inspired by prior evolutionary search approaches for code optimization, such as AlphaEvolve (Novikov et al., 2025). Our algorithm is detailed in Algorithm 1. At the first search epoch, we sample a full batch of new ideas. In all subsequent epochs, we split the idea generation into exploitation and exploration subsets. For exploitation, we choose ideas from previous epochs that outperform the baseline and append them to the idea generation prompt to ask the ideator model to generate new variants that combine their strengths. For exploration, we randomly sample ideas from previous epochs to append to the idea generation prompt until reaching the max context length and instruct the ideator model to generate completely new ideas different from them. We start with 50% exploitation and 50% exploration at epoch 1 and gradually anneal the exploration rate and increase the exploitation ratio in later epochs. We use a batch size of 50 for the GRPO environment and a batch size of 80 for the nanoGPT environment.
我们的搜索方法受此前面向代码优化的进化搜索方法(如 AlphaEvolve,Novikov et al., 2025)启发,算法细节见算法 1。第一个搜索轮次采样一整批新想法;后续所有轮次,我们把想法生成分成「利用」与「探索」两个子集。利用部分:从之前轮次中选出优于基线的想法,附到想法生成提示中,要求想法生成模型生成结合这些想法优点的新变体。探索部分:从之前轮次随机抽取想法附到提示中,直至达到上下文长度上限,并指示想法生成模型生成与之完全不同的新想法。第 1 轮以 50% 利用 / 50% 探索开始,随后逐渐退火降低探索比例、在后续轮次提高利用占比。GRPO 环境批大小为 50,nanoGPT 环境批大小为 80。
4.2 实验结果(Experiment Results)
For each environment, we perform execution-guided search with three different models: Claude-4.5-Opus, Claude-4.5-Sonnet, and GPT-5. For each experiment, we use the same model as both the ideator and executor (i.e., self-execution). We plot the progression of the best performance at each search epoch in Figure 3. We summarize several notable trends below.
对每个环境,我们分别用三个不同模型执行执行引导搜索:Claude-4.5-Opus、Claude-4.5-Sonnet 与 GPT-5。每个实验中,我们用同一模型既当想法生成器又当执行器(即自执行)。我们在图 3 中绘制每个搜索轮次最佳性能的进展,并在下面总结若干显著趋势。
[图 3: Best performance at each epoch when performing execution-guided search with different models. For the nanoGPT environment (left), we use the reciprocal of the validation loss as the metric; for the GRPO environment (right), we use validation accuracy as the metric. Claude-4.5-Opus exhibits a scaling trend on both environments and achieves the best performance on nanoGPT. Claude-4.5-Sonnet achieves the best performance on GRPO due to effective hyper-parameter tuning, but saturates early.]
图 3 说明:用不同模型做执行引导搜索时每个轮次的最佳性能。nanoGPT 环境(左)以验证损失的倒数作为指标;GRPO 环境(右)以验证准确率作为指标。Claude-4.5-Opus 在两个环境上都呈现扩展(scaling)趋势,并在 nanoGPT 上取得最佳性能;Claude-4.5-Sonnet 凭借有效的超参数调优在 GRPO 上取得最佳性能,但早早饱和。
First, we observe a scaling trend with Claude-4.5-Opus, where searching for more epochs leads to a higher upper bound. In contrast, Claude-4.5-Sonnet and GPT-5 tend to saturate early. Second, all models can find ideas that significantly outperform the baselines. On GRPO, Claude-4.5-Sonnet finds that using vanilla policy gradient with the group-average baseline without importance reweighting or clipping outperforms the standard GRPO objective in this particular experiment setup and exploits this finding in all subsequent search epochs, resulting in the best solution of 69.4% at epoch 2 with precise hyper-parameter tuning. On nanoGPT, Claude-4.5-Opus achieves the min validation loss of 3.1407 at epoch 9 by combining various architectural modifications, hyper-parameter tuning, and applying exponential moving average of intermediate checkpoints during validation (see Appendix A.2 for the full idea). We run this top solution on 8 H100s to follow the same setup as the nanoGPT speedrun, and it reaches the 3.28 target validation loss in 19.7 minutes, a significant speedup as compared to the baseline codebase, which takes 35.9 minutes of training time to reach the same target validation loss.
第一,Claude-4.5-Opus 呈现扩展趋势:搜索轮次越多,上界越高;相比之下,Claude-4.5-Sonnet 与 GPT-5 倾向于早早饱和。第二,所有模型都能找到显著超越基线的想法。在 GRPO 上,Claude-4.5-Sonnet 发现:在本实验设置下,使用组平均基线的朴素策略梯度(vanilla policy gradient,不含重要性重加权与裁剪)优于标准 GRPO 目标,并在后续所有搜索轮次中利用这一发现,凭借精细的超参数调优在第 2 轮取得 69.4% 的最佳解。在 nanoGPT 上,Claude-4.5-Opus 通过组合多种架构改动、超参数调优,以及在验证期间对中间检查点做指数移动平均(exponential moving average,EMA),在第 9 轮达到 3.1407 的最低验证损失(完整想法见附录 A.2)。我们把这一最优解按 nanoGPT speedrun 的相同设置在 8 张 H100 上运行,19.7 分钟即达到 3.28 的目标验证损失;相比需要 35.9 分钟训练时间才能达到同一目标验证损失的基线代码库,这是显著的加速。
To better contextualize these solutions optimized by the model, we also compare the top performance of our execution-guided search to human experts (Table 1). For the GRPO environment, we compare with the leaderboard of the Stanford CS336 graduate-level LLM class, which hosted the same environment as an assignment for all students to optimize the validation accuracy under the same training time budget. The best student solution ² achieved an accuracy of 68.8%, lower than Claude-4.5-Sonnet's top solution using our execution-guided search. For the nanoGPT environment, we directly compare with the nanoGPT speedrun leaderboard, ³ where the top human solution as of December 2025 can reach the target validation loss under 2.1 minutes, indicating significant headroom for further model capability and search method improvement on this environment.
为更好地给这些由模型优化出的解定位,我们还将执行引导搜索的最优性能与人类专家做了对比(表 1)。GRPO 环境方面,我们与斯坦福 CS336 研究生 LLM 课程的排行榜对比——该课程把同一环境作为作业,让所有学生在相同训练时间预算下优化验证准确率;最佳学生方案(脚注 2)取得 68.8% 的准确率,低于 Claude-4.5-Sonnet 用我们的执行引导搜索得到的最优解。nanoGPT 环境方面,我们直接与 nanoGPT speedrun 排行榜(脚注 3)对比:截至 2025 年 12 月,人类最优解可在 2.1 分钟内达到目标验证损失,说明该环境上模型能力与搜索方法仍有很大提升空间。
脚注 2:https://github.com/stanford-cs336/assignment5-alignment-leaderboard 脚注 3:https://github.com/KellerJordan/modded-nanogpt
[图 4: Comparison between best-of-N (blue) and our execution-guided search (red) under the same sampling budget where we sample N= 80 ideas at each epoch.]
图 4 说明:在相同采样预算(每轮采样 N=80 个想法)下,best-of-N(蓝)与我们的执行引导搜索(红)的对比。
4.3 与 best-of-N 对比(Comparison with Best-of-N)
To demonstrate the effectiveness of our search scaffold, we compare our execution-guided search with the best-of-N baseline with the same sampling budget on the nanoGPT environment. Since the batch size for our search is 80, we compare the first 3 epochs of the execution-guided search using the GPT-5 backbone with the best-of-N results of GPT-5 with N ∈ {80, 160, 240}. As shown in Figure 4, search and best-of-N start from similar performance at epoch 0 (they are not exactly the same due to variances from sampling), but evolutionary search significantly outperforms best-of-N from epoch 1 onward, demonstrating that the model is effectively leveraging trajectories from previous epochs to generate more effective ideas in future epochs.
为展示搜索脚手架的有效性,我们在 nanoGPT 环境上以相同采样预算,把执行引导搜索与 best-of-N 基线对比。由于搜索批大小为 80,我们把采用 GPT-5 主干(backbone)的执行引导搜索的前 3 轮,与 GPT-5 在 N ∈ {80, 160, 240} 下的 best-of-N 结果相比。如图 4 所示,搜索与 best-of-N 在第 0 轮起点性能相近(因采样方差并不完全相同),但从第 1 轮起,进化搜索显著胜过 best-of-N,说明模型有效地利用了之前轮次的轨迹,在后续轮次生成更有效的想法。
4.4 生成想法的分析(Analysis of Generated Ideas)
表 2:Breakdown of hyper-parameter tuning vs algorithmic ideas throughout the entire execution-guided search. We report the percentage of each type among all generated ideas of each model (N= 500 ideas on GRPO and N= 800 ideas on nanoGPT). We also report the average and best performance for ideas under each category, where we use validation accuracy as the performance metric for GRPO and validation loss as the metric for nanoGPT. Bold numbers every row indicate the best performance by each model. All models generate a substantial amount of algorithmic ideas apart from hyper-parameter changes, while Claude-4.5-Sonnet generates significantly more hyper-parameter ideas than other models.
| 环境 / 模型 | 超参数类占比 | 超参数类平均 | 超参数类最佳 | 算法类占比 | 算法类平均 | 算法类最佳 |
|---|---|---|---|---|---|---|
| GRPO(准确率↑)GPT-5 | 5.0% | 45.0 | 50.2 | 95.0% | 44.5 | 60.0 |
| GRPO Claude-4.5-Sonnet | 41.1% | 48.4 | 69.4 | 58.9% | 45.0 | 67.4 |
| GRPO Claude-4.5-Opus | 3.7% | 44.4 | 50.4 | 96.3% | 46.5 | 61.6 |
| nanoGPT(损失↓)GPT-5 | 15.4% | 3.254 | 3.195 | 84.6% | 3.894 | 3.170 |
| nanoGPT Claude-4.5-Sonnet | 31.3% | 3.251 | 3.208 | 68.7% | 3.679 | 3.208 |
| nanoGPT Claude-4.5-Opus | 8.7% | 3.329 | 3.147 | 91.3% | 3.419 | 3.141 |
表 2 说明:整个执行引导搜索过程中「超参数调整类」与「算法类」想法的分布。我们报告每个模型全部生成想法中各类别的占比(GRPO 上共 N=500 个想法,nanoGPT 上共 N=800 个),以及各类别下想法的平均与最佳性能——GRPO 用验证准确率、nanoGPT 用验证损失作为指标。加粗数字为各模型每行的最佳表现。除超参数改动外,所有模型都生成了大量算法类想法;Claude-4.5-Sonnet 生成的超参数类想法显著多于其他模型。
**Hyper-parameter vs Algorithmic** To quantitatively understand the types of ideas that models generate during the execution-guided search, we perform a stratified analysis by classifying all generated ideas into either hyper-parameter tuning (including any ideas that can be implemented via changing existing configs) or algorithmic (including all ideas that involve implementing new changes not originally supported by the baseline codebase) by using an LLM-judge. Based on Table 2, all three models generate a substantial amount of algorithmic ideas apart from hyper-parameter tuning. Interestingly, different models exhibit different patterns, where Claude-4.5-Sonnet generates significantly more hyper-parameter ideas than both Claude-4.5-Opus and GPT-5. Moreover, the most effective ideas come from algorithmic ideas in most cases, except when using Claude-4.5-Sonnet.
超参数类 vs 算法类。 为定量理解模型在执行引导搜索中生成的想法类型,我们用 LLM 评审(LLM-judge)做分层分析,把全部生成想法分为两类:超参数调整类(包括一切通过修改现有配置即可实现的想法)与算法类(algorithmic,包括所有需要实现基线代码库原本不支持的新改动的想法)。由表 2 可见,除超参数调优外,三个模型都生成了大量算法类想法。有趣的是,不同模型呈现不同模式:Claude-4.5-Sonnet 生成的超参数类想法显著多于 Claude-4.5-Opus 与 GPT-5。此外,多数情况下最有效的想法来自算法类——使用 Claude-4.5-Sonnet 时是例外。
**Qualitative Examples** To complement the quantitative analysis, we provide several executed ideas on the GRPO environment in Table 3 and provide example ideas on the nanoGPT environment in Appendix A.2. When sampling ideas, models would generate a thinking trace, followed by the natural language idea and a brief description of all the code changes needed to implement the idea. For brevity, we only include the natural language ideas in the table, but we present additional examples in Appendix A.3 with more details, including full code execution trajectories. Based on the qualitative examples in Table 3, different models generate different styles of ideas. For example, Claude-4.5-Sonnet generates more intuitive ideas while Claude-4.5-Opus and GPT-5 are more mathematically inclined.
定性示例。 为补充定量分析,我们在表 3 中给出 GRPO 环境上若干已执行的想法,并在附录 A.2 给出 nanoGPT 环境的示例想法。采样想法时,模型会先生成一段思考轨迹(thinking trace),随后给出自然语言想法以及实现该想法所需的全部代码改动的简要描述。为简洁起见,表中只收录自然语言想法;更详细的示例(含完整代码执行轨迹)见附录 A.3。从表 3 的定性示例看,不同模型生成的想法风格各异:例如 Claude-4.5-Sonnet 的想法更直觉化,而 Claude-4.5-Opus 与 GPT-5 更偏数学。
表 3:Examples of successfully executed ideas on the GRPO environment, along with their accuracy on the MATH validation set. The baseline accuracy is 48.0% on this environment.
| Claude-4.5-Opus | Claude-4.5-Sonnet | GPT-5 |
|---|---|---|
| 带动量的残差比率学习边界(Residual Ratio Learning with Momentum Bounds):不直接使用(重要性采样)比率,而是把它分解为「基线」分量(批次均值比率的 EMA)与「残差」分量(ratio − base)。只对残差施加 sigmoid 约束,让基线分量捕捉系统性的策略漂移,同时控制对其的偏离;并结合动量裁剪自适应。公式:residual = ratio − ema_batch_ratio,bounded_residual = sigmoid_bound(residual, deviation),effective_ratio = 1.0 + bounded_residual。验证准确率:61.6 | 基于性能反馈的动态数学题难度平衡(Dynamic Mathematical Problem Difficulty Balancing with Performance Feedback):实现智能难度平衡,依据近期表现趋势动态调整各难度题目的配比。表现强劲时提高难度占比;遇到困难时提供更多基础题目。并与经过验证的超参数组合结合,以获得最优学习进程。验证准确率:64.0 | 经响应分块的 token 级比率去噪(Token-Level Ratio De-noising via Response Chunks, Chunked-Ratio):把响应切成小的连续块、对块内对数比率取平均,以减少带噪 token 尖峰。把每条序列的响应 token 划分为 C 块(如按有效长度取 C=8),在做比率与裁剪之前,把每 token 的 Δlog p 替换为块均值并广播到块内各 token。在平滑极端值的同时保留结构信号。验证准确率:58.2 |
| 优势排名差加权(Advantage Rank Difference Weighting):不使用绝对优势幅度,而是按样本排名与其在均匀分布下期望排名的偏离程度加权;显著优于或劣于其「期望」位置的样本获得更高权重。该方法与分布无关、对离群值稳健。公式:expected_rank = (N−1)/2,rank_diff = |actual_rank − expected_rank| / expected_rank,weight = 0.5 + 0.5 × rank_diff。验证准确率:59.2 | 实现 token 级奖励归因:用注意力权重识别哪些输入 token 对答对贡献最大,然后在策略梯度训练中放大这些 token 的梯度更新。验证准确率:45.2 | 基于奖励散度的逐组课程(Per-Group Curriculum via Reward Spread, PGC-RS):依据组内奖励散度调整步长激进程度。对每个组计算散度 s_g = std(r),再为每个样本计算温度 T_i^grp = clamp(1 + α·(s_ref − s_g), 0.8, 1.5),其中 s_ref 为 rollout 各组标准差的中位数、α = 0.8。把该温度乘到既有比率温度 T_i(若有)上。散度低的组(难以区分好坏)得到更「冷」的比率;散度高的组允许更大胆的更新。验证准确率:49.4 |
| — | 创建数学工作记忆模拟:在解题过程中维护一个由数学事实、定义与中间结果组成的上下文缓冲区。该缓冲区随模型解题不断更新,并为后续数学步骤提供额外上下文,模拟人类在复杂计算中维持数学工作记忆的方式。验证准确率:58.0 | — |
表 3 说明:GRPO 环境上成功执行的想法示例及其在 MATH 验证集上的准确率。该环境的基线准确率为 48.0%。
**Recovering Recent Research Papers** We observed multiple cases where the model-generated ideas (without any RAG) are highly similar to research papers released within the three months prior to the writing of this paper. For example, Claude-4.5-Sonnet proposed: "Implement response diversity rewards within groups where responses to the same prompt receive bonus rewards for being dissimilar to other responses in their group, encouraging exploration of different solution paths.", which is similar to Li et al. (2025). For pre-training, Claude-4.5-Opus proposed: "Causal Context Compression: Before each attention layer, apply a learned compression that mixes local context (previous 2-3 tokens) into the current representation, providing implicit local context without convolutions.", which is similar to the "canon layer" described in Allen-Zhu (2025). Although assessing the novelty of LLM-generated ideas is beyond the scope of this work, models' ability to rediscover ideas from recent research papers nevertheless indicates that automated AI researchers could plausibly support work at the frontier of LLM research.
复现近期研究论文。 我们观察到多起案例:模型生成的想法(未使用任何 RAG)与本文写作前三个月内发布的研究论文高度相似。例如,Claude-4.5-Sonnet 提出:「实现组内响应多样性奖励:同一提示的各响应若与组内其他响应不相似,则获得额外奖励,鼓励探索不同解题路径」,这与 Li et al. (2025) 相似;预训练方面,Claude-4.5-Opus 提出:「因果上下文压缩(Causal Context Compression):在每个注意力层之前,施加一个学习到的压缩,把局部上下文(前 2–3 个 token)混入当前表示,在不使用卷积的情况下提供隐式局部上下文」,这与 Allen-Zhu (2025) 描述的「canon layer」相似。尽管评估 LLM 生成想法的新颖性超出本文范围,但模型重新发现近期论文中想法的能力表明,自动化 AI 研究员有希望支撑 LLM 研究前沿的工作。
5 从执行奖励中强化学习(Reinforcement Learning from Execution Reward)
Different from evolutionary search, reinforcement learning is an alternative learning algorithm that shapes model behaviors through gradient updates. Despite much recent success on verifiable domains like math and coding (DeepSeek-AI et al., 2025), RL's effectiveness on open-ended AI research remains unclear. For the first time, we explore whether we can leverage the automated executor as a reward function to directly finetune LLMs to generate more effective ideas via RL. We detail our implementation, experiment setup, and analysis of the training dynamics.
与进化搜索不同,强化学习(reinforcement learning,RL)是另一种学习算法,通过梯度更新塑造模型行为。尽管 RL 近来在数学、代码等可验证领域屡获成功(DeepSeek-AI et al., 2025),它在开放式 AI 研究上的有效性仍不清楚。我们首次探索能否把自动执行器当作奖励函数,通过 RL 直接微调 LLM 以生成更有效的想法。我们将详述实现、实验设置与训练动态分析。
5.1 奖励设计与实验设置(Reward Design and Experiment Setup)
We use Qwen3-30B-A3B (Yang et al., 2025a) as the base model and finetune it using the standard GRPO algorithm (Shao et al., 2024), motivated by its consistent empirical success on other verified domains. Our prompt batch size is one since we only have one prompt for each environment. In the prompt, we provide the baseline codebase and ask the model to generate new ideas to improve the baseline (GRPO or nanoGPT). This experiment setup is similar to prior work exploring RLVR from one training example (Wang et al., 2025).
我们使用 Qwen3-30B-A3B(Yang et al., 2025a)作为基模型(base model),并采用标准 GRPO 算法(Shao et al., 2024)微调——这是受到 GRPO 在其他已验证领域持续的经验成功所启发。我们的提示批大小为 1,因为每个环境只有一个提示。提示中给出基线代码库,要求模型生成改进基线(GRPO 或 nanoGPT)的新想法。该实验设置类似于此前从单一训练样本做 RLVR 的探索(Wang et al., 2025)。
We use large group sizes to stabilize training. For the post-training environment, we use a group size of 256; for the pre-training environment, we use a group size of 128. Since each idea on the GRPO environment runs on one single GPU and each idea on the nanoGPT environment runs on 8 GPUs, these group sizes correspond to parallel execution on 256 GPUs (for GRPO) or 1024 GPUs (for nanoGPT) to obtain the execution reward on each batch of rollout ideas. Each rollout is a thinking trace followed by the natural language idea. We set a max output length of 8192 tokens for rollout sampling and only feed the extracted ideas to the automated executor without the preceding thinking trace.
我们使用大组规模(group size)以稳定训练:后训练环境组大小为 256,预训练环境组大小为 128。由于 GRPO 环境每个想法在单张 GPU 上运行、nanoGPT 环境每个想法在 8 张 GPU 上运行,这些组大小分别对应每批 rollout 想法动用 256 张(GRPO)或 1024 张(nanoGPT)GPU 并行执行以获取执行奖励。每条 rollout 是一段思考轨迹加自然语言想法;rollout 采样的最大输出长度设为 8192 token,且只把抽取出的想法(不带前面的思考轨迹)送入自动执行器。
For the post-training environment, we directly use the validation set accuracy of each rollout idea after execution as the reward. For ideas without a valid accuracy (i.e., when the execution failed due to code generation errors), we assign a reward of 0. For the pre-training environment, we use the reciprocal of the validation loss as the reward (1/loss) and assign a reward of 0 to ideas with failed execution. Our experiments are based on the Tinker API (Thinking Machines Lab, 2025).
后训练环境直接用每条 rollout 想法执行后的验证集准确率作为奖励;对没有有效准确率的想法(即因代码生成错误导致执行失败),奖励记 0。预训练环境用验证损失的倒数(1/loss)作为奖励,执行失败的想法同样记 0。我们的实验基于 Tinker API(Thinking Machines Lab, 2025)。
5.2 实验结果(Experiment Results)
[图 5: Training curves of RL from execution reward. We plot the average reward per epoch in the upper row, and the max reward per epoch in the lower row. For the GRPO environment, the reward is the accuracy; for the nanoGPT environment, the reward is the reciprocal of the loss. The average reward increases, but not the max reward.]
图 5 说明:执行奖励 RL 的训练曲线。上排为每个训练轮次的平均奖励,下排为每个轮次的最大奖励。GRPO 环境的奖励是准确率;nanoGPT 环境的奖励是损失的倒数。平均奖励上升,最大奖励则不然。
**Positive Training Curves for Average Reward** We plot the average reward of all rollouts of each training epoch in the upper row of Figure 5. For the first time, we demonstrate that the average performance of the generated ideas can increase after sufficient training epochs for open-ended research environments. For instance, the average accuracy on the GRPO environment increases from 0.253 at the beginning to 0.343 after 40 training epochs (top left plot of Figure 5); and the average reward on the nanoGPT environment increases from 0.194 at the beginning to 0.246 after 68 epochs (top right plot of Figure 5), corresponding to a decrease in the average validation loss from 5.150 to 4.066. Such training curves are similar to prior findings on the effectiveness of one-shot RVLR on other verified domains like math (Wang et al., 2025).
平均奖励的正向训练曲线。 我们在图 5 上排绘制每个训练轮次全部 rollout 的平均奖励。我们首次证明:在开放式研究环境上,生成想法的平均性能可以在足够多的训练轮次后提升。例如,GRPO 环境的平均准确率从开始的 0.253 升至 40 个训练轮次后的 0.343(图 5 左上图);nanoGPT 环境的平均奖励从开始的 0.194 升至 68 轮后的 0.246(图 5 右上图),对应平均验证损失从 5.150 降至 4.066。这样的训练曲线与此前关于单样本 RLVR 在数学等其他已验证领域上有效性的发现(Wang et al., 2025)相似。
**The Case of Max Reward** Despite successfully reproducing the positive training curves observed in other domains, we argue that there is a distinction between idea generation and other verifiable domains. For advancing scientific discovery, we often care about the upper-bound of idea generation, rather than the average quality. In our particular case, we care more about having one breakthrough idea that dominates the baselines, rather than having many safe ideas with a high average. Thus, we plot the max reward of all rollouts at each training epoch in the lower row of Figure 5. The trend here is very different – the max reward is fluctuating throughout RL training without a clear upward trend. This reveals the crucial limitation of the standard GRPO algorithm for improving idea generation. In the next subsection, we analyze why RL from execution reward improves the average reward but not the max.
最大奖励的问题。 尽管成功复现了在其他领域观察到的正向训练曲线,我们主张想法生成与其他可验证领域之间存在区别。推动科学发现时,我们关心的往往是想法生成的上界而非平均质量。就本文而言,我们更想要一条压倒基线的突破性想法,而不是一大批平均分很高的「安全」想法。因此,我们在图 5 下排绘制每个训练轮次全部 rollout 的最大奖励。这里的趋势截然不同——整个 RL 训练过程中最大奖励只是波动,没有明确的上升趋势。这揭示了标准 GRPO 算法在改进想法生成上的关键局限。下一小节我们分析:为什么执行奖励 RL 提升了平均奖励却没有提升最大奖励。
5.3 训练动态分析(Analysis of Training Dynamics)
[图 6: Upper Row: Average length of the thinking trace (red line) and the idea (blue line) per training epoch. Lower Row: Execution rate of ideas with the longest (red line) versus shortest (blue line) thinking traces. Ideas with longer thinking have lower execution rates. Correspondingly, thinking lengths decrease in RL.]
图 6 说明:上排:每个训练轮次思考轨迹(红线)与想法(蓝线)的平均长度。下排:思考轨迹最长(红线)与最短(蓝线)的想法的执行率。思考更长的想法执行率更低;相应地,RL 中思考长度不断下降。
[图 7: We plot how many ideas in each epoch are about either replacing RMSNorm with LayerNorm or doing EMA. The model converged on these two ideas after RL training.]
图 7 说明:我们绘制每个轮次中属于「把 RMSNorm 换成 LayerNorm」或「做 EMA」的想法数量。RL 训练后,模型收敛到这两个想法上。
**Thinking Length** We first plot how the lengths of the thinking traces evolve over RL training in the upper row of Figure 6. In both environments, the thinking traces rapidly decrease in length while the idea lengths stay roughly constant. This is the opposite of the thinking emergence trend from prior RLVR work, such as DeepSeek-R1 (DeepSeek-AI et al., 2025). To further understand why the thinking length decreases, we investigate the correlation between the idea execution rate and the thinking trace length. In the bottom row of Figure 6, for the first 20 epochs of the RL training, we sort all ideas in each epoch by their thinking trace lengths and plot the average execution rate of the top-30% longest thinking ideas (red line) and the bottom-30% shortest thinking ideas (blue line). We see a clear trend where ideas with longer thinking consistently have a lower execution rate. We thus hypothesize that longer thinking correlates with more complex ideas with lower execution rates, leading the model to prefer shorter thinking instead in order to maximize the reward.
思考长度。 我们首先在图 6 上排绘制思考轨迹长度在 RL 训练中的演变。两个环境中,思考轨迹长度迅速下降,而想法长度大致不变。这与此前 RLVR 工作(如 DeepSeek-R1,DeepSeek-AI et al., 2025)中思考涌现的趋势相反。为进一步理解思考长度为何下降,我们研究想法执行率与思考轨迹长度的相关性。在图 6 下排,对 RL 训练的前 20 轮,我们把每轮所有想法按思考轨迹长度排序,绘制最长的 30% 想法(红线)与最短的 30% 想法(蓝线)的平均执行率。可以看到清晰趋势:思考更长的想法执行率持续更低。因此我们假设:更长的思考对应更复杂、执行率更低的想法,模型为最大化奖励而偏好更短的思考。
**Diversity Collapse** Upon manual investigation of all the rollouts being sampled throughout the RL training, we also observed a diversity collapse. Specifically, the model learned to converge on a few simple ideas that can consistently get a positive reward. For example, in the nanoGPT environment, the model learned to converge towards two common ideas: (1) replacing RMSNorm with LayerNorm; and (2) performing exponential moving average (EMA) over intermediate model checkpoints. As shown in Figure 7, out of a batch of 128 sampled ideas per epoch, 51 ideas sampled from Qwen3-30B at epoch 0 are one of the two common ideas above. Towards the end of the RL training, 119 out of 128 sampled ideas at epoch 68 are one of the two common ideas, indicating a severe diversity collapse.
多样性坍缩。 通过人工检查 RL 训练中采样的全部 rollout,我们还观察到多样性坍缩(diversity collapse)。具体而言,模型学会了收敛到少数能持续获得正奖励的简单想法。例如在 nanoGPT 环境,模型收敛到两个常见想法:(1)把 RMSNorm 换成 LayerNorm;(2)对中间模型检查点做指数移动平均(EMA)。如图 7 所示,每轮 128 个采样想法中,Qwen3-30B 在第 0 轮有 51 个属于上述两个常见想法;到 RL 训练临近结束时,第 68 轮 128 个采样想法中有 119 个属于这两个想法,表明多样性坍缩非常严重。
The above analyses reveal that RL causes the models to converge on a few simple-to-implement ideas, accompanied by shrinking thinking lengths. These lead to an increase in the average reward, but do not push the upper-bound due to the lack of exploration. This phenomenon is analogous to mode-collapse observations on other verifiable domains, where the pass@k performance stagnates or even decreases after RL (Yue et al., 2025; Wu et al., 2025). Avoiding such convergence and collapse is an open problem and likely requires new algorithmic interventions beyond standard GRPO, which is beyond the scope of this work. However, we do share several preliminary attempts, including: sampling and appending previous epochs' trajectories into the current epoch's prompt for rollout sampling, adding a weighted length reward in the total reward, and adding a weighted similarity penalty in the total reward. We did not observe clear gains in the initial epochs and thus early-stopped them, but we document all these results in Appendix A.1 to inform future work.
以上分析揭示:RL 使模型收敛到少数易于实现的简单想法,并伴随思考长度缩短。这些带来平均奖励的上升,却因缺乏探索而无法推动上界。这一现象与其他可验证领域中的模式坍缩观察类似——RL 后 pass@k 性能停滞甚至下降(Yue et al., 2025; Wu et al., 2025)。避免这种收敛与坍缩是一个开放问题,可能需要超越标准 GRPO 的新算法干预,这超出本文范围。不过,我们确实分享了若干初步尝试,包括:采样先前轮次的轨迹并附入当前轮次提示中做 rollout 采样;在总奖励中加入加权的长度奖励;在总奖励中加入加权的相似度惩罚。我们在最初若干轮未见明确收益,因此提前停止,但全部结果记录于附录 A.1,以供未来工作参考。
6 相关工作(Related Work)
**AutoML** Our work has deep connections to the AutoML literature. For example, the Neural Architecture Search (NAS) line of work typically defines a constrained set of neural network operators and optimizes for architectures based on validation set performance through reinforcement learning (Zoph & Le, 2017; Zoph et al., 2017) or search (Liu et al., 2018; So et al., 2019). In the modern era, recent works also explored directly using LLMs to propose architecture variants and implement them for validation (Liu et al., 2025; Cheng et al., 2025). Beyond architectures, similar automatic optimizations have been applied to improve hyperparameter tuning (Zhang et al., 2023), discover machine learning algorithms (Real et al., 2020), improve post-training objectives (Lu et al., 2024a), discover better neural network optimizers (Chen et al., 2023), and design agent scaffolds (Hu et al., 2025). Different from this line of work, we tackle automated AI research in a fully open-ended setting without any constraint on the type of ideas. Moreover, our goal is to improve the effectiveness of idea generation, where natural language ideas represent a higher level of abstraction than specific architecture variants or code optimizations.
AutoML。 我们的工作与 AutoML 文献有深厚联系。例如,神经架构搜索(Neural Architecture Search,NAS)系列工作通常定义一组受约束的神经网络算子,并基于验证集性能,通过强化学习(Zoph & Le, 2017; Zoph et al., 2017)或搜索(Liu et al., 2018; So et al., 2019)来优化架构。在现代,近期工作也探索直接用 LLM 提出架构变体并实现以供验证(Liu et al., 2025; Cheng et al., 2025)。架构之外,类似的自动优化还被用于改进超参数调优(Zhang et al., 2023)、发现机器学习算法(Real et al., 2020)、改进后训练目标(Lu et al., 2024a)、发现更好的神经网络优化器(Chen et al., 2023)以及设计 Agent 脚手架(Hu et al., 2025)。与这条研究线不同,我们在完全开放式、对想法类型不作任何约束的设定下处理自动化 AI 研究;而且我们的目标是提升想法生成的有效性——自然语言想法代表着比具体架构变体或代码优化更高的抽象层级。
**LLM-based Research Agents** Recent works have been building LLM-based research agents for accelerating scientific discovery in various domains, including AI research. AI-Scientist (Lu et al., 2024b; Yamada et al., 2025), AI-Researcher (Tang et al., 2025), and Agent Laboratory (Schmidgall et al., 2025) are examples of end-to-end research agents that use LLMs to generate ideas and implement them through carefully designed agent scaffolds. They address open-ended AI research as we do, but do not study how to learn from execution feedback and improve idea effectiveness. On the other hand, on more grounded benchmarks with clear performance metrics such as MLE-Bench (Chan et al., 2025), RE-Bench (Wijk et al., 2024), and ML-Gym (Nathani et al., 2025), various works have explored how to learn from execution feedback through either search (Toledo et al., 2025; Jiang et al., 2025) or RL (Yang et al., 2025b) to optimize performance on these targeted ML engineering tasks. While we also study algorithms for learning from execution feedback, we tackle open-ended research problems like pre-training and post-training rather than ML engineering tasks that heavily depend on feature engineering and hyper-parameter tuning rather than algorithm development.
基于 LLM 的研究智能体。 近期工作一直在构建基于 LLM 的研究智能体,以加速包括 AI 研究在内的各领域科学发现。AI-Scientist(Lu et al., 2024b; Yamada et al., 2025)、AI-Researcher(Tang et al., 2025)与 Agent Laboratory(Schmidgall et al., 2025)是端到端研究智能体的例子:用 LLM 生成想法,并通过精心设计的 Agent 脚手架加以实现。它们与我们一样面对开放式 AI 研究,但不研究如何从执行反馈中学习、提升想法有效性。另一方面,在具有明确性能指标的更接地的基准上——如 MLE-Bench(Chan et al., 2025)、RE-Bench(Wijk et al., 2024)与 ML-Gym(Nathani et al., 2025)——已有多项工作探索通过搜索(Toledo et al., 2025; Jiang et al., 2025)或 RL(Yang et al., 2025b)从执行反馈中学习,以优化这些目标明确的 ML 工程任务。虽然我们也研究从执行反馈中学习的算法,但我们处理的是预训练与后训练这类开放式研究问题,而非那些主要依赖特征工程与超参数调优、而非算法研发的 ML 工程任务。
**AI for Research** Apart from fully end-to-end automated AI research, many works have studied how to use LLMs for specific components of the scientific research pipeline, such as literature review (Asai et al., 2024; L'ala et al., 2023), idea generation (Si et al., 2025b; Wang et al., 2024), data analysis (Majumder et al., 2025; Mitchener et al., 2025), experiment plan generation (Goel et al., 2025), research code execution (Starace et al., 2025; Hua et al., 2025; Tian et al., 2024), and paper reviewing (Liang et al., 2024; Zhu et al., 2025). Our work focuses on automated idea execution and learning from the execution feedback. We consider our work complementary to many of the above works that improve other aspects of the scientific research pipeline.
AI for Research。 除完全端到端的自动化 AI 研究外,许多工作研究了如何把 LLM 用于科研流程的具体环节,如文献综述(Asai et al., 2024; L'ala et al., 2023)、想法生成(Si et al., 2025b; Wang et al., 2024)、数据分析(Majumder et al., 2025; Mitchener et al., 2025)、实验计划生成(Goel et al., 2025)、研究代码执行(Starace et al., 2025; Hua et al., 2025; Tian et al., 2024)与论文评审(Liang et al., 2024; Zhu et al., 2025)。我们的工作聚焦于想法的自动执行与从执行反馈中学习;我们认为本文与上述改进科研流程其他环节的许多工作互为补充。
**Execution Grounding for Code** The idea of learning from execution feedback has been explored in the code generation domain. For example, Zheng et al. (2024) curate data and train models to refine code from either human or execution feedback; Gehring et al. (2025) use end-to-end RL training to teach models to improve code based on execution feedback; Lavon et al. (2025) directly guide code generation with execution signals during inference time. In contrast, our work explores execution grounding for the application of idea generation, where the verification is more complicated and expensive.
代码领域的执行接地。 从执行反馈中学习的思想在代码生成领域已有探索。例如,Zheng et al. (2024) 策划数据并训练模型根据人类反馈或执行反馈修正代码;Gehring et al. (2025) 用端到端 RL 训练模型基于执行反馈改进代码;Lavon et al. (2025) 在推理时直接用执行信号引导代码生成。相比之下,我们的工作探索的是把执行接地用于想法生成这一应用,其验证更复杂、也更昂贵。
7 结论(Conclusion)
In this work, we built a large-scale parallel executor for automatically executing model-generated ideas to verify their effectiveness on open-ended LLM research problems, including both LLM pre-training and post-training. Using this executor as a reward function, we analyzed the effectiveness of execution-guided evolutionary search, where frontier LLMs equipped with a simple evolutionary search scaffold can significantly outperform the baseline solutions. We also benchmarked and revealed the limitations of reinforcement learning with execution rewards, where models tend to converge on simple ideas to improve the average reward but lose diversity and do not improve the upper-bound. Our empirical results demonstrate the feasibility and potential of the automated execution feedback loop and also point out the remaining limitations for future improvement. We hope this work paves the foundation and inspires more efforts towards execution-grounded automated AI research.
在本工作中,我们构建了一个大规模并行执行器,用于自动执行模型生成的想法,以在开放式 LLM 研究问题(涵盖 LLM 预训练与后训练)上验证其有效性。以该执行器作为奖励函数,我们分析了执行引导进化搜索的有效性——配备简单进化搜索脚手架的前沿 LLM 能显著超越基线方案;我们也对执行奖励强化学习做了基准测试并揭示其局限——模型倾向收敛到简单想法以提升平均奖励,却失去多样性、无法提升上界。我们的实证结果证明了自动执行反馈回路的可行性与潜力,也指出了未来改进尚存的局限。我们希望这项工作为执行接地的自动化 AI 研究奠定基础,并激励更多努力。
8 讨论(Discussion)
Despite encouraging initial signals, there are still many limitations to our current set of experiments, which would be great directions for future improvement.
尽管有令人鼓舞的初步信号,我们目前的系列实验仍存在许多局限,它们也是未来改进的绝佳方向。
First, our current procedure does not test the generalizability of the generated ideas. It is possible that the best-performing ideas at the small scales may not transfer to gains at a larger scale or on other datasets. Future work should explore methods that explicitly test such generalizability and scalability, and even incorporate them as part of the optimization objectives.
第一,当前流程未检验生成想法的泛化性(generalizability)。小规模上表现最好的想法,未必能带来更大规模或其他数据集上的收益。未来工作应探索明确检验这种泛化性与可扩展性的方法,甚至把它们纳入优化目标的一部分。
Second, we have shown that RL with execution reward in our current setup can only improve the average reward but not the upper bound. There are many possible reasons, such as a lack of diversity in the base model and missing exploration incentives in the current RL objective. Future work should explore remedies and better learning algorithms for LLMs to more efficiently learn from the execution feedback. For instance, future works could explore how to exploit richer learning signals from the execution trajectories beyond just scalar rewards.
第二,我们已表明,当前设定下的执行奖励 RL 只能提升平均奖励而不能提升上界。可能的原因很多,例如基模型缺乏多样性、当前 RL 目标缺少探索激励。未来工作应探索补救措施与更好的学习算法,使 LLM 能更高效地从执行反馈中学习。例如,未来工作可以探索如何从执行轨迹中利用超越标量奖励的更丰富学习信号。
Third, our current experiment scope is bounded by the capability of the execution agent. There exist many promising model-generated ideas that could not be successfully executed by the execution agent (e.g., see the end of Appendix A.2), leading to noise in the reward signal. Future work could develop more capable execution agents and extend our setup to even more complex research problems. For instance, instead of directly prompting an LLM for code diff, future work can implement more capable coding agents with access to external tools and the ability to install new libraries in the execution environments.
第三,当前实验范围受执行智能体(execution agent)能力所限。存在许多有前景的模型生成想法未能被执行智能体成功执行(例如见附录 A.2 末尾),从而给奖励信号引入噪声。未来工作可以开发更强的执行智能体,并把我们的设定扩展到更复杂的研究问题。例如,不直接提示 LLM 生成代码 diff,未来工作可以实现能访问外部工具、能在执行环境中安装新库的更强编码智能体。
Last but not least, we only explored effectiveness as the training reward in this work. There are many other, more subjective alternative metrics that could complement the effectiveness reward, such as the idea novelty and interestingness. Future work could explore how to computationally measure them and incorporate them as part of the training objective to discover more insightful ideas.
最后,本文只探索了把有效性作为训练奖励。还有许多更主观的替代指标可以补充有效性奖励,例如想法的新颖性(novelty)与有趣度(interestingness)。未来工作可以探索如何以计算方式度量它们,并将其纳入训练目标,以发现更具洞见的想法。
致谢(Acknowledgment)
We thank DST Global, Laude Institute, and Thinking Machines Lab for their generous sponsorship of computing resources. We thank Yuandong Tian, Edward Hughes, Ludwig Schmidt, Cong Lu, Jenny Zhang, Jiaxin Wen, Chris Rytting, Chen Zhao, Xinran Zhao, Yanzhe Zhang, John Yang, Shicheng Liu, Andy Zhou, Will Held, Haotian Ye, Luke Bailey, as well as members of Tatsu Lab and SALT Lab for their helpful discussion and feedback. This work was supported by an HAI grant, DSO labs, gifts from Open Philanthropy, Amazon, Schmidt Sciences, the Tianqiao and Chrissy Chen Foundation and a grant under the NSF CAREER IIS-2338866 and IIS-2247357, ONR N00014-24-1-2609 and N00014-24-1-2532, and DARPA Cooperative Agreement HR00112520013. This work does not necessarily reflect the position or policy of the government and no official endorsement should be inferred.
我们感谢 DST Global、Laude Institute 与 Thinking Machines Lab 慷慨赞助算力资源。我们感谢 Yuandong Tian、Edward Hughes、Ludwig Schmidt、Cong Lu、Jenny Zhang、Jiaxin Wen、Chris Rytting、Chen Zhao、Xinran Zhao、Yanzhe Zhang、John Yang、Shicheng Liu、Andy Zhou、Will Held、Haotian Ye、Luke Bailey,以及 Tatsu Lab 与 SALT Lab 成员的有益讨论与反馈。本工作受 HAI 资助、DSO labs 资助,以及 Open Philanthropy、Amazon、Schmidt Sciences、天桥与 Chrissy Chen 基金会的捐助,并受 NSF CAREER IIS-2338866 与 IIS-2247357、ONR N00014-24-1-2609 与 N00014-24-1-2532、DARPA 合作协议 HR00112520013 的支持。本工作未必反映政府的立场或政策,不应推断任何官方背书。
(References 部分为文献条目,从略,见文首译注。)
附录 A(Appendix A)
A.1 其他 RL 尝试(Other RL Attempts)
We present several attempts to improve our RL from the execution reward recipe.
我们给出若干改进「执行奖励 RL」方案的尝试。
**Attempt 1: Dynamic Prompt** At each epoch (except the first epoch), we randomly sample different executed idea trajectories from the previous epoch and append them to the idea sampling prompt when sampling new rollouts. This merges in-context learning with RL and adds diversity to the idea sampling process. We present the experiment results on the GRPO environment in Figure 8. We did not see significant improvement in early epochs and thus early stopped.
尝试 1:动态提示(Dynamic Prompt)。 除第一轮外,每一轮都从上一轮随机抽取不同的已执行想法轨迹,在采样新 rollout 时附入想法采样提示。这把上下文学习(in-context learning)与 RL 结合,为想法采样过程增加多样性。GRPO 环境上的实验结果见图 8。我们在早期轮次未见显著改进,因此提前停止。
[图 8: RL with dynamic prompt by appending trajectories from previous epochs. Results are on the GRPO environment.]
图 8 说明:带动态提示(附入先前轮次轨迹)的 RL。结果在 GRPO 环境上。
**Attempt 2: Length Reward** Since we noted a rapid thinking length decrease in our main RL experiment, we tried a simple fix by adding a weighted length reward that counts the number of tokens in the entire rollout sequence, including the thinking trace and the idea. We cap the length reward to a maximum of 0.3 to avoid it dominating the accuracy reward. We present the experiment results on the GRPO environment in Figure 9. As shown on the right panel, the thinking length no longer decreases after adding the length reward to the total reward, but the total training reward isn't going up as shown on the left panel.
尝试 2:长度奖励(Length Reward)。 由于在主 RL 实验中注意到思考长度迅速下降,我们尝试了一个简单的修正:加入加权的长度奖励,统计整条 rollout 序列(含思考轨迹与想法)的 token 数。我们把长度奖励上限设为 0.3,以避免其压过准确率奖励。GRPO 环境上的实验结果见图 9。如右图所示,把长度奖励加入总奖励后,思考长度不再下降;但如左图所示,总训练奖励不再上升。
[图 9: Average and max reward throughout RL training when adding the length reward (left), as well as the progression of the thinking and idea lengths (right).]
图 9 说明:加入长度奖励后 RL 训练全程的平均与最大奖励(左),以及思考与想法长度的变化(右)。
**Attempt 3: Diversity Reward** We also tried adding a diversity reward in addition to the effectiveness reward. Specifically, when computing the reward for each rollout, we compute its token-level Jaccard similarity with ideas from the previous epoch and add the negative similarity as a penalty to the total reward to discourage repeating ideas that have already been generated before. In fact, this is similar to one of the post-training ideas proposed by Claude-4.5-Sonnet (see example 4 in Appendix A.3). We show the training curves on the GRPO environment in Figure 10. The model maintains a consistent idea similarity (right plot), suggesting sustained exploration. The effectiveness reward is generally showing an upward trend (left plot), but not markedly better than the main RL run with just the effectiveness reward (first sub-plot in Figure 5).
尝试 3:多样性奖励(Diversity Reward)。 我们还尝试在有效性奖励之外加入多样性奖励。具体地,计算每条 rollout 的奖励时,计算它与上一轮想法的 token 级 Jaccard 相似度,并把负相似度作为惩罚加入总奖励,以抑制重复先前已生成的想法。事实上,这与 Claude-4.5-Sonnet 提出的一个后训练想法类似(见附录 A.3 示例 4)。GRPO 环境上的训练曲线见图 10。模型的想法相似度保持稳定(右图),说明探索得以维持;有效性奖励总体呈上升趋势(左图),但并不明显好于仅用有效性奖励的主 RL 实验(图 5 第一个子图)。
[图 10: Effectiveness reward (left) and average idea similarity to previous epoch (right) when doing RL with diversity reward.]
图 10 说明:带多样性奖励做 RL 时的有效性奖励(左)与相对上一轮的平均想法相似度(右)。
A.2 更多想法示例(Additional Idea Examples)
We provide several additional example ideas generated by Claude-4.5-Opus (Table 4) and Claude-4.5-Sonnet (Table 5) on the GRPO environment, including ideas with failed code execution. In most cases, code execution errors happen when the idea involves complicated changes or installing and importing external packages not supported in our execution environment. Future work should explore improvement to the execution agent so that more complicated types of ideas (e.g., training additional auxiliary models or system-level optimizations) can be implemented correctly.
我们给出 GRPO 环境上由 Claude-4.5-Opus(表 4)与 Claude-4.5-Sonnet(表 5)生成的更多示例想法,包括代码执行失败的想法。多数情况下,代码执行错误出现在想法涉及复杂改动、或需要安装并导入我们的执行环境不支持的第三方包时。未来工作应探索改进执行智能体,使更复杂的想法类型(如训练额外的辅助模型或系统级优化)能被正确实现。
译注:以下表 4 / 表 5 及本节各「最优想法」中的 [Code Changes] 均为逐条代码改动指令(改哪个文件、哪一行、写成什么代码),篇幅极长且属纯代码细节,此处仅译出实验想法并概述代码改动要点;完整原文见论文附录 A.2。
表 4:Additional examples on the GRPO environment. Ideas are generated by Claude-4.5-Opus during evolutionary search.
| 成功执行 | 执行失败 |
|---|---|
| 序列位置加权信赖域(Sequence Position Weighted Trust Region):对序列中较早的 token(误差累积之处)施加更紧的 sigmoid 约束、较晚的 token 施加更松的约束。权重:position_weight = 1 − 0.3 × (position / seq_len),effective_deviation = 0.25 + 0.2 × position_weight。这考虑了自回归生成的顺序特性。[代码改动要点] 修改 grpo.py(初始化 current_cliprange=0.2、ema_clip_fraction=0.15,标准动量裁剪更新);修改 grpo_utils.py 的 compute_grpo_clip_loss:计算比率(batch_size × seq_len)后按位置算出权重与有效偏差,做位置感知的 sigmoid 约束(centered_ratio = ratio − 1.0,bounded_ratio = 1.0 + (2.0 × sigmoid(centered_ratio) − 1.0) × effective_deviation),再按 surr1/surr2 取最小值计算损失,并记录 mean_effective_deviation 等元数据。验证准确率:59.8 | 分层位置-组信赖域(Hierarchical Position-Group Trust Region):在两个层级施加信赖域——组级(同一响应组内共享)与位置级(沿序列变化)。组内奖励方差高的组获得更紧的组级约束;组内位置沿用已验证的衰减模式。这同时捕捉跨样本与样本内的结构。公式:group_dev = 0.4 − 0.15 × tanh(group_reward_var / 0.3),position_factor = 1 − 0.2 × rel_pos,effective_dev = group_dev × position_factor。[代码改动要点] 修改 grpo.py(初始化同左,并把 group_size 传入函数);修改 grpo_utils.py 的 compute_grpo_clip_loss(新增参数 group_size=8):由优势按组计算方差作为组奖励方差的代理,计算组级偏差与位置因子得到分层有效偏差并裁剪到 [0.15, 0.45],再按上式做有界比率与损失,记录元数据并上传 wandb。 |
表 4 说明:GRPO 环境上的更多示例。想法由 Claude-4.5-Opus 在进化搜索期间生成。
表 5:Additional examples on the GRPO environment. Ideas are generated by Claude-4.5-Sonnet during evolutionary search.
| 成功执行 | 执行失败 |
|---|---|
| 创建数学步骤复杂度感知的奖励塑形:答对时,推理步骤更多的响应获得略高的基础奖励(3+ 步 1.05 倍、5+ 步 1.1 倍),在不改变核心二元奖励结构的前提下鼓励充分的数学论述。[代码改动要点] 修改 drgrpo_grader.py 的 r1_zero_reward_fn_train:通过检测数学过渡词("therefore"、"thus"、"so"、"="、"=>")统计推理步骤,答对时按 step_multiplier = 1.0 + 0.05 × min(2, max(0, num_steps − 2)) 施加乘数;设学习率 3e-5、损失类型 reinforce_with_baseline。验证准确率:65.6 | 实现时序差分优势估计:优势不仅包含当前奖励,还包含用学习到的价值函数预测的未来奖励,配合已验证的 3e-5 学习率与 reinforce_with_baseline,实现更准确的功劳分配。[代码改动要点] 在 grpo.py 中为策略模型添加预测未来奖励的价值头;按 advantages = rewards + gamma × next_values − current_values(gamma=0.99)计算 TD 优势;用观测回报上的 MSE 损失训练价值函数;让 compute_group_normalized_rewards 改用 TD 优势;设学习率 3e-5、损失类型 reinforce_with_baseline。 |
| 通过维护先前各轮奖励最高的前 20% rollout 的缓冲区实现经验回放,训练时按 25% 回放 / 75% 新样本混合,配合 3e-5 学习率与 reinforce_with_baseline 提升样本效率。[代码改动要点] 修改 grpo.py 的 train_loop:维护存储高奖励(>0.8)rollout 的 replay_buffer;每轮 25% 训练数据取自回放缓冲、75% 取自新 rollout;缓冲超过 1000 条时淘汰最旧样本;设学习率 3e-5、损失类型 reinforce_with_baseline。验证准确率:39.4 | 集成决策训练与投票共识:以集成式决策训练模型,每条 rollout 生成多个候选响应,最终训练信号基于响应间的多数投票。这鼓励模型发展更稳健、一致的推理模式,同时保持解法的多样性。[代码改动要点] 修改 sample.py 的 sample_rollout,用不同随机种子为每个提示生成 3 条响应;在 r1_zero_reward_fn_train 中实现投票共识(2+ 条正确给 +0.15 共识奖励,响应不一致给 −0.05 不确定性惩罚);在 train_loop 中选得票最高的响应训练,并用共识率调节学习率 consensus_lr = 3e-5 × (0.9 + 0.2 × consensus_rate);设 group_size=6、损失类型 reinforce_with_baseline。 |
| 实现组内响应多样性奖励:同一提示的响应若与组内其他响应不相似,则获得额外奖励(0.05–0.15),在保持已验证的 group_size=8 与 3e-5 学习率组合的同时鼓励探索不同解题路径。[代码改动要点] 修改 grpo_utils.py 的 compute_group_normalized_rewards:用 token 级 Jaccard 相似度计算组内响应两两相似度,在优势计算前为每个响应的奖励加上多样性加成 diversity_reward = 0.15 × (1 − max_similarity_in_group);设学习率 3e-5、损失类型 reinforce_with_baseline、group_size 8。验证准确率:19.2 | 实现分层优势估计:在 token 级与序列级两个层面计算优势,token 级优势按位置重要性加权(数学表达式与最终答案的权重更高),配合成功的 3e-5 学习率与 reinforce_with_baseline。[代码改动要点] 修改 grpo_utils.py 的 grpo_microbatch_train_step:构造位置重要性权重(含数学符号(\frac、+、−、*、=)的 token 给 2.0 倍权重、答案部分给 1.5 倍),分别计算序列级优势(现有)与 token 级优势,按 final_advantages = 0.6 × sequence_advantages + 0.4 × token_advantages 合并;设学习率 3e-5、损失类型 reinforce_with_baseline。 |
表 5 说明:GRPO 环境上的更多示例。想法由 Claude-4.5-Sonnet 在进化搜索期间生成。
We also present the top-performing ideas from Claude-4.5-Opus, Claude-4.5-Sonnet, and GPT-5 on the nanoGPT environment below:
下面我们还给出 Claude-4.5-Opus、Claude-4.5-Sonnet 与 GPT-5 在 nanoGPT 环境上表现最好的想法:
Claude-4.5-Opus 在 nanoGPT 上的想法(验证损失:3.1407)
[Experiment] Wider SwiGLU (5x) with MLP Output Scaling (Init 0.97), Skip Connections Every 4 and 8 Layers with Learnable Weights (Init 0.52 and 0.31), Separate Attention/MLP Scales (Init 0.98), Higher LR (0.00168), Reduced Weight Decay (0.065), Warmup 173 iters, Lower Min LR (0.03x), Cosine Annealing, EMA, Untied Embeddings, and Beta2=0.99
Make the dual skip connection weights learnable parameters initialized at proven good values. This allows the model to adapt skip weights during training while combining with separate attention/MLP residual scales.
[实验] 5 倍宽 SwiGLU 并加 MLP 输出缩放(初始 0.97)、每 4 层与每 8 层各加带可学习权重(初始 0.52 与 0.31)的跳跃连接、注意力/MLP 各自独立的缩放(初始 0.98)、更高学习率(0.00168)、更小权重衰减(0.065)、warmup 173 步、更低的最小学习率(0.03x)、余弦退火、EMA、解绑的输入-输出嵌入(untied embeddings)与 beta2=0.99。
把双重跳跃连接权重设为可学习参数,并初始化为已验证的优良值;这使模型能在训练中自适应调整跳跃权重,并与独立的注意力/MLP 残差缩放相结合。[代码改动要点] warmup_iters 256→173、weight_decay 0.1→0.065、learning_rate 0.0015→0.00168;在 GPT/Block/MLP 中新增可学习参数 skip_weight_4=0.52、skip_weight_8=0.31、attn_scale=mlp_scale=0.98、output_scale=0.97;MLP 改为 5 倍宽的 SwiGLU;移除输出端的权重绑定;优化器 betas 改为 (0.9, 0.99);学习率调度改为 warmup 后余弦退火至峰值的 0.03 倍;前向传播中每 4/8 层加入加权的层间跳跃连接;维护 ema_decay=0.999 的 EMA 模型,验证前切换到 EMA 权重、验证后恢复原权重。
Claude-4.5-Sonnet 在 nanoGPT 上的想法(验证损失:3.2081)
[Experiment] Two-phase weight decay (0.1170→0.0210 at 59.65%) + 30.45% plateau + LR 0.001550 + warmup 197 + two-phase grad clip (1.054→0.916 at 59.65%) + quadratic min_lr interpolation (0.0113x at 59.65%, 0.0075x at end via quadratic) + progressive EMA (0.999→0.9992 linear over training) + exponential warmup + cosine LR + beta2=0.99
Use smooth quadratic interpolation for min_lr during low-WD phase AND progressive EMA that gradually increases from 0.999 to 0.9992 linearly throughout training. Early training benefits from faster EMA tracking, while later training gets heavier smoothing. Use conservative settings: WD 0.1170/0.0210, extended plateau 30.45%, moderate LR 0.001550, longest warmup 197, tight grad clip 1.054→0.916.
[实验] 两阶段权重衰减(0.1170→0.0210,切换点 59.65%)+ 30.45% 平台期 + 学习率 0.001550 + warmup 197 步 + 两阶段梯度裁剪(1.054→0.916,切换点 59.65%)+ 二次插值的最小学习率(59.65% 处 0.0113x,结束时经二次插值到 0.0075x)+ 渐进式 EMA(0.999→0.9992 随训练线性增长)+ 指数 warmup + 余弦学习率 + beta2=0.99。
在低权重衰减阶段对 min_lr 使用平滑二次插值,并采用随训练从 0.999 线性升至 0.9992 的渐进式 EMA:训练早期受益于更快的 EMA 跟踪,后期获得更重的平滑。使用保守设置:权重衰减 0.1170/0.0210、延长平台期 30.45%、适中学习率 0.001550、最长 warmup 197、紧的梯度裁剪 1.054→0.916。[代码改动要点] 改 warmup_iters=197、learning_rate=0.001550、betas=(0.9, 0.99);重写 get_lr:指数 warmup、30.45% 训练步的平台期、以 59.65% 为界的两阶段二次插值 min_lr 因子(0.0113→0.0075)再加余弦衰减;按训练进度在 0.916/1.054 之间切换梯度裁剪、在 0.0210/0.1170 之间切换权重衰减;维护随进度 ema_decay = 0.999 + 0.0002 × progress 的 EMA 参数,验证前载入 EMA、验证后恢复原参数。
GPT-5 在 nanoGPT 上的想法(验证损失:3.1697)
[Experiment] SwiGLU-3.5x + Residual Alphas + Min-Floor Cosine + Per-step Beta2 Linear Decay + 3-Group AdamW + Debiased EMA
[实验] SwiGLU-3.5 倍 + 残差 alpha + 带下限的余弦调度 + 每步 beta2 线性衰减 + 三组 AdamW + 去偏 EMA。[代码改动要点] 超参数:hidden_factor=3.5、warmup_iters=256、lr_peak_factor=1.10、min_lr_factor=0.02、beta2 从 0.99 起步、beta2_end=0.95、wd_decay=0.1、wd_embed=0.01、ema_decay=0.9995、ema_warmup_steps=256;MLP 用 SwiGLU、Block 的 alpha 初始 0.9;优化器用三组 AdamW;学习率 warmup 至峰值后余弦衰减到下限;每步设置学习率后更新 beta2 = beta2_start + (beta2_end − beta2_start) × min(1.0, (it+1)/num_iterations) 并更新全部参数组;EMA 在评估时除以 (1 − ema_decay^ema_step) 去偏,评估后恢复。
While the best-performing ideas on nanoGPT tend to be heavily optimized with extensive hyper-parameter tuning mixed with various architecture tweaks, we also pick a few more "atomic" algorithmic ideas that are successfully executed.
虽然 nanoGPT 上表现最好的想法往往是大量超参数调优与各种架构微调混杂的重度优化结果,我们也挑选了几个成功执行的更「原子化」的算法想法。
Claude-4.5-Opus 的示例
- 逐头注意力输出缩放(Head-Wise Attention Output Scaling):为注意力加入逐头的可学习缩放因子,允许不同的头以不同强度贡献到输出。验证损失:3.2386
- 可学习残差连接权重(Learned Residual Connection Weights):为每条残差连接加入初始化为 1.0 的可学习标量权重,让模型在训练中学到最优的残差缩放。验证损失:3.2517
- 带位置混合的嵌入(Mixture of Embeddings with Position):用一个依赖内容的权重学习混合 token 嵌入与位置嵌入,让模型按 token 动态平衡位置与语义信息。验证损失:3.2497
- 带学习非对称性的共享输入-输出嵌入(Shared Input-Output Embedding with Learned Asymmetry):保留权重绑定,但在输出侧加一个小的可学习变换,兼顾权重绑定的好处与输出端特有的适配。验证损失:3.2499
- 门控最终归一化(Gated Final Normalization):把 lm_head 前的最终 RMSNorm 换成门控版本,由学习到的门控制应用多少归一化、多少直接透传原始表示。验证损失:3.2503
- 位置感知 MLP 门控(Position-Aware MLP Gating):依据位置信息门控 MLP 输出,让模型学习依赖位置的处理深度。验证损失:3.2506
- 分组 token 嵌入(Grouped Token Embeddings):把词表聚成簇,在 token 嵌入之上为每簇加一个可学习嵌入,提供层次化的词表结构。验证损失:3.2521
(译注:「可学习残差连接权重」一条在提取文本中重复出现两次,此处仅列一次。)
Lastly, we present several interesting ideas on the nanoGPT environment that didn't get successfully executed. These examples are generated by Claude-4.5-Opus.
最后,我们给出 nanoGPT 环境上几条有趣但未能成功执行的想法。这些示例由 Claude-4.5-Opus 生成。
- 软层重复(Soft Layer Repetition):通过加入一个学习到的门,把当前层的输入混回其输出,让模型能在层间「软性地」重复计算,模拟可变深度。
- 因果上下文压缩(Causal Context Compression):在每个注意力层之前,施加一个学习到的压缩,把局部上下文(前 2–3 个 token)混入当前表示,在不使用卷积的情况下提供隐式局部上下文。
- 经正交损失的注意力头特化(Attention Head Specialization via Orthogonal Loss):加一个软惩罚,通过惩罚不同头的输出之间的相似性,鼓励不同的注意力头关注不同的模式。
- 带可学习残差权重的跳跃连接(Skip Connections with Learned Residual Weights):把跳跃连接与可学习残差权重结合:跳跃连接提供替代的梯度路径,可学习权重允许自适应缩放。
- token 难度感知的损失加权(Token Difficulty-Aware Loss Weighting):根据模型在每个位置的不确定性(熵)加权该 token 的损失贡献,把学习集中在困难 token 上,不过度优化简单 token。
A.3 代码执行示例(Code Execution Examples)
We present a few ideas with their full code execution to demonstrate the full end-to-end trajectories. All examples below are from Claude-4.5-Sonnet on the GRPO environment. For each example, we first present the natural language idea, followed by the code implementation generated by Claude-4.5-Sonnet.
我们给出几个想法及其完整代码执行,以展示完整的端到端轨迹。以下所有示例均来自 GRPO 环境上的 Claude-4.5-Sonnet。每个示例先给出自然语言想法,随后是 Claude-4.5-Sonnet 生成的代码实现。
译注:以下各示例的完整代码 diff(每个数百行,逐文件、逐行列出改动)从略,仅译出自然语言想法与代码改动描述;代码 diff 全文见论文附录 A.3。示例 1、3、4 的想法分别与表 3、表 5 中的相应条目相同。
示例 1(验证准确率:58.0)
[Experiment] Create mathematical working memory simulation by maintaining a context buffer of mathematical facts, definitions, and intermediate results during problem solving. This buffer gets updated as the model works through problems and provides additional context for subsequent mathematical steps, simulating how humans maintain mathematical working memory during complex calculations.
[Code Changes] Modify the prompt template in sample.py to include a "Mathematical Context" section that accumulates relevant mathematical facts during problem solving. Implement context buffer management in train_loop in grpo.py that maintains recent mathematical definitions, formulas, and intermediate calculations across problems in the same batch. Update the buffer with key mathematical expressions extracted from each response. Use enhanced prompts that include relevant context from the buffer. Set -learning_rate 2.5e-5 and -loss_type reinforce_with_baseline.
[实验] 创建数学工作记忆模拟:在解题过程中维护一个由数学事实、定义与中间结果组成的上下文缓冲区。该缓冲区随模型解题不断更新,并为后续数学步骤提供额外上下文,模拟人类在复杂计算中维持数学工作记忆的方式。
[代码改动] 修改 sample.py 中的提示模板,加入在解题过程中累积相关数学事实的「数学上下文(Mathematical Context)」部分;在 grpo.py 的 train_loop 中实现上下文缓冲区管理,在同一批题目之间维护近期的数学定义、公式与中间计算;用从每条响应中提取的关键数学表达式更新缓冲区;使用包含缓冲区相关上下文的增强提示;设学习率 2.5e-5、损失类型 reinforce_with_baseline。(完整代码 diff 从略)
示例 2(验证准确率:56.2)
[Experiment] Implement mathematical solution robustness training through systematic perturbation testing that teaches the model to solve mathematically equivalent problems with varied presentations, notation styles, and problem phrasings. Combine this with proven reward shaping by creating robustness-aware rewards that encourage mathematical understanding that generalizes across problem variations.
[Code Changes] Create problem perturbation system in sample.py that generates equivalent mathematical problems with varied notation, different variable names, alternative problem phrasing, and equivalent mathematical formulations. Track solution consistency across perturbations. Enhance r1_zero_reward_fn_train in drgrpo_grader.py to reward robustness: robustness_bonus = 0.12 * consistency_score_across_perturbations where consistency measures solution method stability across equivalent problems. Apply generalization bonuses: +0.08 for solving problems with unfamiliar notation, +0.06 for handling varied problem presentations. Modify train_loop in grpo.py to include perturbation-based training batches and track robustness improvement. Use robustness-aware learning rate with cosine annealing from 3.4e-5 to 1.8e-5. Implement perturbation-complexity gradient clipping. Set -loss_type reinforce_with_baseline.
[实验] 通过系统化的扰动测试实现数学解法稳健性训练:让模型学会解数学等价但表述、记号风格与题面措辞不同的问题;并结合已验证的奖励塑形,创建鼓励跨问题变体泛化的数学理解的稳健性感知奖励。
[代码改动] 在 sample.py 中创建问题扰动系统,生成记号不同、变量名不同、题面表述不同、数学表述等价的问题;跟踪跨扰动的一致性;增强 drgrpo_grader.py 中的 r1_zero_reward_fn_train 以奖励稳健性:robustness_bonus = 0.12 × consistency_score_across_perturbations,其中一致性度量等价问题之间解题方法的稳定性;施加泛化加成:解决不熟悉记号的问题 +0.08、处理多样题面 +0.06;修改 grpo.py 的 train_loop,纳入基于扰动的训练批次并跟踪稳健性改进;使用从 3.4e-5 余弦退火到 1.8e-5 的稳健性感知学习率;实现扰动-复杂度梯度裁剪;设损失类型 reinforce_with_baseline。(完整代码 diff 从略)
示例 3(验证准确率:45.2)
[Experiment] Implement token-level reward attribution by using attention weights to identify which input tokens contributed most to correct answers, then amplifying the gradient updates for those tokens during policy gradient training.
[Code Changes] Modify get_response_log_probs in utils.py to also return attention weights from the last layer. In grpo_microbatch_train_step, compute token importance scores by averaging attention weights across heads, then multiply the policy gradient loss by (1 + importance_score) for tokens with high attention to mathematical content.
[实验] 实现 token 级奖励归因:用注意力权重识别哪些输入 token 对答对贡献最大,然后在策略梯度训练中放大这些 token 的梯度更新。
[代码改动] 修改 utils.py 的 get_response_log_probs,使其同时返回最后一层的注意力权重;在 grpo_microbatch_train_step 中,把注意力权重跨头平均以计算 token 重要性分数,再对数学内容注意力高的 token 把策略梯度损失乘以 (1 + importance_score)。(完整代码 diff 从略)
示例 4(验证准确率:19.2)
[Experiment] Implement response diversity rewards within groups where responses to the same prompt receive bonus rewards (0.05-0.15) for being dissimilar to other responses in their group, encouraging exploration of different solution paths while maintaining the proven group_size=8 and 3e-5 learning rate combination.
[Code Changes] Modify compute_group_normalized_rewards in grpo_utils.py to compute pairwise similarity between responses in each group using token-level Jaccard similarity. Add diversity bonus: diversity_reward = 0.15 * (1 - max_similarity_in_group) to each response's reward before advantage computation. Reshape responses into groups, compute similarities, and add bonuses before advantage normalization. Set -learning_rate 3e-5, -loss_type reinforce_with_baseline, -group_size 8.
[实验] 实现组内响应多样性奖励:同一提示的响应若与组内其他响应不相似,则获得额外奖励(0.05–0.15),在保持已验证的 group_size=8 与 3e-5 学习率组合的同时鼓励探索不同解题路径。
[代码改动] 修改 grpo_utils.py 的 compute_group_normalized_rewards,用 token 级 Jaccard 相似度计算组内响应两两相似度;在优势计算前,为每个响应的奖励加上多样性加成 diversity_reward = 0.15 × (1 − max_similarity_in_group);把响应重组为组、计算相似度并在优势归一化前加上加成;设学习率 3e-5、损失类型 reinforce_with_baseline、group_size 8。(完整代码 diff 从略)
要点速览
- 构建「自动化 AI 研究员」的关键瓶颈:LLM 生成的想法常看似合理但执行后无效,因此需要执行接地(execution grounding)。
- 系统三模块:Implementer(想法→代码 diff,并行采 10 个、最多 2 次自我修正)、Scheduler(资源调度)、Worker(GPU 集群执行并上传 wandb 日志)。
- 两个执行环境:nanoGPT 预训练(124M GPT-2、FineWeb、8×H100、固定 25 分钟、代理奖励 1/loss)与 GRPO 后训练(Qwen2.5-Math-1.5B、MATH 数据集、验证准确率),均冻结评估防奖励欺骗。
- 可行性:预训练环境上 Claude-4.5-Sonnet/Opus 执行率超 90%;best-of-50 已能超基线(Sonnet 60.4% vs 48.0%;Opus 损失 3.237 vs 3.255);换 GPT-5 做执行器时完成率 42%–78%。
- 进化搜索结果:10 轮内 GRPO 69.4% vs 基线 48.0%,超过 CS336 课程最佳学生方案 68.8%;nanoGPT 19.7 分钟 vs 35.9 分钟,但距人类 speedrun 最优 2.1 分钟仍有差距。
- 搜索设计:利用(基于优于基线的想法生成变体)+ 探索(生成全新想法)按 50/50 起步并退火;同预算下显著优于 best-of-N。
- 想法构成:三个模型都大量生成算法性想法(如 GRPO 上算法类占比 58.9%–96.3%),多数情况最有效想法来自算法类;模型还自发复现了近期论文中的想法。
- RL(Qwen3-30B-A3B + GRPO、组大小 256/128、对应 256/1024 张 GPU)提升平均(GRPO 平均准确率 0.253→0.343)但不提升最大奖励。
- RL 的病根:思考长度持续缩短、多样性坍缩(nanoGPT 上两个常见想法从第 0 轮 51/128 占比升至第 68 轮 119/128);长度奖励、多样性奖励等初步尝试均无明确收益。
- 局限与展望:未验证想法跨规模/跨数据集的泛化;执行智能体能力不足带来奖励噪声;可引入新颖性等更丰富的训练信号。
🖼 原文图表(13 张)—— 提取自原始文献,点击展开


以上图片提取自原始 PDF/网页或 arXiv 官方 HTML 版(含矢量图与原始图注),已镜像至本站;与正文的对应关系请以原文为准。