【原】LRMs：《Beyond ‘Aha!‘: Toward Systematic Meta-Abilities Alignment in Large Reasoning Models》翻譯與解讀

地址

論文地址：[2505.10554] Beyond 'Aha!': Toward Systematic Meta-Abilities Alignment in Large Reasoning Models

時間

2025年5月15日

作者

新加坡國立大學

清華大學

Salesforce人工智能研究

Large reasoning models (LRMs) already possess a latent capacity for long chain-of-thought reasoning. Prior work has shown that outcome-based reinforcement learning (RL) can incidentally elicit advanced reasoning behaviors such as self-correction, backtracking, and verification phenomena often referred to as the model's "aha moment". However, the timing and consistency of these emergent behaviors remain unpredictable and uncontrollable, limiting the scalability and reliability of LRMs' reasoning capabilities. To address these limitations, we move beyond reliance on prompts and coincidental "aha moments". Instead, we explicitly align models with three meta-abilities: deduction, induction, and abduction, using automatically generated, self-verifiable tasks. Our three stage-pipeline individual alignment, parameter-space merging, and domain-specific reinforcement learning, boosting performance by over 10\% relative to instruction-tuned baselines. Furthermore, domain-specific RL from the aligned checkpoint yields an additional 2\% average gain in the performance ceiling across math, coding, and science benchmarks, demonstrating that explicit meta-ability alignment offers a scalable and dependable foundation for reasoning. Code is available at: this https URL

大型推理模型（LRMs）已經(jīng)具備了潛在的長鏈推理能力。先前的研究表明，基于結果的強化學習（RL）能夠偶然引發(fā)諸如自我糾正、回溯和驗證等高級推理行為，這些現(xiàn)象通常被稱為模型的“頓悟時刻”。然而，這些新興行為的出現(xiàn)時間和一致性仍然難以預測和控制，這限制了 LRMs 推理能力的可擴展性和可靠性。為了解決這些局限性，我們不再依賴提示和偶然出現(xiàn)的“頓悟時刻”。相反，我們通過自動生成的、可自我驗證的任務，明確地將模型與三種元能力——演繹、歸納和溯因——對齊。我們的三階段流水線——個體對齊、參數(shù)空間合并和領域特定的強化學習，相對于指令調(diào)優(yōu)的基線，性能提升了超過 10%。此外，從對齊的檢查點獲取的特定領域的強化學習在數(shù)學、編程和科學基準測試中的性能上限平均提高了 2%，這表明明確的元能力對齊為推理提供了一個可擴展且可靠的基石。代碼可在以下網(wǎng)址獲取：this https URL

Large reasoning models, including OpenAI-o1?[11], o3?[17], DeepSeek-R1?[8], Grok 3.5?[27], and Gemini 2.5 Pro?[3], have demonstrated remarkable capabilities. These models excel at generating long Chain-of-Thought (CoT)?[24]?responses when tackling complex tasks and exhibit advanced, reflection-like reasoning behaviors. Recently, DeepSeek-R1 has shown that, starting from pretrained base or instruction-tuned models, pure reinforcement learning (RL) with rule-based rewards can spontaneously lead to the emergence of long CoT reasoning, self-correction, self-reflection, and other advanced behaviors, collectively referred to as the “aha moment”. Other open-source works, such as SimpleRL-Zoo?[31], tinyzero?[18], and Logic-RL?[28], which attempt to reproduce R1’s performance and technical details, have also observed similar aha moments. These behaviors—such as self-correction, self-verification, and backtracking, signal the model’s internal experience of strong reasoning ability.

However, relying solely on emergent behaviors is inherently unreliable and difficult to control. Models may fail to consistently manifest these advanced reasoning schemes, which limits both the predictability and scalability of LLM-based reasoning. To overcome this, we propose to explicitly align LLMs with three domain-general reasoning meta-abilities—deduction, induction, and abduction—drawn from Peirce’s classical inference triad?[19].

Deduction infers specific outcomes from general rules and hypotheses?(H+R→O), enabling rigorous prediction and verification. Induction abstracts rules from repeated co-occurrences?(H+O→R), facilitating pattern discovery and generalization. Abduction infers the most plausible explanation for surprising observations?(O+R→H), promoting creative and backward reasoning.

Together, they form a closed inferential loop for hypothesis generation, testing, and revision, mirroring the scientific method and supporting robust and interpretable reasoning.

1. 引言

包括 OpenAI-o1 [11]、o3 [17]、DeepSeek-R1 [8]、Grok 3.5 [27] 和 Gemini 2.5 Pro [3] 在內(nèi)的大型推理模型展現(xiàn)出了非凡的能力。這些模型在處理復雜任務時能夠生成長鏈的推理過程（CoT）[24]，并表現(xiàn)出類似反思的高級推理行為。最近，DeepSeek-R1 表明，從預訓練的基礎模型或指令微調(diào)模型出發(fā)，僅通過基于規(guī)則的獎勵進行純強化學習（RL），就能自發(fā)地產(chǎn)生長鏈 CoT 推理、自我修正、自我反思等高級行為，這些行為統(tǒng)稱為“頓悟時刻”。其他開源項目，如 SimpleRL-Zoo [31]、tinyzero [18] 和 Logic-RL [28]，在嘗試重現(xiàn) R1 的性能和技術細節(jié)時，也觀察到了類似的頓悟時刻。這些行為，如自我修正、自我驗證和回溯，表明模型內(nèi)部具備強大的推理能力。

然而，僅僅依賴于自發(fā)產(chǎn)生的行為本質上是不可靠且難以控制的。模型可能無法始終如一地展現(xiàn)出這些高級推理模式，這限制了基于 LLM 的推理的可預測性和可擴展性。為了解決這個問題，我們提議明確地將 LLM 與三種通用推理元能力——演繹、歸納和溯因——對齊，這些元能力源自皮爾士的經(jīng)典推理三元組[19]。

演繹是從一般規(guī)則和假設推斷出具體結果（H+R→O），這使得嚴格的預測和驗證成為可能。歸納是從反復出現(xiàn)的共現(xiàn)中抽象出規(guī)則（H+O→R），有助于模式發(fā)現(xiàn)和概括。溯因是從令人驚訝的觀察中推斷出最合理的解釋（O+R→H），促進創(chuàng)造性和逆向推理。

它們共同構成了一個用于假設生成、測試和修訂的封閉推理循環(huán)，這與科學方法相呼應，并支持穩(wěn)健且可解釋的推理。

To operationalize these meta-abilities, we construct a task suite with programmatically generated instances and automatic verifiability. Each task targets one core reasoning mode: Deduction: Propositional satisfiability tasks use rule sets?R?and candidate hypotheses?H?to test if all premises entail the observation?O. Induction: Masked-sequence completion requires models to infer latent rules?R?from partial inputs?H,O. Abduction: Inverse rule-graph search backchains from observed consequences?O?through a rule graph?R?to infer the minimal explanatory?H. These tasks are constructed from synthetic distributions that lie out-of-distribution relative to common pretraining corpora, ensuring that performance improvements reflect genuine meta-ability acquisition rather than memorization or shortcut exploitation.

We observe that models aligned to individual meta-abilities make complementary errors. Aggregating their predictions raises overall accuracy by more than 10% relative to a vanilla instruction-tuned baseline. To incorporate the three competencies into a single network, we compared two approaches: training on a mixed task corpus and parameter-space model merging. Parameter-space merging improves average accuracy across math, coding, and science by ?2% on a 7B model and ?4% on a 32B model over the instruction-tuned baseline, demonstrating the strong generalization of merged meta-abilities.

為了將這些元能力付諸實踐，我們構建了一個任務套件，其中包含程序生成的實例和自動可驗證性。每個任務都針對一種核心推理模式：演繹：命題可滿足性任務使用規(guī)則集 R 和候選假設 H 來測試所有前提是否蘊含觀察結果 O。歸納：掩碼序列補全要求模型從部分輸入 H、O 中推斷出潛在規(guī)則 R。溯因：逆向規(guī)則圖搜索從觀察到的后果 O 通過規(guī)則圖 R 進行反向鏈推斷出最小的解釋 H。這些任務由相對于常見預訓練語料庫處于分布外的合成分布構建而成，確保性能提升反映的是真正的元能力獲取，而非記憶或捷徑利用。

我們觀察到針對單個元能力進行對齊的模型會犯互補的錯誤。將它們的預測進行聚合，整體準確率比普通的指令調(diào)優(yōu)基線提高了 10% 以上。為了將這三種能力整合到一個網(wǎng)絡中，我們比較了兩種方法：在混合任務語料庫上訓練和參數(shù)空間模型合并。參數(shù)空間合并使 70 億參數(shù)模型在數(shù)學、編程和科學領域的平均準確率提高了 2%，使 320 億參數(shù)模型提高了 4%，這表明合并后的元能力具有很強的泛化能力。

Furthermore, to evaluate whether meta-ability alignment offers a stronger foundation for subsequent learning, we resumed domain-specific RL training from a checkpoint that have already been aligned and compared it with the same procedure applied to an instruction-tuned model. Starting from the meta-ability checkpoint raises the attainable performance ceiling: after identical continual domain-specific RL training, the model achieves an average gain of about 2% over its instruction-only counterpart. Our key contributions are as follows:

Task suite for meta-abilities.?We introduce a novel RL task suite aligned with three classical meta-abilities—deduction, induction, and abduction—each constructed to train and validate domain-general reasoning skills in large models.

Recipe for Reasoning Mastery.?We propose a three-stage recipe (1) independently align models to each meta-ability; (2) merge them via parameter-space integration; and (3) fine-tune with domain-specific RL. This leads to improved generalization and downstream task accuracy.

Upper-bound boost and scalability.?We empirically demonstrate that meta-ability alignment raises the performance ceiling: our 7B and 32B models show consistent gains over instruction-tuned baselines, across math, coding, and science benchmarks.

此外，為了評估元能力對齊是否為后續(xù)學習提供了更堅實的基礎，我們從已經(jīng)對齊的檢查點恢復特定領域的強化學習訓練，并將其與應用于指令調(diào)優(yōu)模型的相同過程進行比較。從元能力檢查點開始提高了可達到的性能上限：在相同的持續(xù)特定領域強化學習訓練之后，該模型比僅指令調(diào)優(yōu)的模型平均提高了約 2%。我們的主要貢獻如下：

元能力任務套件。我們引入了一個新穎的強化學習任務套件，與三種經(jīng)典元能力——演繹、歸納和溯因——相一致，每個任務套件都旨在訓練和驗證大型模型中的領域通用推理技能。

推理精通的配方。我們提出了一種三階段配方：（1）將模型分別對齊到每個元能力；（2）通過參數(shù)空間整合將它們合并；并且（3）使用特定領域的強化學習進行微調(diào)。這帶來了更好的泛化能力和下游任務的準確性。

上限提升和可擴展性。我們通過實證表明，元能力對齊提高了性能上限：我們的 70 億和 320 億參數(shù)模型在數(shù)學、編程和科學基準測試中均持續(xù)優(yōu)于指令微調(diào)的基線模型。

This work demonstrates that large reasoning models need not rely on unpredictable 'aha moments’ to acquire advanced problem-solving skills. By explicitly aligning deduction, induction, and abduction through automatically generated, self-verifiable tasks, we create specialist agents whose complementary strengths can be merged—without extra compute—into a single checkpoint that outperforms an instruction-tuned baseline by more than 10% on purpose-built diagnostics and up to 2% on seven diverse math, code, and science benchmarks. When this meta-ability-aligned model is used as the starting point for domain-specific reinforcement learning, it lifts the attainable performance ceiling by a further 4% and widens the gap as model capacity scales from 7B to 32B parameters. These results confirm that systematic, modular training of fundamental reasoning modes provides a controllable and scalable foundation for downstream capability composition. Future work will explore richer fusion strategies, extend the task suite to multimodal settings, and investigate how explicit meta-ability control can improve interpretability and safety in large-scale reasoning systems.

結論

這項工作表明，大型推理模型無需依賴難以預測的“頓悟時刻”來獲取高級問題解決技能。通過自動生成的、可自我驗證的任務來明確對演繹、歸納和溯因進行對齊，我們創(chuàng)建了專業(yè)代理，它們的互補優(yōu)勢可以在無需額外計算的情況下合并為一個檢查點，在專門設計的診斷測試中比指令調(diào)優(yōu)的基線高出 10% 以上，在七個不同的數(shù)學、代碼和科學基準測試中最多高出 2%。當這種元能力對齊模型用作特定領域強化學習的起點時，它將可實現(xiàn)的性能上限再提高 4%，并且隨著模型容量從 70 億參數(shù)擴展到 320 億參數(shù)，差距進一步擴大。這些結果證實，對基本推理模式進行系統(tǒng)化、模塊化的訓練為下游能力組合提供了一個可控且可擴展的基礎。未來的工作將探索更豐富的融合策略，將任務套件擴展到多模態(tài)設置，并研究如何通過顯式的元能力控制來提高大規(guī)模推理系統(tǒng)的可解釋性和安全性。