腾讯混元发布 UniRL 统一多模态强化学习基础设施及 DRPO/Flow-DPPO 两种新算法，支持跨模态单一 RL 循环训练
多源视角
🚀Introducing UniRL, an RL infra for unified multimodal models. Together with two new RL algorithms: DRPO and Flow-DPPO.<br><br>One RL loop across diffusion/flow matching models, L…
查看这条来源🚀Introducing UniRL, an RL infra for unified multimodal models. Together with two new RL algorithms: DRPO and Flow-DPPO.<br><br>One RL loop across diffusion/flow matching models, L…
查看这条来源🚀Introducing UniRL, an RL infra for unified multimodal models. Together with two new RL algorithms: DRPO and Flow-DPPO.<br><br>One RL loop across diffusion/flow matching models, L…
查看这条来源AI 摘要
现有RL框架多针对单模态设计。腾讯混元开源UniRL，提供统一多模态强化学习基础设施，通过单一后训练循环覆盖扩散/流匹配模型、LLM/VLM及统一多模态模型。同步推出DRPO与Flow-DPPO两种原创算法，支持可插拔推理引擎与FSDP2扩展。开发者可用一套基础设施跨模态进行RL后训练，降低多模态模型对齐成本。 核心观点： 1. UniRL以单一RL循环（generate→score→advantage→update→sync）覆盖扩散/流匹配、LLM、VLM及统一多模态模型，模型与算法为独立轴。 2. Flow-DPPO基于精确散度为流匹配/扩散模型引入信任域掩码策略优化；DRPO为LLM RL引入平滑的优势加权二次正则化。 3. 基础设施支持train-side、SGLang、vLLM-Omni三种可插拔rollout引擎，并内置FSDP2分片与单配置三部署模式。
推荐理由高信息密度，值得细读
原文
One RL loop across diffusion/flow matching models, LLMs/VLMs, and unified multimodal models👇
Code: http://github.com/Tencent-Hunyuan/UniRL
(yes — U(you)-ni-(need) RL 😉)
1、Most RL stacks are built for one modality. UniRL applies a single post-training loop — generate → score → advantage → update → sync — across model families. Model and algorithm are two independent axes, so your coverage is the model × algorithm product, not a fixed recipe menu.
2、One loop, every modality: text→image, text/image→video, vision-language, text-only LLM and VLM, the LLM→diffusion prompt-enhancer, and unified autoregressive+diffusion generation (Hunyuan-Image 3 and Bagel) — a model class no single-purpose RL repo can even express.
3、Built to scale: pluggable rollout engines (train-side / SGLang / vLLM-Omni) behind one typed contract, FSDP2 sharding, and three deployment modes from a single config knob.
4、Two team-original algorithms headline the release:
FlowDPPO: Policy optimization for flow/diffusion models with trust-region masks based on exact divergence (See our paper: Flow-DPPO: Divergence Proximal Policy Optimization for Flow Matching Models https://github.com/Tencent-Hunyuan/UniRL/blob/main/FlowDPPO/HY_FlowDPPO.pdf)
DRPO: LLM RL with a smooth, advantage-weighted quadratic regularizer (See our paper: Rethinking the Divergence Regularization in LLM RL [https://arxiv.org/abs/2606.09821])
讨论
暂无评论。