11 Fascinating new Policy Optimization techniques
text
11 Fascinating new Policy Optimization techniques
new_line
new_line
Policy optimization (PO) algorithms are central to training AI models with preference-based feedback. In recent weeks, numerous new PO methods have emerged that build on or replace the popular PPO and GRPO, solving their issues. Here are 11 of them:
text
Policy optimization (PO) algorithms are central to training AI models with preference-based feedback. In recent weeks, numerous new PO methods have emerged that build on or replace the popular PPO and GRPO, solving their issues. Here are 11 of them:
new_line
new_line
1. BAlanced Policy Optimization (BAPO) → 
text
1. BAlanced Policy Optimization (BAPO) → 
BAPO: Stabilizing Off-Policy Reinforcement Learning for LLMs via
  Balanced Policy Optimization with Adaptive Clipping (2510.18927)
https://huggingface.co/papers/2510.18927
2510.18927
paper
resource
https://huggingface.co/papers/2510.18927
new_line
Dynamically adjusting the clipping bounds in PPO-style updates to balance positive and negative gradients and prevent entropy collapse
text
Dynamically adjusting the clipping bounds in PPO-style updates to balance positive and negative gradients and prevent entropy collapse
new_line
text
new_line
2. Training-Free GRPO → 
text
2. Training-Free GRPO → 
Training-Free Group Relative Policy Optimization (2510.08191)
https://huggingface.co/papers/2510.08191
2510.08191
paper
resource
https://huggingface.co/papers/2510.08191
new_line
Instead of using numeric rewards, it compares rollouts semantically to distill useful knowledge as a token prior, which is then applied during inference to guide the model’s behavior
text
Instead of using numeric rewards, it compares rollouts semantically to distill useful knowledge as a token prior, which is then applied during inference to guide the model’s behavior
new_line
new_line
3. Asymmetric Importance Sampling Policy Optimization (ASPO) → 
text
3. Asymmetric Importance Sampling Policy Optimization (ASPO) → 
ASPO: Asymmetric Importance Sampling Policy Optimization (2510.06062)
https://huggingface.co/papers/2510.06062
2510.06062
paper
resource
https://huggingface.co/papers/2510.06062
new_line
Fixes imbalanced token weighting in LLM training. It flips the importance sampling ratios for positive tokens to correct over- and under-updates, and adds a soft dual-clipping step to keep gradients stable
text
Fixes imbalanced token weighting in LLM training. It flips the importance sampling ratios for positive tokens to correct over- and under-updates, and adds a soft dual-clipping step to keep gradients stable
new_line
new_line
4. In-Context Steered Policy Optimization (ICPO) → 
text
4. In-Context Steered Policy Optimization (ICPO) → 
https://arxiv.org/abs/2510.26519
https://arxiv.org/abs/2510.26519
link
new_line
Uses a model’s own in-context learning ability to guide training with existing data. It combines Mixed-Policy GRPO with Implicit Expert Forcing to expand exploration and adds Expert Region Reject Sampling and Annealed Expert-Bonus Reward Shaping to ensure stability and balanced expert influence
text
Uses a model’s own in-context learning ability to guide training with existing data. It combines Mixed-Policy GRPO with Implicit Expert Forcing to expand exploration and adds Expert Region Reject Sampling and Annealed Expert-Bonus Reward Shaping to ensure stability and balanced expert influence
new_line
new_line
5. Graph-Enhanced Policy Optimization (GEPO) → 
text
5. Graph-Enhanced Policy Optimization (GEPO) → 
https://arxiv.org/abs/2510.26270
https://arxiv.org/abs/2510.26270
link
new_line
Builds a graph of an agent’s experiences to understand how different states connect, guide exploration and assign rewards more effectively
text
Builds a graph of an agent’s experiences to understand how different states connect, guide exploration and assign rewards more effectively
new_line
new_line
6. Information Gain-based Policy Optimization (IGPO) → 
text
6. Information Gain-based Policy Optimization (IGPO) → 
Information Gain-based Policy Optimization: A Simple and Effective
  Approach for Multi-Turn LLM Agents (2510.14967)
https://huggingface.co/papers/2510.14967
2510.14967
paper
resource
https://huggingface.co/papers/2510.14967
new_line
Uses the model’s own belief updates to create dense, informative feedback for smoother multi-turn learning
text
Uses the model’s own belief updates to create dense, informative feedback for smoother multi-turn learning
new_line
new_line
Read further below ⬇️
text
Read further below ⬇️
new_line
If you like this, also subscribe to the Turing post: 
text
If you like this, also subscribe to the Turing post: 
https://www.turingpost.com/subscribe
https://www.turingpost.com/subscribe
link
11 Fascinating new Policy Optimization techniques

Policy optimization (PO) algorithms are central to training AI models with preference-based feedback. In recent weeks, numerous new PO methods have emerged that build on or replace the popular PPO and GRPO, solving their issues. Here are 11 of them:

1. BAlanced Policy Optimization (BAPO) → https://huggingface.co/papers/2510.18927
Dynamically adjusting the clipping bounds in PPO-style updates to balance positive and negative gradients and prevent entropy collapse
 
2. Training-Free GRPO → https://huggingface.co/papers/2510.08191
Instead of using numeric rewards, it compares rollouts semantically to distill useful knowledge as a token prior, which is then applied during inference to guide the model’s behavior

3. Asymmetric Importance Sampling Policy Optimization (ASPO) → https://huggingface.co/papers/2510.06062
Fixes imbalanced token weighting in LLM training. It flips the importance sampling ratios for positive tokens to correct over- and under-updates, and adds a soft dual-clipping step to keep gradients stable

4. In-Context Steered Policy Optimization (ICPO) → https://arxiv.org/abs/2510.26519
Uses a model’s own in-context learning ability to guide training with existing data. It combines Mixed-Policy GRPO with Implicit Expert Forcing to expand exploration and adds Expert Region Reject Sampling and Annealed Expert-Bonus Reward Shaping to ensure stability and balanced expert influence

5. Graph-Enhanced Policy Optimization (GEPO) → https://arxiv.org/abs/2510.26270
Builds a graph of an agent’s experiences to understand how different states connect, guide exploration and assign rewards more effectively

6. Information Gain-based Policy Optimization (IGPO) → https://huggingface.co/papers/2510.14967
Uses the model’s own belief updates to create dense, informative feedback for smoother multi-turn learning

Read further below ⬇️
If you like this, also subscribe to the Turing post: https://www.turingpost.com/subscribe
🔥
yashmarathe Dcas89 blanchon ankitd-nx Eryk-Chmielewski ahczhg pascal-maker ogulcanakca C50BARZ Miracle12345 hugobui hexbinai John6666
64838b28c235ef76b63e4999
https://cdn-avatars.huggingface.co/v1/production/uploads/64838b28c235ef76b63e4999/ZhQCYoU3vps71Ag7Jezj6.jpeg
Ksenia Se
Kseniase
user
65f9184ca4214e629527fa5f
/avatars/41bc92229a461c53eceadc8afc687f63.svg
Harsh Nilesh Pathak
harsh306
user