An Exploration of the Application of Reinforcement Learning in Large Model Training

Abstract

This paper conducts a systematic study of the core principles of reinforcement learning and its application mechanisms in the training of large models. First, the paper outlines the foundational framework of reinforcement learning based on Markov decision processes,
which includes value functions, policy optimization, and three major algorithm types. Building on this foundation, it analyzes the integrated
architecture of large models and reinforcement learning. Starting from three key modulesagents, environment interaction, and reward
designthe paper examines the adaptation logic of key algorithms such as RLHF, PPO, and Actor-Critic, and explores their application stages
and optimization pathways throughout the entire training process of large models. The study demonstrates that reinforcement learning can
effectively align large models with human preferences, thereby enhancing output quality and training stability. Through framework optimization, algorithmic improvements, and multidimensional validation, a closed-loop optimization system can be established, significantly improving the decision-making capabilities and generalization performance of large models. This provides a feasible technical pathway for the efficient training and alignment optimization of large models.