Preference tuning can outweigh model size in human evaluations
Preference-tuned large language models can outperform much larger unaligned counterparts on human preference evaluations. The advantage is tied to what people prefer, rather than a claim of superior performance across every task. When human preference is the target, alignment work can matter as much as raw scale, making model size alone an incomplete guide to the responses users favor.
Reinforcement learning from human feedback (RLHF), reinforcement learning from AI feedback (RLAIF) and direct preference optimization (DPO) are among the approaches associated with preference tuning and LLM alignment. The application focus includes model safety training and safer business outputs. IBM Technology also provides an explainer on RLHF, addressing the human-feedback approach within this broader focus on aligning language models with human preferences.