Simple model merging shows promise for LLMs

Researchers including Jiahe Fan, Yinghao Hou, Si Chen, Aiyuan Zhang and Hong Xie found that substantially different LLM checkpoints can be merged through direct weighted averaging, avoiding costly training and intricate semantic alignment. The work challenges the assumption that merging billion-parameter models requires complex fusion methods.

The proposed process uses training-free dimensional adaptation followed by ratio-controlled interpolation. In union-style merging, a smaller model’s parameter space is expanded to match a larger one, while intersection-style merging truncates the larger model. Applied to Qwen-family models across mathematical reasoning, code generation and language understanding benchmarks, deterministic expansion largely preserved the source model’s original functionality.

Interpolation ratios proved critical. Small-ratio interpolation could improve performance over source models by transferring complementary capabilities, but near-balanced interpolation often caused capability collapse. The researchers describe this trade-off as a ‘seesaw effect,’ where gains on some tasks are accompanied by regressions on others.