A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions
Published in ICML, 2026
Recommended citation: Wang, Q., Qin, R., Qin, Z., Shen, W., & Wei, Z. A Unified Approach to Interpreting Knowledge Distillation for Large Language Models via Interactions. In ICML 2026. https://icml.cc/virtual/2026/poster/65719
Abstract. This paper uses game-theoretic interactions to provide a unified interpretation of knowledge distillation for large language models and proposes Complex Interaction Penalty to improve distillation.
Authors: Qingzhuo Wang, Ruiyang Qin, Zhenxin Qin, Wen Shen, Zhihua Wei.