Beta-KD: Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models

Jingchen Sun2,†, Shaobo Han1,‡, Deep Patel1, Wataru Kohno1, Can Jin3, Changyou Chen2
1NEC Laboratories America, Inc. 2University at Buffalo, SUNY 3Rutgers University
CVPR 2026
†Work done during an internship at NEC Laboratories America   ‡Project Lead
Overview of the Beta-KD framework: (a) the learning balance between data and teacher; (b) Beta-KD predicts uncertainty weights beta_1 and beta_2 with an amortized optimization network.

Overview of Beta-KD. (a) Conventional KD struggles to balance learning from data (\(\mathcal{L}_{CE}\)) and learning from the teacher (\(\mathcal{L}_{KD}\)). (b) Beta-KD treats teacher supervision as a Gibbs prior, which induces uncertainty weights \(\beta_1, \beta_2\) predicted by an amortized optimization network. These weights adaptively modulate teacher–student alignment without manual hyperparameter tuning.

Abstract

Knowledge distillation establishes a learning paradigm that leverages both data supervision and teacher guidance. However, determining the optimal balance between learning from data and learning from the teacher is challenging, as some samples may be noisy while others are subject to teacher uncertainty. This motivates the need for adaptively balancing data and teacher supervision. We propose Beta-weighted Knowledge Distillation (Beta-KD), an uncertainty-aware distillation framework that adaptively modulates how much the student relies on teacher guidance. Specifically, we formulate teacher–student learning from a unified Bayesian perspective and interpret teacher supervision as a Gibbs prior over student activations. This yields a closed-form, uncertainty-aware weighting mechanism and supports arbitrary distillation objectives and their combinations. Extensive experiments on multimodal VQA benchmarks demonstrate that distilling student Vision-Language Models from a large teacher VLM consistently improves performance. The results show that Beta-KD outperforms existing knowledge distillation methods.

Highlights

Bayesian view of KD

Teacher signals act as a Gibbs prior over student activations. KD training is MAP inference for those activations, done by amortized neural inference.

Closed-form weighting

A Laplace approximation gives task- and instance-level uncertainty weights \(\beta\). No grid search over loss weights is needed.

Consistent gains

Up to +4.7 on ScienceQA and +2.0 average across six VQA benchmarks, at under 0.01% extra training time.

Method

Teacher-informed Gibbs prior. We treat the teacher–student discrepancy \(\ell(a^s; a^t)\) as an energy. Here \(a^s\) is a student activation (features, logits or probabilities) and \(a^t\) is the matching teacher signal. Lower energy means higher probability, and the concentration \(\beta\) sets how strongly the student is pulled toward the teacher:

$$p(a^s \mid a^t, \beta) = \frac{1}{Z_\beta(a^t)} \exp\!\big[-\beta\,\ell(a^s; a^t)\big], \qquad \beta > 0.$$

MAP of student activation. We assume the teacher affects the output \(y\) only through the student, i.e. \(y \perp (a^t, \beta) \mid a^s\). MAP inference then recovers the familiar CE + KD objective, plus a partition-function term:

$$\min_{a^s}\; \underbrace{-\log p(y \mid a^s)}_{\text{Cross-Entropy}} \;+\; \beta\,\underbrace{\ell(a^s; a^t)}_{\text{Distillation}} \;+\; \log Z_\beta(a^t).$$

Laplace approximation. A second-order expansion around the energy minimizer makes \(Z_\beta\) tractable. It also adds a \(-\tfrac{d}{2}\log\beta\) term that keeps \(\beta\) from diverging:

$$\min_{a^s}\; -\log p(y \mid a^s) \;+\; \beta\,\ell(a^s; a^t) \;-\; \frac{d}{2}\log\beta.$$

Amortized optimization on \(\beta\). We learn \(\beta\) jointly with the student instead of solving for it per sample. With task-level (homoscedastic) uncertainty, \(\beta_k\) is a learnable scalar for each loss. With instance-level (heteroscedastic) uncertainty, a lightweight network predicts \(\beta(x) = g_\phi(h(x)) > 0\) from each input:

$$\min_{\theta,\phi}\; \mathcal{L}_{\mathrm{CE}}(\theta) \;+\; g_\phi(h(x))\,\ell(\theta) \;-\; \frac{d}{2}\log g_\phi(h(x)).$$
Four representative distillation energies visualized in the probability simplex

Energy functions. Four representative distillation losses visualized in the probability simplex. Beta-KD works with any of them, from KL variants (FKL, RKL) to MSE and Cosine distance.

Experimental Results

The teacher is MobileVLM V2 7B and the student is MobileVLM V2 1.7B. Only the language backbone is fine-tuned. For the six-benchmark results, the transfer set is scaled to 2.2M image–text pairs.

Results on six multimodal benchmarks

MethodLLM#Samples MMEPMMEAGQAVQAT POPEMMBdevSQAIAvg.
LLaVA-1.5Vicuna-7B1.2M1510.775.562.058.285.964.366.868.8
ShareGPT4VVicuna-7B1.9M1567.478.463.360.485.768.868.470.8
MobileVLM V2 7BVicuna-7B3.6M1559.078.062.662.386.669.274.772.2
MoE-LLaVA-2BQwen-1.5-1.8B2.2M1292.064.661.548.087.059.763.164.0
MoE-LLaVA-1.6BStableLM-1.6B2.2M1300.065.060.447.884.359.462.663.3
MobileVLM 3BMobileLLaMA 2.7B3.6M1288.964.459.047.584.959.661.262.8
MobileVLM 1.7BMobileLLaMA 1.4B3.6M1196.259.856.141.584.553.257.358.7
MobileVLM V2 1.7BMobileLLaMA 1.4B3.6M1246.362.355.151.285.357.663.262.5
LLAVADIMobileLLaMA 1.4B3.6M1178.658.955.445.384.755.056.059.2
Align-KDMobileLLaMA 1.4B3.6M1288.964.460.052.885.457.061.063.4
w/ Beta-KD (Task)MobileLLaMA 1.4B3.6M1315.1 +26.365.8 +1.360.2 +0.252.9 +0.185.4 +0.058.8 +1.861.4 +0.464.1 +0.6
w/ Beta-KD (Instance)MobileLLaMA 1.4B3.6M1343.0 +54.167.1 +2.760.8 +0.853.9 +1.185.4 +0.059.1 +2.261.2 +0.264.6 +1.2
Cosine KDMobileLLaMA 1.4B3.6M1308.465.459.952.284.657.161.363.4
w/ Beta-KD (Task)MobileLLaMA 1.4B3.6M1352.0 +43.667.6 +2.260.8 +0.953.9 +1.785.4 +0.859.1 +2.061.2 −0.164.7 +1.2
w/ Beta-KD (Instance)MobileLLaMA 1.4B3.6M1350.3 +42.067.5 +2.161.2 +1.354.2 +1.986.6 +1.960.2 +3.163.1 +1.865.5 +2.0

MMEA is MMEP divided by 20 to match the scale of the other benchmarks. Avg. is computed over the six datasets. Align-KD is our reproduction. Cosine KD replaces the FKL loss in Align-KD with our probability-space Cosine distance. Beta-KD improves both variants and sets a new state of the art. The 1.7B student even beats larger models such as MobileVLM 3B and MoE-LLaVA-2B.

Two-loss balancing on ScienceQA (CE + KD)

MethodVQA-AccIMG-Acc MethodVQA-AccIMG-Acc MethodVQA-AccIMG-Acc
CE + FKL48.254.7CE + JS48.554.8CE + CTKD48.655.0
w/ Manual48.654.9w/ Manual49.456.3w/ Manual49.155.1
w/ Beta-KD (Task)49.3 +0.755.3 +0.4w/ Beta-KD (Task)50.5 +1.158.1 +1.7w/ Beta-KD (Task)49.9 +0.855.3 +0.3
w/ Beta-KD (Instance)51.8 +3.261.1 +6.2w/ Beta-KD (Instance)53.3 +3.966.9 +10.6w/ Beta-KD (Instance)53.8 +4.665.1 +10.0
CE + RKL46.250.7CE + TVD49.153.1CE + MSE-Probs47.152.1
w/ Manual47.853.5w/ Manual50.156.9w/ Manual49.356.1
w/ Beta-KD (Task)49.5 +1.856.4 +3.0w/ Beta-KD (Task)51.3 +1.261.0 +4.1w/ Beta-KD (Task)51.7 +2.460.3 +4.2
w/ Beta-KD (Instance)52.4 +4.661.6 +8.1w/ Beta-KD (Instance)52.0 +1.960.0 +3.1w/ Beta-KD (Instance)52.2 +2.962.4 +6.3
CE + SFKL49.457.7CE + AdaptiveKL49.255.2CE + Cosine-Logits48.455.6
w/ Manual51.160.4w/ Manual49.656.0w/ Manual50.759.3
w/ Beta-KD (Task)53.1 +1.963.2 +2.9w/ Beta-KD (Task)50.2 +0.657.0 +1.0w/ Beta-KD (Task)53.1 +2.463.3 +3.9
w/ Beta-KD (Instance)51.0 −0.160.0 −0.3w/ Beta-KD (Instance)54.2 +4.665.5 +9.5w/ Beta-KD (Instance)53.7 +3.064.9 +5.6
CE + SRKL48.655.2CE + TAID46.249.3CE + Cosine-Probs51.659.1
w/ Manual50.257.8w/ Manual50.156.7w/ Manual52.862.1
w/ Beta-KD (Task)52.0 +1.860.7 +2.8w/ Beta-KD (Task)54.1 +4.064.4 +7.7w/ Beta-KD (Task)54.2 +1.465.3 +3.2
w/ Beta-KD (Instance)54.3 +4.167.3 +9.5w/ Beta-KD (Instance)54.8 +4.765.9 +9.2w/ Beta-KD (Instance)54.9 +2.167.5 +5.5

Manual uses fixed weights set from the initial loss scales. Beta-KD (Task) learns one uncertainty weight per loss, shared across samples. Beta-KD (Instance) predicts a weight for each input. VQA-Acc covers all questions, and IMG-Acc covers only questions with image inputs. Gains are measured against Manual.

Show three-loss balancing on ScienceQA (CE + KD + FD)
MethodVQA-AccIMG-Acc MethodVQA-AccIMG-Acc MethodVQA-AccIMG-Acc
CE + FKL + FD45.547.9CE + JS + FD48.453.2CE + MSE-Logits + FD46.451.1
w/ Manual47.753.3w/ Manual48.953.7w/ Manual47.452.6
w/ Beta-KD (Task)50.2 +2.559.1 +5.8w/ Beta-KD (Task)49.7 +0.854.6 +0.9w/ Beta-KD (Task)48.6 +1.254.5 +1.9
w/ Beta-KD (Instance)52.9 +5.261.9 +8.6w/ Beta-KD (Instance)52.7 +3.862.3 +8.6w/ Beta-KD (Instance)51.3 +3.960.9 +8.3
CE + RKL + FD50.356.5CE + TVD + FD49.755.1CE + MSE-Probs + FD48.254.0
w/ Manual49.856.0w/ Manual51.859.7w/ Manual48.154.5
w/ Beta-KD (Task)49.5 −0.355.8 −0.2w/ Beta-KD (Task)54.1 +2.464.7 +4.9w/ Beta-KD (Task)48.2 +0.155.3 +0.8
w/ Beta-KD (Instance)50.1 +0.357.5 +1.5w/ Beta-KD (Instance)49.0 −2.856.8 −2.9w/ Beta-KD (Instance)47.1 −1.053.1 −1.4
CE + SFKL + FD51.559.0CE + AdaptiveKL + FD47.451.5CE + Cosine-Logits + FD44.948.4
w/ Manual51.359.1w/ Manual49.756.6w/ Manual48.155.0
w/ Beta-KD (Task)51.4 +0.159.6 +0.5w/ Beta-KD (Task)52.3 +2.662.1 +5.4w/ Beta-KD (Task)51.5 +3.462.0 +6.9
w/ Beta-KD (Instance)50.0 −1.359.1 −0.1w/ Beta-KD (Instance)49.5 −0.256.4 −0.2w/ Beta-KD (Instance)51.4 +3.360.1 +5.1
CE + SRKL + FD48.352.4CE + TAID + FD48.354.4CE + Cosine + FD47.752.3
w/ Manual49.455.1w/ Manual49.055.9w/ Manual49.456.9
w/ Beta-KD (Task)50.9 +1.558.2 +3.1w/ Beta-KD (Task)50.0 +1.057.7 +1.8w/ Beta-KD (Task)51.3 +1.961.8 +4.9
w/ Beta-KD (Instance)53.9 +4.563.3 +8.1w/ Beta-KD (Instance)53.2 +4.264.8 +8.9w/ Beta-KD (Instance)54.2 +4.864.8 +7.9

Each baseline combines cross-entropy, a KL-based distillation loss and a feature-level distillation (FD) objective. Gains are measured against Manual.

How Uncertainty Weighting Works

Training loss curves and evolution of the uncertainty weights beta for FKL+CE and RKL+CE

Faster, more stable optimization. Total training loss (top) and how the task- and instance-level weights \(\beta\) evolve (bottom) for FKL + CE and RKL + CE. Adapting \(\beta\) during training speeds up convergence and stabilizes optimization.

Teacher and student logit distributions at training step 10 and step 190

Closer teacher–student alignment. Teacher and student logit distributions early (Step 10) and late (Step 190) in training. Both Beta-KD variants shrink the logit matching distance compared with the baseline, and the instance-level variant aligns most closely.

Student entropy and the instance-level distillation weight over training

Instance-level adaptivity. Early in training, student entropy is high and the student is uncertain. Beta-KD assigns the distillation loss a large weight (e.g. 1.98), so the student follows the teacher closely. As entropy falls and the student grows more confident, the weight drops (e.g. 0.3) and the student relies more on its own predictions.

BibTeX

@inproceedings{sun2026uncertainty,
  title={Uncertainty-Aware Knowledge Distillation for Multimodal Large Language Models},
  author={Sun, Jingchen and Han, Shaobo and Patel, Deep and Kohno, Wataru and Jin, Can and Chen, Changyou},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  year={2026}
}