Sign in
← AI PulseAug 5, 2026

Policy · research · Single-source brief

Diagnosing Interface Injury in Qwen3-0.6B-Base After KDA Linearization

Researchers converted 21 layers of Qwen3-0.6B-Base to KDA linear attention, observing a significant drop in multiple-choice accuracy despite preserved perplexity, which was traced to an interface injury.

By Illumora Editorial

Source · Aug 5, 2026, 4:00 AM · On Illumora · Aug 5, 2026, 4:03 AM

Media from the primary source — shown here so you can stay on Illumora.

Rewritten from one allowlisted primary — not independent enterprise reporting. Lanes →

Brief drafted by Illumora’s editorial model from the linked primary source. Ops desk reviews flagged pieces. How we write →

Read the source →arXiv cs.CL — Stuck on "A": Diagnosing and Repairing Interface Injury in Attention-to-KDA Linearization of a 0.6B Language Model
Save

A recent paper on arXiv cs.CL details an investigation into the effects of converting layers of the Qwen3-0.6B-Base language model to Kimi Delta Attention (KDA) linear attention. The study converted 21 of 28 full-attention layers on a single consumer-grade GPU. While hidden-state alignment and end-to-end KL distillation brought the student model's perplexity close to its teacher, multiple-choice accuracy remained near random chance on C-Eval.

Key Points

  • 21 of 28 full-attention layers of Qwen3-0.6B-Base were converted to KDA linear-attention layers.
  • The conversion was performed on a single consumer-grade GPU budget.
  • Multiple-choice accuracy on C-Eval remained between 25-29% after conversion, compared to the teacher's 50.6%.
  • The model predicted "A" 81% of the time, indicating a tendency to stick to option labels rather than content.
  • 106 out of 161 questions retained the same label under all four rotations of a diagnostic test.
  • A 1,000-step format-targeted completion-only KL stage improved C-Eval scores by +12.48 points.
  • This repair stage roughly halved label-stickiness.

Context

According to the arXiv paper, the conversion process involved replacing full-attention layers with KDA linear-attention layers. The researchers then used a four-permutation diagnostic to rotate answer options while keeping content fixed. This diagnostic revealed that the model was consistently predicting the label "A" regardless of the actual answer content, a phenomenon termed "interface injury" that standard distillation metrics did not detect.

Why It Matters

This research highlights a specific failure mode that can arise during model architecture modifications and distillation, particularly when evaluating performance solely on perplexity or standard metrics. Builders and practitioners need to be aware that models can exhibit unexpected behaviors, such as label-stickiness, which can significantly impact downstream task performance even if general language understanding appears preserved.

What To Do

  • Note that standard distillation metrics may not capture all forms of model degradation.
  • Consider employing diagnostic tests, such as answer option rotation, to uncover interface injuries.
  • Explore targeted fine-tuning stages, like format-targeted completion-only KL, to address specific behavioral issues.
  • Watch for the release of the code, weights, recipes, and audit trail mentioned in the paper to understand the engineering lessons.

Keep Exploring

/atlas/llama-open /techniques/output-schema /techniques/constraints