重新审视交叉熵：LM Loss还有哪些选择？

TL;DR - A theoretical deep-dive by Su Jianlin (PaperWeekly) deriving why cross-entropy is the standard LM loss, and what alternatives exist within the proper scoring rules / Fenchel-Young framework. It matters because it reframes the Softmax+cross-entropy pairing as a derivable optimum rather than convention, opening a principled space for loss/activation redesign.

• Because language is one-to-many and corpora arrive as scattered samples, the LM loss must be linear in the target distribution p so it can be estimated by sampling; this rules out metrics like Total Variation. Solving that constraint yields exactly the family of proper scoring rules, generated by any concave function of p.
• Cross-entropy (log score) is uniquely singled out if one additionally requires the loss to depend only on the predicted probability of the observed token; Brier, spherical, and Tsallis scores are generalizations that reduce to log score in a limiting case.
• Gradient analysis under Softmax: cross-entropy gives the clean q - p gradient (convex in logits z, zero only at the target), while squared/Brier loss carries an extra factor that vanishes when the model is confidently wrong — low early-training efficiency and possible non-convex saturation traps, but better late-stage noise robustness.
• Reversing the derivation (demanding a clean q - p gradient) shows the optimal activation for a given score is the convex conjugate of its generating function — i.e. Fenchel-Young losses. Cross-entropy recovers Softmax; Tsallis scores recover Sparsemax (α=2) and Entmax-α, with α>1 giving sparse distributions.