Omega-S: A Functional Resilience Index for LLM Fine-Tuning
Ranking
Overall
84
Content
100
Popularity
45
Observed public metrics from 1 member.
Merged summary
TL;DR - Omega-S is a lightweight regularization penalty designed to reduce capability loss during LLM fine-tuning without retaining prior data or weights. On Llama-3-8B with LoRA, it improved code-capability retention after prose fine-tuning while adding under 4% per-step cost.
- HumanEval retention improved from 62.9% to 84.1% across ten seeds.
- Omega-S outperformed tuned weight decay on all ten seeds and tuned EWC on eight.
- Analysis showed its effective mechanism is primarily a node-degree variance penalty, rather than the intended composite topological objective.
- Identical runs showed substantial retention variability (0.104 standard deviation), highlighting reproducibility limits in seed-paired comparisons.
Sources (1)
Omega-S: A Functional Resilience Index for LLM Fine-Tuning
Public signals
Hugging Face upvotes 7
TL;DR - Omega-S is a lightweight regularization penalty designed to reduce capability loss during LLM fine-tuning without retaining prior data or weights. On Llama-3-8B with LoRA, it improved code-capability retention after prose fine-tuning while adding under 4% per-step cost.
- HumanEval retention improved from 62.9% to 84.1% across ten seeds.
- Omega-S outperformed tuned weight decay on all ten seeds and tuned EWC on eight.
- Analysis showed its effective mechanism is primarily a node-degree variance penalty, rather than the intended composite topological objective.
- Identical runs showed substantial retention variability (0.104 standard deviation), highlighting reproducibility limits in seed-paired comparisons.