JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
Ranking
Overall
83
Content
100
Popularity
43
Observed public metrics from 1 member.
Merged summary
TL;DR - JustFit is an MLX-based runtime that serves Qwen3.8-27B with more than 200K tokens of context on a 24 GiB MacBook by compressing and managing execution state just in time. It expands local long-context inference capacity without depending on additional model-weight quantization.
- Completed 196,608 input plus 16,384 output tokens in three runs, reaching 212,992 positions—6.93× the mlx-vlm baseline.
- Combines compressed KV execution, component residency swapping, and state-preserving serving transitions.
- Achieved 19.11 tokens/s on a 32K-input, 64-output test and a median 16,374 MiB peak footprint on a repeated 32K+6K workload.
- The integrated runtime correctly answered 29 of 30 AIME 2026 problems.
Sources (1)
JustFit: 200K-Token LLM Serving on a 24 GiB Laptop with Just-in-Time State Management
Public signals
Hugging Face upvotes 0
TL;DR - JustFit is an MLX-based runtime that serves Qwen3.8-27B with more than 200K tokens of context on a 24 GiB MacBook by compressing and managing execution state just in time. It expands local long-context inference capacity without depending on additional model-weight quantization.
- Completed 196,608 input plus 16,384 output tokens in three runs, reaching 212,992 positions—6.93× the mlx-vlm baseline.
- Combines compressed KV execution, component residency swapping, and state-preserving serving transitions.
- Achieved 19.11 tokens/s on a 32K-input, 64-output test and a median 16,374 MiB peak footprint on a repeated 32K+6K workload.
- The integrated runtime correctly answered 29 of 30 AIME 2026 problems.