In a paper titled “Breaking the Token Ceiling: Distilling Smaller, Stronger Byte Models,” researchers from Meta FAIR and the University of Washington propose an intriguing idea: during distillation, can the learning unit be changed from tokens to bytes, allowing the student model to learn directly from byte-level probability distributions? In experiments using Llama 3-8B as the teacher, the team used scaling laws to predict that as training compute increases, byte-level distillation will overtake token-level distillation, with a downstream average accuracy ceiling about 4 percentage points higher.
Distillation is a familiar technique: a large model acts as teacher and transfers capabilities to a smaller student model to reduce memory, compute, and response costs. Unlike supervised training, which only tells the model the “next correct token,” distillation also lets the student learn the teacher’s probability judgments across candidate tokens. That is also where the problem lies. For Llama 3-8B, for example, the vocabulary contains 128,256 tokens, and each prediction position corresponds to more than 100,000 candidates. The full distribution is difficult to store, so in practice only top-k truncation is feasible.
Byte models neatly compress the candidate space. A byte consists of 8 bits and has 256 possible values; even with a few special symbols, each position requires probabilities for only a little over 200 possibilities, making the full distribution easier to retain. The challenge is that the teacher predicts an entire token while the student predicts individual bytes. To align them, the teacher’s probability distribution must be converted as well.
The paper proposes two approaches. Marginalize-It directly aggregates and redistributes probabilities and requires only one teacher forward pass, but it loses some probability information from already-completed tokens, making it fundamentally an approximation. End-Of-Token adds a special symbol at the end of each token, giving “token end” an explicit prediction position and thereby mapping the teacher distribution more completely into byte space.
This connects with the direction StarWar Cloud is pursuing in its LLM API marketplace and compute scheduling: smaller models and cheaper inference do not automatically mean better usability. The key is unified access across models of different sizes and scheduling by task and cost. Enterprises need both capable models for difficult problems and small, economical models for high-frequency requests. Only an orchestrated, metered resource pool can keep cost and performance controllable at the same time.
The results also remind observers not to rely on a single metric. Byte models surpass token models early on per-byte prediction loss, but their scores on actual tasks may still be lower—bits per byte (BPB) measures how accurately predictions are made, while downstream tasks assess whether the answer is selected correctly. In addition, although End-Of-Token has the highest ceiling, for the same amount of text its training compute is about 30.94% higher than that of ordinary byte models, and inference costs have not yet been compared on an equal-cost basis.
This work demonstrates a new path for small-model training: using a smaller prediction space and less training text to pursue a higher capability ceiling. But for that potential to be realized, sustained investment in compute and fairer evaluation will still be required.