Byteification helps LLMs read individual characters

Many LLMs struggle with simple character-level questions because they encode words as tokens, which represent sequences of letters rather than the letters themselves. In one common failure, models asked how often the letter ‘r’ appears in “Strawberry” answer twice, even though it appears three times.

Minixhofer et al. report a technique called byteification that retrofits token-based LLMs to operate at the byte level, where individual characters are represented as binary sequences called bytes. The byteified models can retain the ability to read individual characters while delivering competitive performance.