Groundbreaking artificial intelligence research from the Allen Institute for AI (Ai2) detailing how to retrofit language models to operate over bytes has officially been published in the scientific journal Nature. The study outlines the foundational work behind Bolmo, a family of fully open byte-level language models introduced by Ai2.
Traditional large language models rely on subword tokenization, breaking text into chunks from a fixed vocabulary. However, this process can sometimes obscure fine-grained details. By contrast, byte-level models work directly with the underlying bytes representing characters, giving them an advantage in handling subtle text features—such as accurately counting specific letters within words.
Expanding Byteification Beyond Olmo
Following the journal publication, Ai2 released new checkpoints on Hugging Face that demonstrate how the byteification method generalizes beyond the initial Olmo models. Researchers successfully applied the same training process to other major model families, creating two new byte-level iterations: Bwen 8B (derived from Qwen 3 8B) and Blama 8B (derived from Llama 3 8B).
According to Ai2, both newly derived models closely match the performance of their source counterparts. Notably, Bwen 8B emerged as the organization’s strongest byteified model to date, outperforming Bolmo 7B across aggregate evaluation suites.
Integration and Future Applications
The Nature publication also explores how byte-ified components interact with wider artificial intelligence ecosystems. Researchers investigated merging existing post-trained versions of Olmo without incurring extra training costs, utilizing task arithmetic to infuse instruction-following capabilities into Bolmo models.
As open-source research continues to progress, developers and researchers are already adapting the technology for specialized tasks. Recent applications include fine-tuning Bolmo 7B for Russian-to-English poetry and lyric translation, signaling broad potential utility for byte-level AI architectures across diverse linguistic and computational domains.
