Startups rethink the transformer era

Transformers remain the foundation of major LLMs, but their dense attention mechanism has become a bottleneck as models process longer inputs and support reasoning features. A document 10,000 words long might require a transformer to perform 50 million multiplications, driving up energy use and making larger context windows harder to manage. Data center electricity consumption is also projected to double by 2030.

Several startups are attacking the problem directly. Subquadratic says its sparse attention model can match top LLMs on tasks including search and coding by identifying which words matter for each input. Manifest AI is replacing attention with power retention, which keeps a rolling summary of relevant context. Liquid AI combines transformers with liquid neural networks, saying its compact models have reached almost 34 million downloads and use a hybrid design made of 20% transformers and 80% liquid neural networks.

Other companies are exploring more radical shifts. Inception uses diffusion to generate blocks of text at once and claims its models can run 10 times faster than comparable systems. Pathway’s Dragon Hatchling uses state spaces instead of attention and beat more than 97% of more than 250,000 very hard sudoku puzzles, a result meant to show how models could reason beyond language-first methods.