Yoav Artzi to present new approaches to LLM pre-training

Yoav Artzi will deliver a Siebel School colloquium, “Sparks of New Pre-Training,” on Oct 19, 2026 at 3:30 pm in a hybrid format at 2405 Siebel Center for Computer Science and online. The seminar focuses on pre-training, the first stage of LLM training, where core model qualities are formed and most training data is seen.

The talk will cover two projects aimed at changing pre-training beyond scale. One introduces the hypothesis that state and prediction representations, which are entangled in transformers, should be separated through a simple architectural modification that delivers 2.6x data efficiency. The second centers on externalizing knowledge by training an LLM to rely on an external knowledge base induced from pre-training data. Artzi will introduce Limited Memory Language Models, including a tiny 360M parameter LMLM model that outperforms models trained on 40x the data and reaches factuality performance on par with Claude Sonnet 4.5 and above OpenAI’s 120B open model.

Artzi is an associate professor in the Department of Computer Science and Cornell Tech at Cornell University and a visiting faculty research at Google DeepMind. His work focuses on language modeling and learning in interactive and situated scenarios.