Karpathy says continual learning remains a major gap for LLMs

Andrej Karpathy has identified continual learning as one of the clearest weaknesses in current large language models, saying current systems cannot simply be told something and expected to remember it. In October 2025, he estimated that addressing these cognitive limits could take about another decade, placing memory, multimodal perception, continuous learning, and computer use among the unresolved gaps in the LLM stack.

The central obstacle is catastrophic forgetting, where fine-tuning on new data can overwrite capabilities learned earlier. Research has split into several approaches: external memory systems such as MemGPT, Letta, Mem0, Zep, and A-MEM; context engineering systems such as ACE; continual post-training methods such as LoRA-based fine-tuning and Self-Distillation Fine-Tuning; and continued pretraining for domain, language, and knowledge updates.

ACE, proposed in October 2025 by Stanford, SambaNova, and UC Berkeley, treats context as an evolving playbook and reports a 10.6% improvement over baselines on agent tasks, an 8.6% improvement on financial reasoning, and adaptation latency reduced by approximately 86.9%. More experimental directions include MIT’s SEAL, Google’s Nested Learning, and knowledge editing methods such as ROME and MEMIT. The emerging consensus is that future systems may combine external memory, evolving context, parameter updates, and self-directed learning rather than rely on a single technique.