NVDA 155.46 ▼0.44%GOOGL 280.20 ▼1.24%MSFT 506.24 ▼2.27%AMD 503.41 ▼0.17%INTC 55.17 ▼0.54%TSMC 359.69 ▼3.68%AMZN 257.89 ▲0.05%META 618.87 ▼1.20%AAPL 351.91 ▼0.25%PLTR 156.56 ▼2.39%
Markets at last close

Models

Marin shares lessons from scaling open model pretraining

·1 min read

Marin completed pretraining a 535B-parameter mixture of experts model on 18T tokens using GPUs from the Huang Foundation. The effort was described as the largest live-streamed pre-training run to date, marking a significant expansion from an academic project that initially trained medium-sized language models on Google’s TPU Research Cloud.

Scaling the project required advances in compute, data preparation, training efficiency and team organization. Researchers ran thousands of smaller models to study scaling behavior, developed systems to turn trillions of raw tokens into training mixtures, and built infrastructure to operate reliably across large, heterogeneous clusters. The team also worked on efficient sparse-model training and securing support for open development at this scale.

Open Athena’s event at Laude Labs in San Francisco focused on the research, engineering and organizational lessons behind the run. The program featured Stanford professor Percy Liang, AI researcher David Hall, Databricks co-founder Andy Konwinski and Marin researcher Will Held, with discussions aimed at the academic, engineering and research community. The gathering has ended.

Originally reported by superpowerdaily.comRead the source →
Related coverage