Hugging Face uses Langfuse to scale model discovery

Hugging Face’s Hub supports nearly 3 million public models and 1 million public datasets, but many artifacts described in the 100 to 350 AI papers posted to arXiv each day remain in GitHub repositories or Google Drive links. Machine Learning Engineer Niels Rogge built an overnight agent on Hugging Face Jobs to identify missing models or datasets, open GitHub issues for unpublished artifacts, and submit pull requests when existing Hub entries lack metadata.

Rogge added Langfuse tracing to understand the agent’s decisions before letting it run unattended. Each arXiv ID becomes a session, with each paper run captured as a trace and the LLM steps recorded as observations, including prompts, context, outputs, and handoffs between steps. That visibility exposed whether dynamic prompts contained the right GitHub link and showed that a single classification call was unreliable, leading him to run it five times in parallel and use a majority vote.

Manual error analysis on 50 recent pull requests found hallucinated code and fabricated GitHub URLs. Traces pointed to a missing GitHub README in the agent’s context, so Rogge added it to the system prompt and shipped other fixes. A later review found the hallucinations gone, and after two rounds of manual error analysis the system received just two negative comments across thousands of issues and pull requests. Rogge now plans to explore MCP, Langfuse annotation queues, and broader evaluation for work including Papers with Code.