Forge targets reliability gaps in local tool-calling models

Forge is an open-source reliability layer designed to make self-hosted LLM tool-calling more dependable on consumer hardware. The system adds domain-and-tool-agnostic guardrails, including retry nudges, step enforcement, error recovery, rescue parsing, and VRAM-aware context management, without changing the underlying model.

Eval results cited for a paper accepted to ACM CAIS ’26 cover 97 model/backend configurations, 18 scenarios, and 50 runs each. Ministral 8B with Forge reaches 99.3%, while Claude Sonnet with Forge reaches 100%, leaving less than 1 point between a local model on a ? GPU and a frontier API. The same 8B local model with Forge is reported to outperform Claude Sonnet without guardrails, 99.3% versus 87.2%.

The work highlights reliability failures that standard tool-calling stacks can miss, including cases where a tool succeeds technically but finds nothing, causing bad data to flow downstream. Forge introduces ToolResolutionError for that class of failure and uses nvidia-smi at startup to set a token budget, avoiding silent CPU fallback when Ollama or Llamafile exceed VRAM.

Backend choice also emerged as a major variable: the same Mistral-Nemo 12B weights produced 7% accuracy on llama-server with native function calling and 83% on Llamafile in prompt mode. Forge is available through a repository, eval harness, dashboard, and proxy mode for OpenAI-compatible clients.