Promptfoo supports configurable model testing with reusable workflows

Promptfoo uses YAML configurations to run prompts against model providers with different inputs and evaluate the results. Assertions are optional, allowing teams to review outputs manually through a web interface or automate checks for JSON content, JavaScript conditions, semantic similarity and model-graded criteria.

Evaluation components can be organized into separate files, while shared defaults and reusable references reduce duplication. Test cases can come from local files, CSV spreadsheets, Google Sheets or Azure Blob Storage. JavaScript and Python scripts can generate variables dynamically, including context retrieved from vector databases. Array-valued variables automatically expand tests across input combinations, and Nunjucks templates support conditional logic, loops and nested objects.

Custom transformations can preprocess input variables, modify model responses before assertions or extract context for specialized checks. Provider transformations run before test and context transformations. For supported reasoning models, the showThinking setting can exclude thinking content from the output used by assertions. Tool use and function calling are also supported across Google, OpenAI and Anthropic models.

Configuration handling includes security considerations. Environment variables resolve when configurations load, and copying secrets into configuration objects can expose them in exported results. Secrets already available in the shell environment should instead be read directly. Remote configuration references block private and loopback network destinations.