# Simon Willison 发布了 smevals 评估套件，使开发者能快速创建和运行模型评估并生成报告

- 来源：Simon Willison
- 发布时间：2026-08-01 05:15
- AIWatch 分数：63
- AIWatch 标记：当日精选
- AIWatch 链接：https://aiwatch.icu/events/evt_01kyx1fht98r3b9kers4nmczaa
- 原文链接：https://simonwillison.net/2026/Jul/31/smevals/#atom-everything

## 精选理由

常规快讯，保留列表

## AI 摘要

Simon Willison 发布了 smevals 评估套件，使开发者能快速创建和运行模型评估并生成报告。

## 正文

smevals - a small eval suite for evaluating models, prompts, and harnesses

The result is smevals, a new tool for running small eval suites across different model configurations and grading the results.

This blog entry describes the tool in detail. Here's the 10 second version:

Tell your coding agent to run uvx smevals docs to learn the tool (this outputs the README)
Then tell it to build you an eval suite

Once you've created an eval - which takes the form of a directory with some YAML files - you can run it against models like this:

uvx smevals run path-to-eval/ -m gpt-5.5 -m claude-opus-4.6

Runs are treated separately from grading operations - you can grade your runs (against your defined set of checks) using:

uvx smevals grade path-to-eval/

Then you can run a localhost web server to explore the results:

uvx smevals serve path-to-eval/

Or run the smevals build command to build that report as static HTML, which you can then host anywhere. Here's an example showing an eval suite I built to evaluate how well models can write haikus.

Screenshot of an evaluation dashboard for a haiku-writing benchmark, testing whether models can reply with exactly three non-empty lines. A header describes the eval, with panels below showing a leaderboard ranking three GPT models by score, lists of recent runs and recent grades, tag pass rates, the two haiku prompts that were tested, and details of the graders used with a 0.8 pass threshold.

I've been trying to figure out an approach I like for evals for several years now. smevals is my third iteration on the idea and it feels right to me. I'm looking forward to expanding this more in the future, as well as pointing it at some of my own projects.

Tags: projects, ai, generative-ai, llms, llm, evals, jesse-vincent