"谁来验证验证者?让 LLM 辅助评估与人类偏好对齐"
"Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences"
推荐全译对照查看原文(PDF) ↗
导读
本文位于课程第 7 周「数据选择与质量」单元,回答一个在 LLM 评测实践中尖锐的问题:当人们用 LLM(LLM-as-a-judge)来评估 LLM 输出时,「验证者」本身由谁来验证?作者来自 UC Berkeley(Shreya Shankar、J.D. Zamfirescu-Pereira、Björn Hartmann、Aditya G. Parameswaran)与蒙特利尔大学(Ian Arawjo),发表于 UIST 2024。
论文贡献有二。其一是系统层面:EvalGen——嵌入开源提示工程工具 ChainForge 的混合主动(mixed-initiative)界面,把「评估标准的提出、候选断言(代码函数或 LLM 评分提示)的生成、人工打分、对齐度报告」整合进一个循环,并用基于断言选择率(selectivity)的采样策略来决定让用户给哪些输出打分。其二、也是更具思想冲击力的发现:标准漂移(criteria drift)——用户需要先有标准才能打分,而打分又反过来帮助他们(重新)定义标准;评估标准依赖于所见的具体模型输出,而非可以预先独立写定。这对所有假设「评估标准先于观察输出而存在」的对齐方法(如基于专家预标注的校准流程)提出了根本性质疑。对于构建 AI Agent 评测体系的工程师,本文是「评测本身就是迭代过程」这一世界观的重要实证。
全文对照翻译
译注:以下为论文全文中英对照,覆盖摘要与第 1–9 节全部正文,并收录附录 A(断言选择率、打分采样策略、对齐度形式化定义与采样策略评估)与附录 B(医疗与商品两条流水线的完整任务提示)。References(参考文献列表)按本站惯例不收录;文内方括号编号(如 [45])为原文的参考文献编号。表 1(EvalGen 与 SPADE 对比)与表 2(参与者评分)转为 Markdown 表格,数据一仍其旧;图 1–4 以「[图 N: 英文图题]」转录并附中文说明。英文原段仅去除了 PDF 提取产生的断行连字符、页眉页码及少量因斜体排版导致的缺空格,内容一字未改;原文本身存在个别笔误(如结论首句的 "mixed-initative"、3.1 节一句明显缺字),均按原样保留,中文按上下文补足语意。术语首现处给出中英对照。
摘要(Abstract)
Due to the cumbersome nature of human evaluation and limitations of code-based evaluation, Large Language Models (LLMs) are increasingly being used to assist humans in evaluating LLM outputs. Yet LLM-generated evaluators simply inherit all the problems of the LLMs they evaluate, requiring further human validation. We present a mixed-initiative approach to “validate the validators”—aligning LLM-generated evaluation functions (be it prompts or code) with human requirements. Our interface, EvalGen, provides automated assistance to users in generating evaluation criteria and implementing assertions. While generating candidate implementations (Python functions, LLM grader prompts), EvalGen asks humans to grade a subset of LLM outputs; this feedback is used to select implementations that better align with user grades. A qualitative study finds overall support for EvalGen but underscores the subjectivity and iterative process of alignment. In particular, we identify a phenomenon we dub criteria drift: users need criteria to grade outputs, but grading outputs helps users define criteria. What is more, some criteria appears dependent on the specific LLM outputs observed (rather than independent criteria that can be defined a priori), raising serious questions for approaches that assume the independence of evaluation from observation of model outputs. We present our interface and implementation details, a comparison of our algorithm with a baseline approach, and implications for the design of future LLM evaluation assistants.
由于人类评估的繁琐本性以及基于代码的评估的局限,大语言模型(LLM)正被越来越多地用于辅助人类评估 LLM 输出。然而,LLM 生成的评估器(evaluator)直接继承了其所评估的 LLM 的全部问题,因此仍需人类的进一步验证。我们提出一种「验证验证者(validate the validators)」的混合主动(mixed-initiative)方法——让 LLM 生成的评估函数(无论是提示还是代码)与人类需求对齐。我们的界面 EvalGen 为用户提供自动化辅助,用于生成评估标准(criteria)并实现断言(assertion)。在生成候选实现(Python 函数、LLM 评分提示)的同时,EvalGen 请人类为一部分 LLM 输出打分;这些反馈被用来挑选与用户打分更对齐的实现。一项定性研究总体上支持 EvalGen,但凸显了对齐过程的主观性与迭代性。特别地,我们识别出一种我们称为「标准漂移(criteria drift)」的现象:用户需要先有标准才能给输出打分,而给输出打分又帮助用户定义标准。更有甚者,某些标准看起来依赖于所观察到的具体 LLM 输出(而非可以先验定义的独立标准),这对那些假设「评估独立于对模型输出的观察」的方法提出了严肃质疑。我们呈现了界面与实现细节、我们的算法与基线方法的对比,以及面向未来 LLM 评估助手的设计启示。
CCS CONCEPTS
• Human-centered computing→ Interactive systems and tools; • Computing methodologies→ Natural language processing.
KEYWORDS
language models, auditing, evaluation, interfaces, prompt engineering, active learning
CCS 分类:以人为中心的计算 → 交互系统与工具;计算方法 → 自然语言处理。
关键词:语言模型、审计、评估、界面、提示工程、主动学习
1 引言(Introduction)
Large Language Models (LLMs) make mistakes—they hallucinate, ignore instructions, and generate outputs that require validation [25]. But validating the behavior of LLMs is challenging. In response, researchers and industry developers have created tools for prompt engineering and auditing that help people with testing outputs more systematically [1, 16, 23, 24, 27, 36, 41, 52]. Such approaches require metrics—a set of functions to automatically score LLM outputs, each typically an assertion with true or false values. These metrics increasingly include calls to “evaluator” LLMs (e.g., [1, 27, 52, 58]) that act as “judges,” grading outputs on qualities hard to articulate in code; for instance, the “conciseness” of an output.
大语言模型(LLM)会犯错——它们产生幻觉、无视指令,并生成需要验证的输出 [25]。但验证 LLM 的行为本身就很困难。作为回应,研究者与工业界开发者打造了提示工程(prompt engineering)与审计(auditing)工具,帮助人们更系统地测试输出 [1, 16, 23, 24, 27, 36, 41, 52]。这类方法需要指标(metric)——一组为 LLM 输出自动打分的函数,每项通常是一个取真/假二值的断言(assertion)。这些指标越来越多地包含对「评估器 LLM(evaluator LLM)」的调用(如 [1, 27, 52, 58]),后者充当「裁判」,为难以用代码表述的质量(例如输出的「简洁性」)打分。
The problem is that, just like the LLMs they evaluate, LLMs that perform evaluations cannot be trusted. These “grader” prompts suffer from the same problems as any other prompt—they are unintuitively sensitive to seemingly minor changes in wording or structure [43]. Yet many existing systems do not include support for verifying the quality of LLM-generated evaluations, asking users to simply trust these outputs. How can users reap the efficiency benefits of LLM-assisted evaluation of LLM outputs, while minimizing or avoiding misalignment? How can we help users validate the validators?
问题在于,正如被它们评估的 LLM 一样,执行评估的 LLM 同样不可信。这些「评分提示(grader prompt)」与任何其他提示一样问题缠身——它们对措辞或结构上看似微小的变化有着反直觉的敏感性 [43]。然而,许多现有系统并不包含验证 LLM 生成评估质量的支持,而是要求用户单纯相信这些输出。用户如何才能在收获 LLM 辅助评估 LLM 输出之效率红利的同时,最小化或避免错位(misalignment)?我们如何帮助用户验证验证者?
In this paper, we propose a mixed-initiative approach, EvalGen, to address this automated-evaluation alignment problem in the context of prompt engineering. Our approach streamlines the selection of metrics under practical constraints of user effort and latency. Specifically, an LLM suggests criteria in natural language, based on user context (e.g., the prompt under test), that the user can modify. An LLM then generates a pool of candidate assertions for each criterion—either code or LLM grader prompts that output “true” or “false.” While the user waits for the LLM to generate candidates, they are asked to grade outputs with a simple “good” (thumbs-up) or “bad” (thumbs-down) voting scheme. These grades then guide the automatic selection of assertions that optimize for alignment with user preferences (Section 4.2). After assertion selection, a final report card reveals the alignment between the chosen assertions and the user’s grades. Our approach generalizes beyond the particulars of our specific design, and could be extended to, for instance, update metric implementations with feedback from human preferences, or query the user for finer-grained individual grades.
本文提出一种混合主动方法 EvalGen,在提示工程的语境下解决这一自动评估对齐问题。我们的方法在用户工作量与延迟的实际约束下简化了指标的选择。具体而言,一个 LLM 基于用户上下文(如被测提示)用自然语言建议标准(criteria),用户可以修改;随后 LLM 为每条标准生成一个候选断言池——要么是代码,要么是输出「真」或「假」的 LLM 评分提示。在用户等待 LLM 生成候选期间,系统请他们用简单的「好」(点赞)/「坏」(点踩)投票方案给输出打分。这些打分随后指导断言的自动选择,以优化与用户偏好的对齐(第 4.2 节)。断言选定后,最终的成绩单(report card)揭示所选断言与用户打分之间的对齐情况。我们的方法可推广到这一具体设计之外,例如可扩展为:用人类偏好反馈更新指标实现,或向用户查询更细粒度的单项打分。
EvalGen is embedded inside an existing open-source interface for prompt engineering and auditing, ChainForge [1]. Our alignment algorithm adapts SPADE [45], a fully-automated algorithm for generating Python assertions from the revision history of a prompt. We performed an off-line verification of our human-guided alignment algorithm with SPADE as a baseline [45], then ran a qualitative user study with nine (9) industry practitioners that use LLMs in production contexts. Because our participants were industry practitioners and thus possibly dealing with NDA-protected data, we offered a task adapted from a real LLM pipeline prompt. Our study design did not impose restrictions on how participants used EvalGen, and users could choose whether to ask the tool to suggest criteria, enter criteria manually, or grade a few LLM outputs first before proceeding onto the criteria specification screen.
EvalGen 内嵌于一个现有的开源提示工程与审计界面 ChainForge [1] 之中。我们的对齐算法改编自 SPADE [45]——一个从提示修订历史全自动生成 Python 断言的算法。我们先以 SPADE 为基线,对我们的人类引导对齐算法做了离线验证 [45];随后与九(9)位在生产环境中使用 LLM 的产业从业者开展了一项定性用户研究。由于参与者是产业从业者、可能接触受 NDA 保护的数据,我们提供了一个改编自真实 LLM 流水线提示的任务。研究设计不限制参与者如何使用 EvalGen,用户可自行选择:让工具建议标准、手动输入标准、还是先给若干 LLM 输出打分再进入标准指定界面。
Our findings find overall support for EvalGen, with one important caveat. We observed a “catch-22” situation: to grade outputs, people need to externalize and define their evaluation criteria; however, the process of grading outputs helps them to define that very criteria. We dub this phenomenon criteria drift, and it implies that it is impossible to completely determine evaluation criteria prior to human judging of LLM outputs. Even when participants graded first, we observed that they still refined their criteria upon further grading, even going back to change previous grades. Thus, our findings suggest that users need evaluation assistants to support rapid iteration over criteria and implementations simultaneously. Since criteria are dependent upon LLM outputs (and not independent from them), this raises questions about how to contend with criteria drift in the context of other “drifts”—e.g., model drift [4], prompt edits, or upstream changes in a chain. Our findings also (i) underscore the necessity of mixed-initiative approaches to the alignment of LLM-assisted evaluations that also embrace messiness and iteration, and (ii) raise broader questions about what “alignment with user preferences” means for evaluation assistants.
我们的发现总体上支持 EvalGen,但有一个重要警示。我们观察到一个「catch-22(第二十二条军规式的两难)」局面:要给输出打分,人们需要先外化并定义其评估标准;然而,给输出打分的过程恰恰帮助他们定义这套标准。我们将这一现象命名为标准漂移(criteria drift),它意味着:在人类评判 LLM 输出之前完全确定评估标准是不可能的。即使是先打分的参与者,我们仍观察到他们会在继续打分时精炼标准,甚至回头修改此前的打分。因此,我们的发现表明,用户需要评估助手同时支持对标准与实现的快速迭代。既然标准依赖于 LLM 输出(而非独立于它们),这就引出了一个问题:如何在其他各种「漂移」——如模型漂移(model drift)[4]、提示改动或链路上游变化——的背景下应对标准漂移。我们的发现还 (i) 强调了采用混合主动方法来对齐 LLM 辅助评估的必要性——这类方法须同时拥抱混乱与迭代;并 (ii) 提出了更宏观的问题:对评估助手而言,「与用户偏好对齐」究竟意味着什么。
We first position our work (Section 2) and present EvalGen’s design (Sec. 3) and implementation details (Sec. 4). We then present two evaluations: an off-line evaluation of our approach (Sec. 5), and a qualitative study with developers (Sec. 6 & 7). Finally, we suggest implications for future work (Sec. 8).
我们首先定位我们的工作(第 2 节),然后呈现 EvalGen 的设计(第 3 节)与实现细节(第 4 节)。接着我们呈现两项评估:对我们方法的离线评估(第 5 节),以及与开发者的定性研究(第 6、7 节)。最后,我们给出对未来工作的启示(第 8 节)。
2 动机与相关工作(Motivation and Related Work)
In response to the popularity of black-boxed LLMs like ChatGPT, prompt engineering (PE) has emerged as a new practice and research area. Alongside PE is the auditing of model behavior in practices such as “red-teaming,” used to identify harmful outputs in internal teams to tweak LLM behavior, usually prior to release [32, p.17]. These tasks have spurred the advent of new tools for “LLM operations” (hereafter called LLMOps) and new terminology such as “prompt template,” “chain of thought”, “chains,” etc.
随着 ChatGPT 这类黑盒 LLM 的流行,提示工程(prompt engineering,PE)作为一种新实践与研究方向兴起。与 PE 并行的是对模型行为的审计,如「红队测试(red-teaming)」——内部团队用它识别有害输出以调整 LLM 行为,通常在发布之前 [32, p.17]。这些任务催生了面向「LLM 运维(LLM operations,下文称 LLMOps)」的新工具,以及「提示模板(prompt template)」「思维链(chain of thought)」「链(chains)」等新术语。
[图 1: EvalGen's approach to assisting users in aligning evaluations. Users iterate through the process of refining criteria and grading. Note that LLM pipeline inputs and outputs are provided by our larger system, and outside the scope of this paper.]
图 1:EvalGen 辅助用户对齐评估的方法。用户在「精炼标准」与「打分」之间循环迭代。注意,LLM 流水线的输入与输出由我们更大的系统提供,不在本文范围内。(a) 典型评估流水线:输入经「被测提示」送给 LLM 产生输出,再由指标(评估器 LLM 的指标提示、代码指标等)打分得到测试结果,如此迭代;(b) EvalGen 评估流水线:LLM 先生成候选标准与候选断言,用户一边编辑标准、一边给 LLM 输出打分,系统选出选定断言并给出对齐成绩单。
提示评估的自动化(Automating Evaluations of Prompts)
When evaluating LLM behavior, users typically send off hundreds or thousands of queries to models. As users reach the limits of manual evaluation, users set up automated evaluation pipelines (Figure 1a) in code or with other LLMs (here we use the term LLM-based evaluators; other work uses terms such as “LLM-as-a-judge” [58] or “co-audit” [18]).[^fn1] Public PE tools like promptfoo [52] and ChainForge [1] allow users to write their own evaluation metrics to score LLM response quality, and support both code-based and LLM-based evaluators. For instance, in promptfoo users can write a rubric in a config file to specify how an LLM should evaluate responses, and may use pre-created grader prompt templates or customize them; an example is the assertion “The response is not apologetic.” Prototypes such as EvalLM [27] and PromptsRoyale [40] also support LLM evaluators, oftentimes exclusively, to help users compare between two prompts. Of PE tools, only EvalLM offers a way to help users calculate the alignment of LLM evaluators with their expectations, but this feature is mentioned only in authors’ Design section and is absent from their user study.[^fn2] At best, users of PE tools inspect LLM-generated evaluator outputs manually to double-check; at worst, the tool hides individual scores entirely. Regardless of aligning implementations of metrics with user preferences, even identifying what metrics to evaluate for custom tasks remains challenging for LLM practitioners, as evidenced by a recent study [37]. While many evaluation tools require users to declare metrics they care about, some prior work [45] and EvalGen employ LLMs to propose custom metrics based on prompts in the user’s LLM pipelines.
在评估 LLM 行为时,用户通常会向模型发出成百上千条查询。当用户达到人工评估的极限时,他们会搭建自动化评估流水线(图 1a)——用代码或用其他 LLM(这里我们使用「基于 LLM 的评估器(LLM-based evaluator)」一词;其他工作使用的术语有「LLM-as-a-judge(LLM 当裁判)」[58] 或「co-audit(协同审计)」[18])。[^fn1] promptfoo [52] 与 ChainForge [1] 等公开 PE 工具允许用户编写自己的评估指标为 LLM 回复质量打分,并同时支持代码型与 LLM 型评估器。例如,在 promptfoo 中,用户可以在配置文件里写一份评分细则(rubric),指定 LLM 应如何评估回复,还可以使用预置的评分提示模板或自行定制;一个例子是断言「回答不得有道歉语气」。EvalLM [27] 与 PromptsRoyale [40] 等原型也支持(且往往只支持)LLM 评估器,帮助用户在两个提示之间做比较。在各 PE 工具中,只有 EvalLM 提供了帮助用户计算 LLM 评估器与自身期望对齐度的功能,但该功能只在作者的设计章节中提及,并未出现在其用户研究中。[^fn2] 乐观地说,PE 工具的用户可以人工抽查 LLM 生成的评估器输出加以复核;悲观地说,工具干脆完全隐藏单项得分。抛开「让指标实现与用户偏好对齐」不谈,就连「该评估哪些指标」本身对 LLM 从业者也是难题,最近的一项研究 [37] 即为例证。虽然许多评估工具要求用户声明自己在意的指标,但一些先行工作 [45] 与 EvalGen 则用 LLM 基于用户 LLM 流水线中的提示来提出定制指标。
[^fn1]: 脚注 1 原文:An analog of this problem exists in software engineering as well: developing a set of assertions, often in the form of a set of unit tests or regression tests, that give developers confidence that their code is correct and that code changes do not (re)introduce bugs.(软件工程中也存在这一问题的类似物:开发一组断言——通常表现为一组单元测试或回归测试——让开发者确信代码正确、且代码改动不会(重新)引入缺陷。) [^fn2]: 脚注 2 原文:As of this writing, EvalLM is not publicly released.(截至本文写作时,EvalLM 尚未公开发布。)
对 LLM 行为的过度信任与过度泛化(Over-trust and Over-generalization of LLM Behavior)
That tools provide little assistance to validate evaluator quality is alarming, considering that other research shows people tend to over-rely and over-trust AI systems [3, 28, 31, 50]. For instance, in one high-profile incident, researchers from MIT posted a pre-print on arXiv claiming that GPT-4 could ace the MIT EECS exam. Within hours, work by Chowdhuri et al. debunked the study [5], citing problems arising from over-reliance on GPT-4 to grade itself. Other work has found further reasons to be cautious: LLMs asked to choose the best response from a set can be consistently biased by set ordering [30, 51]; and LLMs can be highly sensitive to seemingly innocuous formatting changes [43].
工具几乎不为验证评估器质量提供辅助,这一点令人担忧,因为其他研究表明人们倾向于过度依赖(over-rely)与过度信任 AI 系统 [3, 28, 31, 50]。例如,在一起高调事件中,MIT 的研究者在 arXiv 上发布预印本宣称 GPT-4 能轻松通过 MIT EECS 考试。数小时内,Chowdhuri 等人的工作 [5] 便推翻了该研究,指出问题正出在过度依赖 GPT-4 给自己评分。其他工作给出了更多需要警惕的理由:被要求从一组回复中选出最佳时,LLM 会受到集合排列顺序的一致性偏差影响 [30, 51];LLM 对看似无害的格式变化也可能高度敏感 [43]。
A related problem to over-reliance is over-generalization. Zamfirescu et al. [57] found that users unfamiliar with PE tend to over-generalize from single failures (causing them to throw out potentially good prompts), rather than having a holistic view of the overall performance of a prompt or chain. This was despite the fact that the interface had support for systematic testing. Similarly, Arawjo et al. [1] found that even people familiar with LLMs (developers, academics in ML) struggled to scale up their evaluations, appearing to over-generalize from a limited number of outputs even after an automated evaluation pipeline was setup. The authors identified three modes of PE on open-domain tasks, with the second, “limited evaluation,”’ characterized as users “prototyping an evaluation” [1], and suggested that future work focus on supported users in prototyping evaluation pipelines. Over-generalization is common in traditional ML, too—Kocielnik et al. [29] found that AI systems that showcase subsets of errors, like false positives or false negatives, that have the same accuracy, can lead to vastly different perceptions of accuracy.
与过度依赖相关的一个问题是过度泛化(over-generalization)。Zamfirescu 等 [57] 发现,不熟悉 PE 的用户倾向于从单次失败过度泛化(以致抛弃可能不错的提示),而不是对提示或链路的整体表现形成全局认识——尽管该界面其实支持系统性测试。类似地,Arawjo 等 [1] 发现,即使熟悉 LLM 的人(开发者、ML 学界人士)也难以扩展其评估规模:即便搭好了自动化评估流水线,他们看起来仍会从有限数量的输出过度概括。作者识别出开放域任务上 PE 的三种模式,其中第二种「有限评估(limited evaluation)」被刻画为用户「对评估做原型(prototyping an evaluation)」[1],并建议未来工作聚焦于支持用户对评估流水线做原型。过度泛化在传统 ML 中也很常见——Kocielnik 等 [29] 发现,展示具有相同准确率的不同错误子集(如假阳性或假阴性)的 AI 系统,会导致人们对准确率的感知截然不同。
对齐 LLM 的各类方法(Approaches to Aligning LLMs)
The HCI community has extensively studied interactive machine learning (iML). In iML, users iteratively develop models by selecting training examples, labeling data, and evaluating model performance [10]. Interfaces that facilitate seamless transitions between these activities result in fewer errors and outputs that better match users’ expectations [38, 46]. Some iML interfaces even use ML to assist users, for example, in scaling up labeling—reducing overall user effort required [8]. When using iML concepts for developing LLM pipelines, we must acknowledge a key challenge with LLMs: they often work with little to no specific training data [37]. Users may simply prototype with inputs they imagine the LLM would see, hoping the prompt generalizes.
HCI 社区已对交互式机器学习(interactive machine learning,iML)做了大量研究。在 iML 中,用户通过选择训练样例、标注数据、评估模型性能来迭代地开发模型 [10]。支持在这些活动之间无缝切换的界面能带来更少的错误、以及更符合用户期望的输出 [38, 46]。一些 iML 界面甚至用 ML 辅助用户,例如扩大标注规模——从而降低所需的总体用户工作量 [8]。将 iML 概念用于开发 LLM 流水线时,我们必须承认 LLM 的一个关键挑战:它们往往在极少甚至没有特定训练数据的情况下工作 [37]。用户可能只是用自己想象 LLM 会看到的输入做原型,寄希望于提示能泛化。
In the ML and NLP communities, researchers have explored many ways to align LLMs—and their evaluations—to specific user tasks. Many approaches rely on custom model training or fine-tuning [6], but all strategies heavily rely on humans to identify examples of desirable and undesirable outputs. For instance, Liu et al. [34] demonstrated using annotated LLM outputs—judged on criteria like consistency and relevance—as “few-shot examples” for calibrating LLM-based evaluators. Beyond classical summarization and NLP tasks, in response to the ad-hoc tedium of PE [56], academics and developers are building automated prompt optimization tools, maximizing some user-defined metric on a labeled set of examples. For instance, given some metrics and prompts, Khattab et al. [26] automatically run variations of inserted few-shot examples and LLM-generated rephrasings to optimize the prompt. Other work urges users to write assertions to guide outputs with a mix of code and natural language suggestions [42, 48], but writing these assertions is left up to developers, which is often time-consuming and error-prone. A broader point is that research in LLMOps optimization tends to come from the domains of NLP and ML, where authors generally validate tool performance against benchmark datasets with pre-defined metrics, leaving open the question of how well they perform in the wild on idiosyncratic user tasks (e.g. EvoPrompt, PromptBreeder, AutoCalibrate [12, 20, 34]). It thus remains unclear how to support developers in their prototyping of evaluations, with the problem becoming even more pressing as the popularity of prompt optimization increases.
在 ML 与 NLP 社区,研究者已探索出许多将 LLM 及其评估对齐到特定用户任务的方法。许多方法依赖定制模型训练或微调(fine-tuning)[6],但所有策略都在很大程度上依赖人类识别「合意与不合意输出」的示例。例如,Liu 等 [34] 演示了用带标注的 LLM 输出——按一致性、相关性等标准评判——作为校准 LLM 评估器的「少样本示例(few-shot examples)」。超越经典的摘要与 NLP 任务,为应对 PE 的临时性繁琐 [56],学界与业界正在构建自动提示优化工具,在带标注的样本集上最大化某个用户定义的指标。例如,给定一些指标与提示,Khattab 等 [26] 自动运行「插入的少样本示例」与「LLM 生成的改写」的变体来优化提示。另一些工作建议用户书写断言、以代码与自然语言建议的混合来引导输出 [42, 48],但书写这些断言仍留给开发者,往往耗时且易错。更大的一点是:LLMOps 优化的研究大多来自 NLP 与 ML 领域,作者们通常在带预定义指标的基准数据集上验证工具性能,而把它们在真实世界中千奇百怪的用户任务上的表现留作开放问题(如 EvoPrompt、PromptBreeder、AutoCalibrate [12, 20, 34])。因此,如何支持开发者对评估做原型仍不清楚,而随着提示优化日益流行,这一问题愈发紧迫。
Overall, this work reveals that users need more support for (a) prototyping evaluations and (b) validating evaluators of LLM outputs. It also reveals that auditing LLM outputs is far from easy, with humans prone to the dual biases of over-generalization and over-reliance. One recent LLM-assisted approach, SPADE [45], makes headway on these issues, helping developers generate Python assertion functions for LLM outputs from prompt history. Here we leverage a similar algorithmic approach to SPADE, but embed it inside an LLM-assisted user interface for evaluator prototyping, EvalGen, that also assists with criteria generation, measuring alignment with human preferences, and visualizing results.
总体而言,这些工作揭示出用户需要更多支持来 (a) 对评估做原型、(b) 验证 LLM 输出的评估器。它还揭示出审计 LLM 输出远非易事,人类容易陷入过度泛化与过度依赖的双重偏差。最近一种 LLM 辅助方法 SPADE [45] 在这些问题上取得了进展,帮助开发者从提示历史为 LLM 输出生成 Python 断言函数。本文借用与 SPADE 类似的算法思路,但把它嵌入一个面向评估器原型化的 LLM 辅助用户界面 EvalGen,后者还辅助标准生成、度量与人类偏好的对齐以及结果可视化。
3 EvalGen 设计(EvalGen Design)
In designing EvalGen, our goal was (1) to investigate how to assist developers in creating evaluators to grade LLM outputs, and (2) to help them “validate the validators” through both automated assistance and transparency around how aligned each evaluator is with their expectations. As we covered in Section 2, emerging practices in prompt engineering, LLM auditing, and prompt optimization involve the writing of evaluation functions (metrics) to automate grading. These functions may be code- or LLM-based. Based on this context, we set out to design an LLM-powered evaluation assistant that provided developers control over metric criteria, evaluator type (code or LLM), and implementation (i.e., function) generation and selection processes, without asking them to come up with criteria or write code or grader prompts themselves.
在设计 EvalGen 时,我们的目标是 (1) 研究如何辅助开发者创建为 LLM 输出打分的评估器,(2) 通过自动化辅助与「每个评估器与用户期望对齐程度」的透明化,帮助他们「验证验证者」。如第 2 节所述,提示工程、LLM 审计与提示优化的新兴实践都涉及编写评估函数(指标)以自动化打分。这些函数可以是代码型或 LLM 型。基于这一背景,我们着手设计一个由 LLM 驱动的评估助手:让开发者对指标标准、评估器类型(代码或 LLM)以及实现(即函数)的生成与选择过程保有控制权,同时不要求他们自己构思标准、编写代码或评分提示。
3.1 EvalGen 工作流(EvalGen Workflow)
We implemented EvalGen in an existing open-source system for prompt engineering, ChainForge [1], which handles querying multiple LLMs with parametrized prompts, running code- and LLM-based evaluators, plotting scores, chaining, etc. In ChainForge, users write LLM pipelines by creating nodes of various types to represent their dataflow, such as an “input” node feeding into a “prompt” node. We discuss here only our extension, chiefly a pop-up screen that helps the user define, implement, and validate evaluation functions. We also implemented a new node, Multi-Eval, that allows users to include multiple evaluators in a single node and run all evaluators on the outputs of the pipeline’s previous node. Finally, we made improvements to plotting per-criteria scores in the Table View of the LLM output inspector, which can be accessed via the Multi-Eval node. Fig. 1b provides a high-level overview of the EvalGen architecture compared to the typical LLM output evaluation pipeline; we discuss implementation details in Sec. 4.
我们将 EvalGen 实现于一个现有的开源提示工程系统 ChainForge [1] 之上,后者负责以参数化提示查询多个 LLM、运行代码型与 LLM 型评估器、绘制分数、链式编排等。在 ChainForge 中,用户通过创建不同类型的节点来表示数据流、从而编写 LLM 流水线,例如一个「输入(input)」节点接入一个「提示(prompt)」节点。我们在此只讨论自己的扩展:主要是一个帮助用户定义、实现并验证评估函数的弹出式界面。我们还实现了一个新节点 Multi-Eval,允许用户在单个节点中包含多个评估器,并对流水线上一节点的全部输出运行这些评估器。最后,我们改进了在 LLM 输出检查器的表格视图(Table View)中绘制每标准得分的功能,该视图可经 Multi-Eval 节点访问。图 1b 给出了 EvalGen 架构与典型 LLM 输出评估流水线的高层对比;实现细节见第 4 节。
[图 2: The workflow of our EvalGen prototype, from (a) a Prompt Node attached to an empty Multi-Eval Node, showing a Generate Criteria button; (b) the pop-up EvalGen Wizard with three options, Infer, Manual, and Grade First; (c) the Pick Criteria screen, allowing users to describe criteria in natural language and toggle Code or LLM implementations; (d) the Grade screen, with the LLM output (top), input variables (left), and prompt (right), Good and Bad grade buttons, and an “I’m Tired” button (bottom-right) to finish; and finally (e) the Report Card screen, showing the alignment of each criteria and across criteria. Hovering over the alignment shows a confusion matrix. Note that some descriptions and elements have been clipped for space.]
图 2:我们 EvalGen 原型的工作流:(a) 附着于空 Multi-Eval 节点的提示节点,显示一个 Generate Criteria(生成标准)按钮;(b) 弹出的 EvalGen 向导,含 Infer(推断)、Manual(手动)、Grade First(先打分)三个选项;(c) Pick Criteria(挑选标准)界面,允许用户用自然语言描述标准并在代码/LLM 实现之间切换;(d) Grade(打分)界面,上方为 LLM 输出、左侧为输入变量、右侧为提示,配有 Good 与 Bad 打分按钮,以及右下角用于结束的「I'm Tired(我累了)」按钮;最后是 (e) Report Card(成绩单)界面,展示每条标准及跨标准的对齐情况。悬停对齐度可查看混淆矩阵。注意部分描述与元素因篇幅被裁剪。
Figure 2 depicts the workflow of in the context of the EvalGen interface, excluding returning to the main workflow with selected implementations and using the Table View to inspect scores. EvalGen assists a developer in engineering an evaluation of LLM outputs for a single prompt template. First, EvalGen is accessed as a button on a “Multi-Eval” node we added to ChainForge, which is attached to a Prompt Node (Fig. 2a). A Wizard opens, depicting three options (Fig. 2b): Infer, Manual, and Grade First. A description of EvalGen (not shown) appears above the options. Clicking Infer or Manual leads to the Pick Criteria screen (Fig. 2c); clicking Grade First leads to the Grading screen (Fig. 2d) and asks users to grade at least five outputs, before continuing to the Pick Criteria screen.
图 2 描绘了 EvalGen 界面语境下的工作流(不含带着选定实现返回主工作流、以及用表格视图检查得分)。EvalGen 辅助开发者为单个提示模板工程化一套 LLM 输出评估。首先,EvalGen 通过我们添加到 ChainForge 的「Multi-Eval」节点上的按钮进入,该节点附着于一个提示节点(图 2a)。随后打开一个向导,给出三个选项(图 2b):Infer、Manual 与 Grade First。选项上方有 EvalGen 的描述(图中未示)。点击 Infer 或 Manual 进入 Pick Criteria(挑选标准)界面(图 2c);点击 Grade First 则先进入打分界面(图 2d),要求用户至少给 5 条输出打分,然后继续进入挑选标准界面。
The Pick Criteria interface is depicted in Fig. 2c. An LLM has generated criteria suggestions in natural language (Sections 4.1 and 4.2), along with a toggle to prefer a Python code-based or LLM-based evaluator. The user can edit all parts—including the titles or descriptions and type of evaluator—or add new criteria not suggested by the LLM. They can also delete criteria or deselect criteria as needed. Pressing “Implement It” passes the criteria to a second LLM that generates candidate implementations.
挑选标准界面如图 2c 所示。一个 LLM 已用自然语言生成了标准建议(第 4.1、4.2 节),并带有「偏好 Python 代码型或 LLM 型评估器」的切换开关。用户可以编辑所有部分——包括标题、描述与评估器类型——或添加 LLM 未建议的新标准;也可以按需删除或取消选择标准。按下「Implement It(实现它)」后,标准被传给第二个 LLM 来生成候选实现。
While implementations are generated and executed on LLM outputs, users are asked to grade outputs. EvalGen uses these grades to implementations with their preferences. Fig. 2d depicts the Grading screen. A single LLM response is presented to the user, centered in focus in the grader window. The context of the prompt and any input variables (vars) is also present. The user grades outputs with the Good and Bad buttons. Since it may be time-consuming to ask the developer to grade on a per-criterion basis, for the grader interface we decided on the simplicity of thumbs-up/down scoring. Such scoring is a noisy yet informative signal of quality—if a response is given a thumbs-up, it is assumed to pass all criteria, and so if a candidate assertion fails on that response, the candidate is down-ranked in the pool (details in Section 4.1). Users may also click arrows to navigate through outputs (for instance, if they are unsure about a grade, or want to revise a prior grade).[^fn3]
在实现被生成并在 LLM 输出上执行的同时,系统请用户给输出打分。EvalGen 用这些打分来处理「实现与其偏好」的关系(译注:此句原文疑漏一词,按上下文意为「选出与其偏好对齐的实现」)。图 2d 展示了打分界面。单条 LLM 回复居中呈现在打分窗口的焦点处,提示及任何输入变量(vars)的上下文也在场。用户用 Good 与 Bad 按钮给输出打分。由于请开发者逐条标准打分可能很耗时,打分界面采用了最简的点赞/点踩二元评分。这种评分是质量的一个有噪声却信息丰富的信号——若一条回复被点赞,则假定它通过全部标准;于是若某候选断言在该回复上失败,该候选就在候选池中被降权(细节见第 4.1 节)。用户也可以点击箭头在各输出之间导航(例如当他们不确定某个打分、或想修改先前打分时)。[^fn3]
[^fn3]: 脚注 3 原文:Initially, we had asked users to grade a preset number of outputs and a bar showed their progress. However, calling LLMs and executing assertions are asynchronous operations that take an indeterminate amount of time: suggesting an “end point” to user grading may lose valuable information when the user still has to wait for generations to return. The user may also find grading enjoyable or important. For these reasons, we did not seek to limit user grading. However, we kept this number-left progress bar in the Grade First screen (accessed via Fig.2b).(最初,我们曾要求用户给预设数量的输出打分,并用一个进度条显示进度。然而,调用 LLM 与执行断言都是耗时不确定的异步操作:给用户打分暗示一个「终点」,可能在用户仍需等待生成返回时丢失宝贵信息。用户也可能觉得打分愉快或重要。出于这些原因,我们没有设法限制用户打分。不过,在 Grade First 界面(经图 2b 进入)中我们保留了这个「剩余数量」进度条。)
Finally, after the user is done grading and all candidate implementations are generated, executed, and filtered for alignment with grades, a Report Card screen appears with feedback on per-criteria and aggregate measures of alignment with user grades (Fig. 2e). Hovering over per-criteria metrics shows a confusion matrix of how aligned that particular criterion is to the human grades, while the aggregate metrics show the coverage and false failure rate (see Section 5) of the selected subset of EvalGen-generated assertions. The user then returns to the main ChainForge interface (not shown), where the selected implementations are available in a “Multi-Eval” node, titled by criteria. The user can edit or add more criteria, inspect and visualize evaluation results (Fig. 3), etc.; however, this is outside the scope of our design discussion.
最后,当用户打分完毕、所有候选实现都已生成、执行并按与打分的对齐度过滤后,会出现成绩单界面,给出每标准与整体上与用户打分对齐度的反馈(图 2e)。悬停单项指标可查看该标准与人类打分对齐情况的混淆矩阵,而整体指标显示 EvalGen 所生成断言中选定子集的覆盖率(coverage)与假失败率(false failure rate,见第 5 节)。随后用户回到 ChainForge 主界面(未示出),所选实现以标准为标题出现在「Multi-Eval」节点中。用户可以编辑或添加更多标准、检查与可视化评估结果(图 3)等;不过这超出了我们设计讨论的范围。
Our design reflects trade-offs between developer effort and robust human verification of LLM-generated metrics. The human cannot completely validate an LLM-based evaluator: the point of LLM evaluators is to reduce the effort required by the developer, who would otherwise have to grade outputs manually. The only way to fully align an LLM evaluator would be to ask the user to label all outputs; obviously, this defeats the purpose. Asking the developer to grade some outputs, using some time they would have spent waiting anyway, is the key idea behind our design.
我们的设计反映了开发者投入与「LLM 生成指标之鲁棒人类验证」之间的权衡。人类无法完全验证一个基于 LLM 的评估器:LLM 评估器的意义就在于减轻开发者的负担——否则开发者只能人工给输出打分。完全对齐一个 LLM 评估器的唯一办法是让用户标注全部输出;显然,这就违背了初衷。请开发者利用一部分本来就要用来等待的时间给一些输出打分,是我们设计背后的关键思想。
4 实现(Implementation)
4.1 系统架构(System Architecture)
Like prior work on evaluator assistants [27, 45], our solution decomposes evaluations into criteria and assertions (boolean functions that implement the criteria, evaluating outputs). We employ LLMs in generating criteria, based on the prompt [45], and in generating various candidate implementations of each criterion [27, 45]. As users grade, we rank candidate assertions that implement each criteria based on their alignment with user grades (see Appendix A.3 for how we define alignment). At a high level, alignment is a combination of the assertion’s coverage, or ability to fail outputs that the user thinks are bad, and the false failure rate, or ability to not fail outputs that the user thinks are good. We give a formal definition of assertion alignment in Appendix A.3.
与评估器助手的先行工作 [27, 45] 一致,我们的方案把评估分解为标准(criteria)与断言(assertion,实现标准、对输出求值的布尔函数)。我们用 LLM 基于提示生成标准 [45],并为每条标准生成多个候选实现 [27, 45]。在用户打分的同时,我们依据候选断言与用户打分的对齐度对其排序(对齐度的定义见附录 A.3)。概言之,对齐度是断言的覆盖率——让用户认为坏的输出失败的能力——与假失败率——不让用户认为好的输出失败的能力——的组合。断言对齐度的形式化定义见附录 A.3。
EvalGen’s architecture differs from prior work in two main components: first, EvalGen solicits grades from the user on a sample of LLM outputs—requiring some policy to sample LLM outputs to grade. Second, in contrast to SPADE [45], which operates offline and solves an integer linear program to generate the optimal assertion set, EvalGen employs an online (i.e., streaming) system architecture to progressively optimize for the most aligned assertion set. When EvalGen generates a new candidate implementation, it immediately executes this implementation on the set of LLM outputs, culling implementations that are obviously bad (e.g., Python functions with runtime errors). EvalGen maintains a dynamic estimate of selectivity (i.e., pass rate) for each candidate assertion, which in turn informs how grades are sampled in the interface. Our system, as depicted in Figure 1b, is structured into three components:
EvalGen 的架构在两个主要组件上不同于先行工作:其一,EvalGen 就一个 LLM 输出样本征求用户打分——这需要某种策略来采样待打分的 LLM 输出。其二,与离线运行、通过求解整数线性规划来生成最优断言集的 SPADE [45] 不同,EvalGen 采用在线(即流式)系统架构来渐进地优化出对齐度最高的断言集。当 EvalGen 生成一个新的候选实现时,它会立即在 LLM 输出集上执行该实现,剔除明显糟糕的实现(如运行时报错的 Python 函数)。EvalGen 为每个候选断言维护一个动态的选择率(selectivity,即通过率)估计,它反过来指导界面中打分的采样方式。如图 1b 所示,我们的系统由三个组件构成:
Criteria Suggestion. We use an GPT-4 to propose various binary evaluation criteria in natural language, such as response length or tone. Developers can select from these suggestions or add their own criteria. For each criterion, developers can select whether it should be evaluated with a purely code-based function or a function that involves calls to another LLM.
标准建议(Criteria Suggestion)。我们用 GPT-4 以自然语言提出各种二元评估标准,如回复长度或语气。开发者可以从这些建议中选择,也可以添加自己的标准。对每条标准,开发者可以选择用纯代码函数、还是一个涉及调用另一个 LLM 的函数来评估。
Candidate Assertion Synthesis and Execution. Based on the selected criteria, we use GPT-4 to asynchronously generate one or more candidate assertions as code or a grader prompt to be evaluated by an LLM. For each criterion, we issue one call to GPT-4 to generate multiple candidate assertions within markers in a streaming fashion. Every time we detect the end of marker in any GPT-4 response, we parse the candidate assertion and submit it to EvalGen’s executor, which will run it on LLM pipeline outputs. Generating multiple candidate assertions improves the probability that there is at least one implementation that aligns with developer expectations. Moreover, for code-based assertions, LLMs occasionally synthesize erroneous functions (e.g., hallucinating a function in a Python library), requiring several candidate assertions.
候选断言合成与执行(Candidate Assertion Synthesis and Execution)。基于所选标准,我们用 GPT-4 异步地生成一个或多个候选断言,形式为代码或由 LLM 评估的评分提示。对每条标准,我们发起一次 GPT-4 调用,以流式方式在标记(markers)内生成多个候选断言。每当在任何 GPT-4 响应中检测到标记结束,我们就解析出该候选断言并提交给 EvalGen 的执行器,由后者在 LLM 流水线输出上运行它。生成多个候选断言可提高「至少存在一个与开发者期望对齐的实现」的概率。此外,对代码型断言而言,LLM 偶尔会合成有错误的函数(如幻觉出某个 Python 库里的函数),因此需要多个候选断言。
Grading Sampler. This component samples LLM pipeline outputs for the user to give binary feedback on (thumbs up/down).
打分采样器(Grading Sampler)。该组件采样 LLM 流水线输出,请用户给出二元反馈(点赞/点踩)。
When the user grades an LLM output, we update internal estimates of alignment for each candidate assertion, and we sample the next output for the user to grade.
当用户给某条 LLM 输出打分后,我们更新每个候选断言对齐度的内部估计,并采样下一条请用户打分的输出。
Once the user does not want to grade LLM outputs anymore, or is finished grading all outputs, for each criterion, we select the candidate assertion with the highest alignment with the user’s grades. The user can provide a threshold for the false failure rate (as defined in Section 5) such that EvalGen only selects assertions that do not exceed this threshold.
一旦用户不想再给 LLM 输出打分、或已打完全部输出,我们就为每条标准选出与用户打分对齐度最高的候选断言。用户可以为假失败率提供一个阈值(定义见第 5 节),使 EvalGen 只选择不超过该阈值的断言。
4.2 选择断言与征求打分(Selecting Assertions & Eliciting Grades)
Once the user selects criteria, EvalGen’s executor begins generating candidate assertions and executing them on all LLM outputs. EvalGen maintains dynamic estimates for the following:
一旦用户选定标准,EvalGen 的执行器便开始生成候选断言并在全部 LLM 输出上执行它们。EvalGen 维护以下动态估计:
Selectivity of Candidate Assertions. The selectivity is the probability that an assertion will classify an LLM output as passing. This probability is adjusted each time EvalGen processes the outcome of executing a candidate assertion on an LLM output.
候选断言的选择率(Selectivity of Candidate Assertions)。选择率是一条断言把 LLM 输出判为通过的概率。每当 EvalGen 处理完一次「候选断言在 LLM 输出上执行」的结果,该概率就被调整。
Confidence Scores for Potentially Poor Outputs. These scores estimate the likelihood that an LLM output is of low quality, without having been explicitly evaluated by the user. The scores are dependent on assertion selectivity and are revised whenever EvalGen evaluates a new assertion against an LLM output, or when a user grades an LLM output directly.
潜在劣质输出的置信分(Confidence Scores for Potentially Poor Outputs)。这些分数估计某条 LLM 输出质量低下的可能性,而无需用户显式评估过它。这些分数依赖于断言选择率,并在 EvalGen 对一条 LLM 输出评估新断言、或用户直接给某条 LLM 输出打分时被修正。
Assertion Alignment. Alignment, or the harmonic mean of coverage and false failure rate (Appendix A.3), is recalculated for each assertion every time the user grades an LLM pipeline output.
断言对齐度(Assertion Alignment)。对齐度,即覆盖率与假失败率的调和平均(附录 A.3),在用户每次给一条 LLM 流水线输出打分时对每个断言重算。
See Appendix A for a complete description of assertion selectivity and how it impacts confidence scores; how EvalGen uses these confidence scores to sample grades from the user; formal definitions of alignment; and how EvalGen determines the resulting assertion set based on alignment with grades.
关于断言选择率及其对置信分的影响;EvalGen 如何用这些置信分向用户采样打分;对齐度的形式化定义;以及 EvalGen 如何依据与打分的对齐度确定最终断言集——完整描述见附录 A。
5 算法评估(Algorithm Evaluation)
Here we present results on the effectiveness of EvalGen’s selection algorithm. Our experiment aimed to understand how soliciting human input at the criteria suggestion stage impacts the size (number of assertions) and alignment of the resulting assertion set. We compared to a baseline, SPADE [45], a fully automated system that generates criteria and candidate assertions and chooses the minimal assertion set that meet coverage and false failure rate constraints.
这里我们呈现 EvalGen 选择算法有效性的结果。我们的实验旨在理解「在标准建议阶段征求人类输入」如何影响所得断言集的规模(断言数量)与对齐度。我们与基线 SPADE [45] 做了对比——那是一个全自动系统,生成标准与候选断言,并选出满足覆盖率与假失败率约束的最小断言集。
5.1 评估设置(Evaluation Setup)
We developed two LLM pipelines based on real-world datasets. The medical pipeline operates on a dataset of 84 unstructured text transcripts from doctor-patient calls [54], aiming to extract specific information (e.g., symptoms, medication) without revealing any personally identifiable information (PII). This task requires assertions to ensure compliance with privacy laws. The product pipeline involved crafting SEO-friendly descriptions for 100 Amazon products and their reviews [21]. We selected this task because it mirrors actual LLM applications (there are a number of startups using AI to write SEO-optimized product descriptions), and it benefits from assertions: for example, even if there are negative reviews, the descriptions should not say negative things about the products, which would adversely affect the products’ sales potential. Our prompts are presented in Appendix B. For both prompts, the placeholder variables (i.e., transcript and document) represent the input context to inject at pipeline runtime. For each dataset, we executed the corresponding pipeline once for each input using OpenAI’s GPT-3.5-Turbo to generate outputs—resulting in 84 medical LLM outputs and 100 product LLM outputs.
我们基于真实世界数据集构建了两条 LLM 流水线。医疗流水线处理一个含 84 份医患通话非结构化文本转写的数据集 [54],目标是抽取特定信息(如症状、用药)且不泄露任何个人身份信息(personally identifiable information,PII)。该任务需要断言以确保遵守隐私法规。商品流水线为 100 个 Amazon 商品及其评论撰写 SEO 友好的描述 [21]。我们选择该任务是因为它映射真实的 LLM 应用(有不少初创公司用 AI 撰写 SEO 优化的商品描述),而且它受益于断言:例如,即使存在差评,描述也不应说产品的坏话,否则会损害产品的销售潜力。我们的提示见附录 B。对两个提示而言,占位变量(即 transcript 与 document)代表流水线运行时注入的输入上下文。对每个数据集,我们用 OpenAI 的 GPT-3.5-Turbo 对每条输入各执行一次相应流水线来生成输出——得到 84 条医疗 LLM 输出与 100 条商品 LLM 输出。
Two of the paper authors manually graded all LLM outputs to establish ground-truth labels. Overall, 68% and 51% of the outputs were good for the medical and product LLM pipelines, respectively. Common issues included the presence of personal information in the medical pipeline outputs and bad reviews or lengthy content in the product pipeline outputs.
两位论文作者人工为全部 LLM 输出打分以建立真值(ground-truth)标签。总体上,医疗与商品 LLM 流水线分别有 68% 与 51% 的输出为「好」。常见问题包括:医疗流水线输出中出现个人信息,商品流水线输出中出现差评或内容冗长。
5.2 标准生成步骤中人类输入的影响(Impact of Human Input in the Criteria Generation Step)
Here, we report quantitative and qualitative differences in SPADE’s and EvalGen’s assertion sets. There are two differences between SPADE and EvalGen in how they generate assertion sets. The first difference is that EvalGen asks the user to add, edit, or remove criteria before generating different candidate assertions, whereas SPADE does not solicit any input from the user about the criteria. The second difference is in the selection of the assertions themselves: given user-confirmed criteria and a sample of grades provided in a UI, EvalGen picks the most aligned assertion per criterion that meets some false failure rate threshold. Meanwhile, SPADE solves an optimization problem to select a minimal assertion set that meet a false failure rate threshold and cover all SPADE-generated criteria.
这里我们报告 SPADE 与 EvalGen 断言集在定量与定性上的差异。两者在生成断言集的方式上有两点不同。其一,EvalGen 请用户在生成不同候选断言之前增删改标准,而 SPADE 完全不就标准征求用户输入。其二在于断言本身的选择:给定用户确认的标准与在 UI 中提供的一份打分样本,EvalGen 为每条标准挑选满足某假失败率阈值的、对齐度最高的断言;而 SPADE 求解一个优化问题,选出一个满足假失败率阈值并覆盖全部 SPADE 生成标准的最小断言集。
5.2.1 Evaluation Procedure. We first ran SPADE end-to-end for both LLM pipelines, supplying all labeled LLM outputs to see the resulting assertion sets. We initially set both false failure rate thresholds to 10%. While SPADE met this threshold for the medical pipeline, the product pipeline required adjusting the false failure rate to 40% to find a viable assertion set. This illustrates the challenge of balancing coverage with false failures, underscoring the need for evaluator systems to effectively navigate these trade-offs. Subsequently, we ran EvalGen for both pipelines with the same false failure rate thresholds. For the medical pipeline, we defined three evaluation criteria: word count, presence of the six targeted keys, and absence of PII, with the first two implemented via code-based assertions and the last via an LLM evaluator. The product pipeline criteria included absence of negative reviews, absence of links, adherence to markdown format, and word count limitation, with only the first criterion requiring LLM implementation. To create the aligned assertion sets, we provided EvalGen with 16 graded outputs per pipeline instead of all 80-100 graded outputs—given the impracticality of expecting users to extensively grade in a single session.
5.2.1 评估流程(Evaluation Procedure)。我们首先对两条 LLM 流水线端到端运行 SPADE,提供全部带标注的 LLM 输出以查看所得断言集。我们最初把两个假失败率阈值都设为 10%。SPADE 在医疗流水线上达到了该阈值,而商品流水线需要把假失败率放宽到 40% 才能找到可行的断言集。这说明了在覆盖率与假失败之间取得平衡之难,凸显了评估器系统有效驾驭这些权衡的必要性。随后,我们以相同的假失败率阈值为两条流水线运行 EvalGen。医疗流水线方面,我们定义了三条评估标准:字数、六个目标键的存在、以及不出现 PII——前两条用代码型断言实现,最后一条用 LLM 评估器实现。商品流水线的标准包括:不出现负面评论、不出现链接、遵循 Markdown 格式、字数限制——只有第一条需要 LLM 实现。为构造对齐的断言集,我们为每条流水线向 EvalGen 提供 16 条已打分输出,而非全部 80–100 条已打分输出——因为期待用户在单次会话中大量打分并不现实。
5.2.2 Results. Our results in Table 1 show that EvalGen, by incorporating human judgment during criteria selection, achieved equal or better alignment than SPADE with fewer assertions for both pipelines. Specifically, in the product pipeline, EvalGen produced an assertion set less than half the size of SPADE’s and increased coverage from 49% to 73%. This underscores the benefit of involving humans in selecting criteria, as a fully automated tool may generate assertions for criteria that humans may not actually care about writing assertions for. In the medical pipeline, SPADE added unnecessary assertions, such as one for a neutral tone, not chosen for EvalGen, and split checks for specific keys into more criteria than needed. For the product pipeline, SPADE generated twice the number of assertions compared to EvalGen, some of which were unrealistic, like a Python function designed to flag specific negative phrases such as “never order” and “disappointed” in the output. In contrast, EvalGen returned a more pragmatic assertion for this criterion—an LLM-based validator to ensure the product descriptions remained entirely positive.
5.2.2 结果(Results)。表 1 的结果显示,通过在标准选择阶段引入人类判断,EvalGen 在两条流水线上都以更少的断言取得了与 SPADE 相同或更好的对齐度。具体而言,在商品流水线上,EvalGen 产出的断言集规模不到 SPADE 的一半,并把覆盖率从 49% 提高到 73%。这凸显了让人类参与标准选择的好处:一个全自动工具可能会为人类其实并不想为其书写断言的标准生成断言。在医疗流水线上,SPADE 添加了不必要的断言,比如一条 EvalGen 未选择的「语气中性」断言,并把对特定键的检查拆成多于所需的标准。在商品流水线上,SPADE 生成的断言数量是 EvalGen 的两倍,其中一些并不现实,比如一个专门标记输出中「never order」「disappointed」等特定负面短语的 Python 函数。相比之下,EvalGen 为该标准返回了一个更务实的断言——一个确保商品描述保持全然正面的 LLM 型校验器。
表 1(Table 1):EvalGen 与 SPADE 在各流水线上的对比
| 指标 | 医疗流水线 EvalGen | 医疗流水线 SPADE | 商品流水线 EvalGen | 商品流水线 SPADE |
|---|---|---|---|---|
| 数据集大小(Dataset Size) | 84 | 84 | 100 | 100 |
| 坏输出数(# Bad Outputs) | 27 | 27 | 49 | 49 |
| 断言数(# Assertions) | 3 | 5 | 4 | 9 |
| 覆盖率(Coverage) | 0.33 | 0.33 | 0.73 | 0.49 |
| 假失败率(FFR) | 0.10 | 0.10 | 0.39 | 0.39 |
| 对齐度(Alignment,%) | 48.29 | 48.29 | 66.46 | 54.35 |
表 1:EvalGen 与 SPADE 在各流水线上的对比。在标准阶段引入用户输入后,EvalGen 以更少的函数取得相同或更高的对齐度。
6 用户研究(User Study)
To understand how developers might use EvalGen to build evaluators for LLM pipelines, we asked nine industry practitioners who had prior experience with LLMs to use EvalGen and think aloud.
为理解开发者可能如何用 EvalGen 为 LLM 流水线构建评估器,我们请九位先前有 LLM 经验的产业从业者使用 EvalGen 并出声思考。
Recruitment and Participants. We recruited nine industry practitioners via a Twitter post, calling for anyone interested in solving the problem of “who validates the validators.” We selected the first nine who had experience coding and building LLM pipelines for some company or product. Participants included software engineers, ML scientists, startup executives, and independent consultants. We focused on developers with experience with LLMs because they can best intuit what features of an assertion assistant they would find useful, compared to their existing workflows.
招募与参与者(Recruitment and Participants)。我们通过一条 Twitter 帖子招募了九位产业从业者,帖子号召任何有兴趣解决「谁来验证验证者」问题的人。我们从最先报名、具备为公司或产品编写代码与搭建 LLM 流水线经验的人中选出前九位。参与者包括软件工程师、ML 科学家、初创公司高管与独立顾问。我们聚焦于有 LLM 经验的开发者,因为相较于他们现有的工作流,他们最能凭直觉判断一个断言助手的哪些功能对自己有用。
Procedure. All studies were conducted over Zoom. We first spent 5 minutes establishing rapport and asking participants about their backgrounds and experience. We then introduced the participants to our LLM pipeline in ChainForge, where the pipeline’s task was to do named entity recognition (NER) on a dataset of tweets. The LLM used in our pipeline was GPT-3.5-Turbo from OpenAI. The prompt for our LLM pipeline was as follows: You will be doing named entity recognition (NER). Extract up to 3 well-known entities from the following tweet: {tweet_full_text} For each entity, write one sentence describing the person or entity. All the entities you extract should be found in a knowledge base like Wikipedia, so don’t make up entities. Return your answer as a bulleted Markdown list, where each bullet is formatted as ‘- entity: description‘. Do not extract hashtags as entities. We chose this task and set of inputs for three reasons: first, NER is a common, real-world task that language models excel at; second, tweets are short and can be displayed in a UI without any scrolling; third, “NER for tweets” is a problem that many people have studied [15, 33, 49]. We allowed participants to change the task or prompt if they wanted.
流程(Procedure)。所有研究均通过 Zoom 进行。我们先花 5 分钟建立融洽关系并询问参与者的背景与经验。然后向参与者介绍我们在 ChainForge 中的 LLM 流水线,其任务是对一个推文数据集做命名实体识别(named entity recognition,NER)。流水线使用的 LLM 是 OpenAI 的 GPT-3.5-Turbo。我们 LLM 流水线的提示如下:「你将做命名实体识别(NER)。从下面的推文中最多抽取 3 个知名实体:{tweet_full_text}。对每个实体,写一句话描述该人物或实体。你抽取的所有实体都应能在维基百科等知识库中找到,所以不要编造实体。以 Markdown 项目符号列表返回你的答案,每条格式为『- entity: description』。不要把话题标签抽取为实体。」我们选择该任务与这组输入出于三个原因:第一,NER 是语言模型擅长的常见真实任务;第二,推文很短,可以不加滚动地显示在 UI 中;第三,「推文 NER」是许多人研究过的问题 [15, 33, 49]。我们允许参与者按意愿更改任务或提示。
Next, we spent a few minutes describing EvalGen’s functionality. We showed participants how to view LLM outputs, trigger EvalGen, write an assertion from scratch and add it to their set, run the assertion set on all the LLM outputs, and inspect the Table View plotting results of each assertion on each LLM output. We then gave the participant remote control access to our screen, instructing them to come up with an assertion set for the pipeline. Participants were allowed up to 40 minutes to explore the tool while thinking aloud. We communicated that we were mainly interested in observing their process of creating assertions, not interacting with other features of ChainForge such as comparing different LLM APIs. If the participant had any questions about the interface, we answered them.
接下来,我们用几分钟介绍 EvalGen 的功能。我们向参与者演示了如何查看 LLM 输出、触发 EvalGen、从零编写一条断言并加入其断言集、在全部 LLM 输出上运行断言集、以及检查为每条断言在每条 LLM 输出上绘制结果的表格视图。然后我们把屏幕的远程控制权交给参与者,请他们为流水线构思一套断言集。参与者最多可用 40 分钟边出声思考边探索工具。我们说明:我们主要感兴趣的是观察他们创建断言的过程,而不是与 ChainForge 其他功能(如比较不同 LLM API)的交互。若参与者对界面有疑问,我们会解答。
After the participant had an assertion set they liked, or ran out of time, we asked them open-ended questions. We first asked them to comment on EvalGen’s approach to generating assertions, and whether they felt that the assertions aligned with their grades. We then asked follow-up questions to learn more about why they felt a particular way or found some aspect of assertion alignment challenging. We limited the post-interview to 10 minutes. At the end, we asked participants to rate how they felt about the assertions’ alignment on a 7-point Likert scale (1 = strongly disagree, 7 = strongly agree). Overall, the study ranged from 45min to 1h15min. Our study was approved by our institutional review board (IRB), and participants generously volunteered their time.
当参与者得到满意的断言集或时间用尽后,我们提出开放性问题。我们先请他们评论 EvalGen 生成断言的方法,以及他们是否感到断言与自己的打分对齐。然后我们提后续问题,以更多了解他们为何有特定感受、或发现断言对齐的哪些方面有挑战。我们将访谈后段限制在 10 分钟。最后,我们请参与者在 7 点李克特量表(Likert scale,1 = 强烈不同意,7 = 强烈同意)上为「他们对断言对齐情况的感受」评分。总体上,研究时长从 45 分钟到 1 小时 15 分钟不等。我们的研究经机构审查委员会(institutional review board,IRB)批准,参与者慷慨地贡献了志愿时间。
Analysis. We asked participants to think aloud while using the tool, while we took notes on their thoughts and any visible emotions (e.g., delight when EvalGen suggested a criterion they struggled to externalize, or frustration when they could not find a good assertion for a criterion). We also recorded the transcripts for each video call. We employed open and axial coding [13] to identify common themes across the transcripts and notes for each participant. Initially, we coded individual sentences of interest for each participant, then grouped these into broader themes on a per-participant basis in a second pass of coding. Finally, we consolidated these themes across all participants, reorganizing them as needed.
分析(Analysis)。我们请参与者在使用工具时出声思考,同时我们记录他们的想法与任何可见情绪(例如当 EvalGen 建议了一条他们难以外化的标准时的欣喜,或当他们找不到某标准的好断言时的沮丧)。我们还录下了每次视频通话的转录稿。我们采用开放编码与轴心编码(open and axial coding)[13] 在每位参与者的转录与笔记中识别共同主题。最初,我们为每位参与者对感兴趣的单句编码;然后在第二轮编码中按参与者归组成更宽的主题。最后,我们跨所有参与者合并这些主题,并按需重新组织。
7 用户研究发现(User Study Findings)
Overall, we found that:
• Participants felt that EvalGen was a great starting point for assertions, and wanted to—and could—exercise control over EvalGen’s assistance.
• Participants struggled to align assertions with their preferences due to two main challenges in grading: (i) some criteria are difficult for humans to grade (e.g., under a target word count), and (ii) as they grade more LLM outputs, we observe a criteria drift phenomenon, in which criteria change as participants grade more LLM outputs (both definitions of existing criteria, and changes to the overall set of criteria).
• Participants’ perceptions of alignment and needs varied based on the evaluator type (i.e., code-based vs. LLM-based).
We unpack these findings below. We first describe the typical participant workflow and highlight where participants wanted to exercise control in the process. Then, we discuss the challenges participants faced in aligning assertions with their preferences.
总体上,我们发现:
- 参与者认为 EvalGen 是生成断言的绝佳起点,并且想要——也能够——对 EvalGen 的辅助行使控制。
- 参与者难以让断言与自己的偏好对齐,主要源于打分的两大挑战:(i) 有些标准对人类而言难以打分(如低于目标字数);(ii) 随着他们给更多 LLM 输出打分,我们观察到标准漂移现象——标准随参与者打分增多而变化(既有对既有标准定义的改变,也有对标准整体集合的改变)。
- 参与者对对齐的感知与需求随评估器类型(即代码型 vs. LLM 型)而不同。
下面我们展开这些发现。我们首先描述典型的参与者工作流,并突出参与者在流程中希望行使控制之处;然后讨论他们在让断言与偏好对齐时面临的挑战。
7.1 典型参与者工作流(Typical Participant Workflow)
All participants (𝑛 = 9) used the provided task (a prompt template for NER of a dataset of 100 tweets, described in 6). Three (3) participants changed the prompt: of these three, one (1) person changed the task from NER to sentiment analysis. After participants had settled on their own prompt (or decided they wanted to use our prompt), each engaged in roughly the following activities:
所有参与者(n = 9)都使用了所提供的任务(一个对 100 条推文数据集做 NER 的提示模板,见第 6 节)。三(3)位参与者修改了提示:其中一(1)人把任务从 NER 改成了情感分析。在参与者定下自己的提示(或决定用我们的提示)之后,每人大致经历了以下活动:
(1) Eyeballing LLM outputs: Participants viewed the table of 100 LLM outputs, making sure the outputs seemed reasonable at a glance.
(2) Starting EvalGen: Participants clicked the button to start EvalGen. This wizard presents three options (Fig. 2b): auto-generate criteria, write criteria, and grade outputs first (before generating criteria). 6 participants clicked the auto-generate button; 1 participant wrote a criterion themselves and then clicked the auto-generate button. The remaining 2 participants wanted to grade first (P4, P9).
(3) Grading outputs: Participants who graded LLM outputs first graded between 5 and 10 outputs, with 2 to 4 “thumbs-down” grades. After grading, both participants clicked the button to auto-generate criteria.
(4) Refining criteria: After receiving criteria suggestions from EvalGen, participants removed some suggestions and added 1-2 criteria of their own. They usually left evaluation type (code-based or LLM-based) unchanged from what EvalGen suggested, even if EvalGen suggested a type that did not make sense (e.g., checking word count with an LLM API call, rather than code), which happened rarely.
(5) Grading more outputs: Participants graded outputs while EvalGen generated and evaluated different candidate assertions. Some participants graded continuously for up to 10 minutes; others stopped after 10 grades.
(6) Understanding alignment on graded outputs: Participants viewed the “Report Card” screen, where they all spent a few minutes inspecting the resulting assertion set. Only one participant did not understand what the overall set’s coverage and false failure rate meant (P9), but did once the researcher explained it. Participants chose to view the different assertion implementations EvalGen evaluated, including the ones that did not align with their grades.
(7) Eyeballing alignment on ungraded outputs: Returning to the main screen, participants clicked the Run button of the Multi-Eval node to run all assertions on all 100 LLM outputs and inspected the table of results (Fig. 3).
(8) Iterating on criteria: 3 out of 9 participants again opened the EvalGen wizard to iterate on their criteria and assertions (back to step 2), or stopped because they were satisfied or tired. More than half of the participants expressed interest in iterating on their assertions if they had more time.
(1) 粗看 LLM 输出:参与者查看 100 条 LLM 输出的表格,确保输出乍看之下合理。
(2) 启动 EvalGen:参与者点击按钮启动 EvalGen。向导给出三个选项(图 2b):自动生成标准、手写标准、先给输出打分(再生成标准)。6 位参与者点了自动生成按钮;1 位参与者先自己写了一条标准再点自动生成。其余 2 位参与者想先打分(P4、P9)。
(3) 给输出打分:先打分的参与者给 5 到 10 条输出打了分,其中 2 到 4 条为「点踩」。打分后,两位参与者都点击了自动生成标准的按钮。
(4) 精炼标准:收到 EvalGen 的标准建议后,参与者删掉一些建议、添加 1–2 条自己的标准。他们通常保持评估类型(代码型或 LLM 型)不偏离 EvalGen 的建议,即使 EvalGen 建议的类型并不合理(例如用 LLM API 调用而非代码检查字数)——不过这很少发生。
(5) 打更多分:参与者在 EvalGen 生成并评估不同候选断言的同时继续给输出打分。有人连续打了最多 10 分钟;有人在 10 次打分后停止。
(6) 理解已打分输出上的对齐:参与者查看「成绩单」界面,都在上面花了几分钟检查所得断言集。只有一位参与者(P9)不理解整体的覆盖率与假失败率的含义,但经研究者解释后就懂了。参与者选择查看 EvalGen 评估过的不同断言实现,包括与他们的打分不对齐的那些。
(7) 粗看未打分输出上的对齐:回到主界面,参与者点击 Multi-Eval 节点的 Run 按钮,在全部 100 条 LLM 输出上运行所有断言,并检查结果表格(图 3)。
(8) 迭代标准:9 位中的 3 位再次打开 EvalGen 向导迭代其标准与断言(回到步骤 2),或因满意或疲倦而停止。超过半数参与者表示若有更多时间,愿意继续迭代其断言。
7.2 EvalGen 作为起点(EvalGen as a Starting Point)
We found that participants liked EvalGen as a starting point to generating assertions. They felt that exerting control was necessary—since EvalGen sometimes made mistakes—and they did not mind doing so. P8 said:
> This is how I would want a workflow to assist me in evals—basically I want the AI to do 80% of it, and there can be escape hatches if the AI fails.
Here, we discuss what participants liked and disliked in different components of EvalGen’s UI.
我们发现参与者喜欢把 EvalGen 作为生成断言的起点。他们感到行使控制是必要的——因为 EvalGen 有时会犯错——而且他们不介意这么做。P8 说:
这正是我希望辅助我做评估的工作流——基本上我想让 AI 做 80%,AI 失败时可以有逃生口。
这里,我们讨论参与者对 EvalGen 界面不同组件的好恶。
7.2.1 LLM-generated criteria alleviates writer’s block. Eight (out of 9) participants were pleasantly surprised that suggested criteria reflected the criteria they wanted (all except P3). P4 said, “I get writer’s block when thinking about what assertions to write, so this is great.” P6 said, “I feel that if I gave my prompt [for my own pipeline], this would create really good criteria automatically.” Some participants had negative initial reactions: for example, P7 initially expressed dissatisfaction that 7 criteria were generated, as they had expected less. But after deleting some criteria, they realized that the suggestions covered all the criteria they wanted (i.e., they did not have to add criteria themselves), and stated, “this auto-generation is sweet”. P3 found that some EvalGen-suggested criteria did not make sense given that assertions only operate on LLM pipeline input-output pairs, like response latency and response stability over time. P3 did not feel that this was a failure of EvalGen, but rather expressed that LLMs are not perfect and stressed the importance of having control over selected criteria.
7.2.1 LLM 生成的标准缓解写作瓶颈。九人中有八位(除 P3 外)惊喜地发现,建议的标准反映了他们想要的标准。P4 说:「想该写什么断言时我会有写作瓶颈,所以这很棒。」P6 说:「我感觉如果把我自己流水线的提示给它,它会自动生成非常好的标准。」一些参与者最初的反应是负面的:例如 P7 起初对生成了 7 条标准表示不满,因为他们预期更少。但在删掉一些标准后,他们意识到建议覆盖了他们想要的全部标准(即不必自己添加),并表示「这个自动生成很妙」。P3 发现 EvalGen 建议的某些标准没有道理,因为断言只作用于 LLM 流水线的输入-输出对,比如响应延迟与响应随时间的稳定性。P3 并不认为这是 EvalGen 的失败,而是表示 LLM 并不完美,并强调对所选标准拥有控制权的重要性。
7.2.2 Grading outputs first helps refine initial criteria. Participants who graded before selecting criteria (P4 and P9) expressed that they found the grading process very useful. Both participants were happy that EvalGen prompted them to give feedback on why an LLM output was bad. While P9 selected the grading option initially because it seemed like the “option that required the least thinking,” they later realized that it was the correct thing to do:
> Of the 3 options, [selecting criteria] probably wasn’t the best to start with because I couldn’t have extracted [all the] rules directly from the prompt. [While grading, I found] some rules that aren’t included in the prompt.
Other participants who did not grade before selecting criteria regretted this decision: for example, P2 said, “you should enforce that we all look at at least 20 examples first.” We discuss criteria refinement more in 7.3.1.
7.2.2 先给输出打分有助于精炼初始标准。在选择标准之前先打分的参与者(P4 与 P9)表示打分过程非常有用。两位参与者都很高兴 EvalGen 提示他们给出「LLM 输出为何坏」的反馈。P9 最初选择打分选项只是因为它看似「最不需要动脑的选项」,后来意识到这才是正确的做法:
三个选项里,(先选标准)大概不是最好的起点,因为我本不可能直接从提示里提炼出(所有)规则。(打分时我)发现了一些提示里没包含的规则。
其他没有先打分就选标准的参与者则后悔这一决定:例如 P2 说:「你们应该强制我们都先看至少 20 个例子。」我们在 7.3.1 进一步讨论标准精炼。
7.2.3 Users were happy to grade while waiting. Overall, all participants but one (P7) found grading while waiting for EvalGen to finish generating and executing candidate assertions to be a good use of their time. The participant who did not agree said that they would find it useful if, on the grading screen, EvalGen showed what it was doing with the grades. However, three participants found it difficult to enumerate all criteria in their head and grade against all criteria, wanting EvalGen to prompt them to grade per each criteria (P6, P7, P8). Several participants realized they needed to go back and change their grades in EvalGen and were happy they could do this.
7.2.3 用户乐于在等待期间打分。总体上,除一人(P7)外,所有参与者都认为在等待 EvalGen 完成候选断言生成与执行期间打分是对时间的良好利用。不同意的那位参与者说,如果打分界面上 EvalGen 能展示它正在用这些打分做什么,他们会觉得有用。不过,三位参与者觉得在脑中枚举全部标准、并对照全部标准打分很困难,希望 EvalGen 按每条标准分别提示他们打分(P6、P7、P8)。若干参与者意识到需要回头修改在 EvalGen 里的打分,并很高兴可以这么做。
7.2.4 Users need more control when reviewing alignment on graded results. In the Report Card screen, participants liked that EvalGen generated multiple assertion implementations for each criterion and picked the most aligned one (i.e., highest coverage within a false failure rate threshold of 20%). However, for some criteria, EvalGen generated no good assertions, and participants deleted the criteria without complaints. P8 said, “I like that it tries, because maybe there will be a good implementation!” This suggests that, compared to an opaque approaches we discussed in Related Work, seeing alignment scores helped participants cull assertions that were unaligned rather than blindly trusting them. For criteria where EvalGen did not find any good assertion implementations, participants also expressed interest in writing assertions. Some participants said that they had not graded enough responses to get their ideal alignment numbers, so they wanted to go back and grade more (P1, P7). Participants also liked seeing the coverage and false failure rate of the set, and how each assertion contributed to these overall statistics. However, in this screen, some participants expressed interest in seeing per-criterion alignment and asked to provide grades for each criterion (P5, P6, P7, P8). Another place where participants wanted to exercise control but were limited by EvalGen’s interface was in the selected assertions: P2 and P5 wanted to change a selected assertion to another candidate assertion. Both P2 and P5 wanted to change code-based assertions: for example, P5 said, “this is a weird implementation of entity count” and preferred another assertion with a small modification.[^fn4]
7.2.4 查看已打分结果上的对齐时,用户需要更多控制。在成绩单界面,参与者喜欢 EvalGen 为每条标准生成多个断言实现并挑选最对齐者(即假失败率阈值 20% 以内覆盖率最高者)。然而对某些标准,EvalGen 没能生成好断言,参与者毫无怨言地删除了这些标准。P8 说:「我喜欢它肯尝试,因为也许会有一个好实现!」这表明,与我们在相关工作中讨论的不透明方法相比,看到对齐分数帮助参与者剔除不对齐的断言,而非盲目信任。对 EvalGen 没找到好断言实现的标准,参与者也表达了亲自书写断言的兴趣。一些参与者说自己打分不够多、得不到理想的对齐数值,所以想回去再打一些分(P1、P7)。参与者也喜欢查看断言集的覆盖率与假失败率、以及每个断言对这些整体统计的贡献。不过在该界面,一些参与者希望看到每标准的对齐度,并要求为每条标准分别打分(P5、P6、P7、P8)。参与者想行使控制却受限于 EvalGen 界面的另一处是选定的断言:P2 与 P5 想把选定断言换成另一个候选断言。两人想换的都是代码型断言:例如 P5 说「这个实体计数的实现很怪」,偏好另一个稍作修改的断言。[^fn4]
[^fn4]: 脚注 4 原文:Note that assertions can be edited in the MultiEval node after returning to the main window, but not in EvalGen’s report card screen.(注意,返回主窗口后,断言可以在 MultiEval 节点中编辑,但在 EvalGen 的成绩单界面中不行。)
[图 3: The Table View, showing inputs, LLM outputs, and evaluation results per criteria for the NER task (Sec. 6).]
图 3:表格视图(Table View),展示 NER 任务(第 6 节)的输入、LLM 输出与每标准的评估结果。
表 2(Table 2):参与者对「断言与我的打分对齐」的评分
| 参与者 | P1 | P2 | P3 | P4 | P5 | P6 | P7 | P8 | P9 |
|---|---|---|---|---|---|---|---|---|---|
| 评分 | 6 | 5 | 3 | 4 | 5 | 3 | 1 | 2 | 5 |
表 2:对陈述「我感觉这些断言与我的打分对齐(I felt like the assertions aligned with my grades)」的评分(1–7,7 为最佳)。回答两极分化。
7.2.5 Users differ in how they assess alignment for ungraded results. Participants really liked viewing the table of assertion results on all LLM outputs, with all participants expressing interest upon first viewing it (Figure 3)—but they verified the results differently. In this table, rows represent LLM outputs, and columns represent assertions. P2 said that this table “earns trust”; P5 said that it was “cool to see all the results on examples [they] didn’t grade.” P5 initially did not like using a GUI to come up with assertions, and then said that they “prefer looking at this table to [their] current workflow in Jupyter notebooks.” Interestingly, only one participant assessed alignment in this table via coverage—by inspecting each row in the table, one at a time—and everyone else assessed alignment via false failure rate: for each column (assertion), they typically looked for rows (LLM outputs) that the assertion returned False. Some participants wanted this table to automatically recompute, with an accompanying visualization, whenever there were changes to the LLM pipeline; for example, P6 said:
> One thing I would find cool is if there is a way to easily see how changes to my prompt impact the overall [coverage and false failure rate] scores. Just very quickly being able to visualize how [my prompt edit] changes the classifications on a bunch of [LLM outputs.]
This suggests users want data visualization plots between the original prompt and its revision, akin to LLM Comparator and EvalLM [24, 27]. Note that ChainForge [1] also has visualization plots, but we did not introduce participants to this feature.
7.2.5 用户核查未打分结果之对齐的方式因人而异。参与者非常喜欢查看所有 LLM 输出上的断言结果表格,所有人在初见时都表现出兴趣(图 3)——但他们核查结果的方式各不相同。在该表格中,行代表 LLM 输出,列代表断言。P2 说这张表「赢得信任」;P5 说「看到所有没打过分的例子上的结果很酷」。P5 起初不喜欢用 GUI 构思断言,后来说「比起我在 Jupyter notebook 里的现有工作流,更喜欢看这张表」。有趣的是,只有一位参与者通过覆盖率评估该表中的对齐——逐行检查表中每一行——其他所有人都通过假失败率评估对齐:对每一列(断言),他们典型地查找该断言返回 False 的行(LLM 输出)。一些参与者希望只要 LLM 流水线有变化,这张表就自动重算并附带可视化;例如 P6 说:
我觉得很酷的一件事是,如果有办法轻松看到我对提示的改动如何影响整体(覆盖率与假失败率)分数就好了。能非常快速地可视化(我的提示修改)如何改变一批(LLM 输出)上的分类。
这表明用户想要原始提示与其修订版之间的数据可视化图,类似 LLM Comparator 与 EvalLM [24, 27]。注意 ChainForge [1] 也有可视化图,但我们没有向参与者介绍该功能。
7.3 对齐是迭代的、依标准与实现而异(Alignment is Iterative, Criteria- and Implementation-specific)
Perceptions of the tools’ support of alignment were polarized across participants, as shown in Table 2. The main reason why people had low confidence in the assertion report card’s alignment with their grades was because they were still uncertain of their own criteria while grading. Participants would say utterances with a questioning tone, like “I guess” and “sure,” when grading, indicating their uncertainty (P2, P5, P7, P8, P9). Looking more closely at their interactions, we observed a catch-22 situation: participants needed to externalize criteria in order to grade outputs, but they also needed to grade outputs—providing feedback on why bad outputs were bad—in order to externalize criteria. Here, we explore their challenges.
如表 2 所示,参与者对该工具对齐支持的感知两极分化。人们对断言成绩单与自身打分对齐度信心不足的主因,是打分时对自己的标准仍不确定。参与者打分时常带疑问语气说「我猜(I guess)」「行吧(sure)」,显示出其不确定(P2、P5、P7、P8、P9)。更细致地观察他们的交互,我们发现一个 catch-22 局面:参与者需要先外化标准才能给输出打分,但他们也需要给输出打分——就坏输出为何坏提供反馈——才能外化标准。这里我们探究他们的挑战。
7.3.1 Criteria drift. Grading outputs spurred changes or refinements in participants’ criteria, which we refer to as criteria drift. We observed two types of drift.
7.3.1 标准漂移(criteria drift)。给输出打分促成了参与者标准的变化或精炼,我们称之为标准漂移。我们观察到两类漂移。
First, participants wanted to add new criteria when they observed new “types” of bad LLM outputs (P2, P5, P6, P8, P9). In the EvalGen interface, they could not go back and add new criteria; they had to wait for all candidate assertions to finish executing, move past the report card screen, and start a new EvalGen process.
第一,当参与者观察到新的坏 LLM 输出「类型」时,他们想添加新标准(P2、P5、P6、P8、P9)。在 EvalGen 界面中,他们无法回去添加新标准;只能等全部候选断言执行完毕、走过成绩单界面、再启动一轮新的 EvalGen 流程。
Second, as participants graded more outputs, we found that they reinterpret existing criteria to better fit the LLM’s behavior (P2, P5, P6, P8, P9). For example, P2 and P8 had a “proper noun” criterion, which was supposed to assess that “the entities extracted were proper nouns.” At first, they rated as bad any LLM outputs that contained any entity that was not a proper noun. But, after observing that responses had varying numbers of proper nouns, both wanted to change their criteria such that most of the entities were proper nouns, rather than all. P9 appreciated the ability to provide feedback on unsatisfactory outputs before finalizing the initial criteria set, as this process helped them articulate their criteria. Twice, P8 mentioned that they gave a bad grade not because they believed the output was bad, but because they wanted to be consistent with previous grades—good labeling practice, perhaps, but not good for alignment. P7 noticed that in some outputs included hashtags from the original tweet inputs (e.g., #justdoit), while in other cases, the outputs did not use the hashtag symbol but still mentioned the entities referred to by the hashtags (e.g., “Nike” instead of #Nike). This inconsistency led P7 to rethink the criteria definition for including hashtags; specifically, whether it was acceptable for the LLM to replicate entities from hashtags, provided the hashtag was removed. P5 expressed the same uncertainty. “I think it’s hard to know until you see it,” P7 said.
第二,随着参与者打分增多,我们发现他们会重新解释既有标准以更贴合 LLM 的行为(P2、P5、P6、P8、P9)。例如,P2 与 P8 有一条「专有名词」标准,本应评判「抽取的实体都是专有名词」。起初,他们把任何含有非专有名词实体的 LLM 输出都判为坏。但在观察到回复中专有名词数量参差不齐后,两人都想把标准改成「大多数实体是专有名词」,而非全部。P9 很欣赏能在敲定初始标准集之前对不满意输出提供反馈,因为这一过程帮助他们清晰表达标准。P8 两次提到,自己给某个坏评分并不是因为认为输出坏,而是想与先前的打分保持一致——这或许是好的标注习惯,却不利于对齐。P7 注意到,一些输出包含了原推文输入中的话题标签(如 #justdoit),而另一些输出没用标签符号、却仍提到了标签所指的实体(如用「Nike」而非 #Nike)。这种不一致使 P7 重新思考纳入话题标签的标准定义:具体而言,只要去掉标签符号,LLM 复现标签中的实体是否可以接受。P5 表达了同样的不确定。「我觉得不亲眼看到就很难知道,」P7 说。
7.3.2 Users prefer to adjust their grading approach based on the difficulty of evaluating a criterion. Overall, participants generally liked the process of grading LLM responses and feeling like the grades were useful, but they wanted to prioritize grading criteria they felt needed their alignment—especially for LLM-based assertions (P3, P5, P7, P8). For example, P3 expressed that they would trust the assertions more if the EvalGen process allowed them to set different false failure rates per criteria (since LLMs might be bad at evaluating some criteria), instead of one global false failure rate constraint for the entire assertion set:
> There are criteria where you can be okay with failing, and then there are other criteria where you are like, ‘this must absolutely pass’... [T]here’s a [spectrum] of failure as opposed to: it just passes or fails.
Relatedly, some participants expressed that they didn’t trust their grades because they themselves couldn’t evaluate some criteria as well as an automated solution (P2, P5, P6, P7, P8; we discuss further in Section 7.4). A criterion like word count is hard for humans to assess but easy for a good Python function to evaluate. P8 desired to grade for only one criteria, reasoning that it might improve efficiency (“I generally want to be in the loop for these tests...but I want to put myself in the loop in a way that is efficient.”).
7.3.2 用户偏好依「标准评估难度」调整打分方式。总体上,参与者普遍喜欢给 LLM 回复打分的过程、并感觉打分有用,但他们想优先给自己认为最需对齐的标准打分——尤其是 LLM 型断言(P3、P5、P7、P8)。例如 P3 表示,如果 EvalGen 流程允许他们为每条标准设不同的假失败率(因为 LLM 评估某些标准可能很差),而不是对整个断言集设一个全局假失败率约束,他们会更信任这些断言:
有些标准你可以接受它失败,而另一些标准你会说「这绝对必须通过」……失败是一个(谱系),而不是非过即败。
相关地,一些参与者表示不信任自己的打分,因为他们自己无法像自动化方案那样好评某些标准(P2、P5、P6、P7、P8;进一步讨论见第 7.4 节)。像字数这样的标准人类很难评估,而一个好的 Python 函数极易评估。P8 希望每次只为一条标准打分,理由是这可能提高效率(「这些测试我一般都想参与……但我想以高效的方式参与」)。
7.3.3 What constitutes “alignment” is subjective, especially when converting natural language criteria to code-based assertions. For code-based assertions, EvalGen’s interpretation of the criterion (i.e., GPT-4’s interpretation) did not match what the participants expected. As such, no matter how many grades the participant gave, all candidate assertions were similarly misaligned. Participants who observed this were confused why their grades seemingly had little impact on some of the chosen assertions (P3, P5, P7, P9). For example, while all participants had a criterion to enforce that there were no entities with hashtags in the output, some participants interpreted this as any hashtags representing entities should not be extracted as entities: e.g., if the output included the hashtag #Nike, P5 did not want Nike to be extracted at all. On the other hand, P9 wanted Nike to be extracted as an entity, but they did not want the LLM output to include the hashtag. Both P5 and P9 got the same code-based assertion for the criterion, which simply checked for the presence of the hashtag character in the output—this assertion did not align with P5’s grades, but did with P9’s. This particular misalignment can also be viewed as an instance of criteria drift, as described in Section 7.3.1, since P5 was only able to refine the criterion after grading several LLM outputs. For another criterion, P5 felt that EvalGen could not find a good assertion that aligned with their grades, but also said that they were “lost at what would be a good implementation.”
7.3.3 何谓「对齐」是主观的,尤其在把自然语言标准转换为代码型断言时。对代码型断言,EvalGen(即 GPT-4)对标准的解释与参与者预期不符。因此,无论参与者打多少分,所有候选断言都同样错位。观察到这一点的参与者感到困惑:他们的打分为何看起来对某些选定断言几乎没有影响(P3、P5、P7、P9)。例如,虽然所有参与者都有一条「输出中不得含带话题标签的实体」的标准,一些参与者把它理解为「任何代表实体的话题标签都不应被抽为实体」:如输出包含话题标签 #Nike,P5 希望 Nike 根本不被抽取。另一方面,P9 希望 Nike 被抽为实体,但不希望 LLM 输出包含话题标签。P5 与 P9 拿到的是同一条代码型断言,它只检查输出中话题标签字符的存在——该断言与 P9 的打分对齐,却与 P5 的不对齐。这一特定错位也可视为 7.3.1 节所述标准漂移的一个实例,因为 P5 是在给若干 LLM 输出打分之后才得以精炼该标准。对另一条标准,P5 感到 EvalGen 找不到与其打分对齐的好断言,但也说自己「不知道什么才算好实现」。
Overall, alignment is not merely a matter of performance (i.e., the idea that “a better LLM would do better”): we observed that misalignment sometimes occurred due to tacit criteria that participants held which was not explicitly explained in natural language. Like prior work has found for cross-LLM comparison [1], this tacit understanding of criteria could be highly subjective and contradictory across participants.
总体而言,对齐并非性能问题(即「更好的 LLM 就能做得更好」的想法):我们观察到,错位有时源于参与者持有、却未用自然语言明确解释的隐性(tacit)标准。如同先行工作在跨 LLM 比较上的发现 [1],这种对标准的心照不宣的理解在参与者之间可能高度主观且相互矛盾。
7.4 代码型与 LLM 型评估器的对齐需求与偏好不同(Alignment Needs and Preferences Differ for Code vs. LLM Evaluators)
All participants liked EvalGen’s ability to assist in generating both code-based and LLM-based assertions, expressing the need for both types of assertions. Both P4 and P6 mentioned that they liked the ability to correct EvalGen’s suggested type. Yet participants reasoned about code-based assertions and LLM-based assertions differently: while they liked having assertions of both types, they wanted to construct and iterate on them differently in order to feel like the resulting set of assertions aligned with their preferences.
所有参与者都喜欢 EvalGen 辅助生成代码型与 LLM 型两类断言的能力,并表达了对两类断言的需求。P4 与 P6 都提到喜欢能纠正 EvalGen 建议的类型。但参与者对代码型断言与 LLM 型断言的推理方式不同:虽然他们喜欢两类断言兼有,却希望以不同方式构建和迭代它们,才会觉得最终的断言集与自己的偏好对齐。
7.4.1 Users like having control over the evaluation type and thought each evaluation type had a different affordance. Participants seemed to have a clear idea of when a criterion should be evaluated by code, and when it should be evaluated by an LLM. Generally, for formatting checks (e.g., asserting that an output is in Markdown), count-based checks, and checks to include or exclude specific phrases, participants preferred code-based assertions. P6 and P8 expressed that they wanted to use LLM-based assertions for their own LLM pipelines because “most of their criteria was fuzzy.” P2 selected criteria to be evaluated via LLM whenever they could not immediately think of a Python function that could implement that criteria. We also observed a participant (P8) preferring LLM-based to code-based assertions: LLMs are relatively forgiving when encountering outputs with an unexpected quality or format, whereas code assertions check exact constraints.
7.4.1 用户喜欢掌控评估器类型,并认为各类型有不同的可供性(affordance)。参与者似乎清楚何时该用代码评估一条标准、何时该用 LLM。总体上,对格式检查(如断言输出为 Markdown)、计数型检查、以及包含/排除特定短语的检查,参与者偏好代码型断言。P6 与 P8 表示想在自己的 LLM 流水线中用 LLM 型断言,因为「他们的大多数标准是模糊的」。P2 的做法是:凡不能立刻想出可实现该标准的 Python 函数,就选 LLM 评估。我们还观察到一位参与者(P8)偏好 LLM 型甚于代码型断言:遇到质量或格式出乎意料的输出时,LLM 相对宽容,而代码断言检查的是精确约束。
7.4.2 Users can—and want to—directly verify code-based implementations. Regardless of the type of assertion (code or LLM), EvalGen follows the same process of exploring multiple candidate assertions per criterion and, for each criterion, selecting the assertion that aligns most with the user’s grades. While participants generally liked this approach for LLM-based assertions, they did not like this for code-based assertions. This observation may be specific to users who are experts in coding (as all of our participants are). Some participants wanted to see the code for each candidate assertion and select the best Python function for each criterion themselves (P2, P5, P7). P5 said:
> When something can be solved using Python code, I do have an envisioned [implementation] in mind that I can easily verify. Just showing [me] the [code] will be quicker.
P7 wanted to iterate on code-based assertions simply by providing feedback to make the assertion more or less “fancy”—for example, in checking for a pattern in the LLM output, they might want the regex to be more or less complicated depending on the sample of LLM outputs they’ve seen. However, P7 then acknowledged that their ability to edit depends on code length and complexity:
> For something that’s like 10 lines of code or less, I’ll look at [the code]. But for [longer functions,] I probably want to eyeball [the function results]. This is where it gets a bit unclear.
While participants had lots of ideas for improvement, participants overall felt that EvalGen provided a good start to code-based assertions, and it was easy for them to edit the code if they wanted: two participants even asked for the ability to export code-based assertions as unit tests or in a Python file (P4, P6).
7.4.2 用户能够——也希望——直接核验代码型实现。无论断言类型(代码或 LLM),EvalGen 都遵循同一流程:为每条标准探索多个候选断言,并为每条标准选出与用户打分最对齐者。参与者对 LLM 型断言普遍喜欢这一做法,对代码型却不喜欢。这一观察可能特定于精通编程的用户(我们所有参与者都是)。一些参与者想看到每个候选断言的代码、并自己为每条标准挑最好的 Python 函数(P2、P5、P7)。P5 说:
当某件事能用 Python 代码解决时,我脑中已有一个可轻松验证的(预想实现)。直接给我看(代码)会更快。
P7 想仅靠提供反馈来迭代代码型断言,让断言更「花哨」或更朴素——例如在检查 LLM 输出中的模式时,根据所见 LLM 输出样本,他们可能想要正则表达式更复杂或更简单。不过 P7 随即承认,其编辑能力取决于代码长度与复杂度:
大概 10 行以内的代码,我会看(代码)。但(更长的函数)我可能就想粗看(函数结果)了。这里就有点说不清了。
尽管参与者有大量改进想法,他们总体上认为 EvalGen 为代码型断言提供了良好起点,而且想改代码很容易:两位参与者甚至要求能把代码型断言导出为单元测试(unit tests)或 Python 文件(P4、P6)。
7.4.3 Users want their grades to aid automated criteria generation. While participants liked that EvalGen tried multiple candidate assertions for each criterion, they wanted EvalGen to use their grades, as well as natural language feedback on thumbs-down grades, in prompting LLMs for candidate assertions (P3, P4, P5, P7). For example, any time a user gives a grade, EvalGen could query an LLM for a new candidate assertion, including the new grade in the prompt so that the candidate assertion may be more aligned with the user’s grades. This strategy can, of course, be used to generate new code-based assertions too. Not only did participants want grades to be included in the process of generating new candidate assertions, but some wanted their labeled LLM outputs to be included in the LLM-based assertions’ prompts themselves (P3, P5). P3 excitedly realized this when looking at the table of results (“I can take these good and bad examples and use them as few-shot exemplars in the prompt for the LLM evaluators...”). This suggests an optimization loop akin to ConstitutionMaker and DSPy [26, 39], but for assertion generation and validation, rather than prompt optimization.
7.4.3 用户希望自己的打分反哺自动标准生成。参与者喜欢 EvalGen 为每条标准尝试多个候选断言,但他们希望 EvalGen 把自己的打分、以及对点踩打分的自然语言反馈,用于向 LLM 请求候选断言的提示中(P3、P4、P5、P7)。例如,用户每给一次分,EvalGen 就可以向 LLM 请求一个新的候选断言,把新打分纳入提示,使候选断言更可能与用户打分对齐。这一策略当然也可用于生成新的代码型断言。参与者不仅希望打分被纳入新候选断言的生成过程,一些人还希望他们标注过的 LLM 输出被纳入 LLM 型断言的提示本身(P3、P5)。P3 在看结果表时兴奋地意识到了这一点(「我可以拿这些好坏例子,用作 LLM 评估器提示里的少样本示例……」)。这提示了一个类似 ConstitutionMaker 与 DSPy [26, 39] 的优化回路,但面向断言的生成与验证,而非提示优化。
7.4.4 LLM-based assertions are harder to trust. While participants found EvalGen’s suggested code-based assertions to be more obviously misaligned than the LLM-based assertions (Section 7.3.3), they also acknowledged that LLM-based assertions are harder for them to trust—since they can edit the code-based assertions more easily than the LLM-based assertions (P2, P5, P6, P8, P9). Some participants were skeptical of how LLM-based assertions might transfer to monitoring LLM outputs in a production pipeline (P3, P6, P8). P8 said, “I cannot begin to think about how LLMs as validators [in production] can work, I’m very skeptical.” P9 asked, “How do I maintain my evals over time; do I have to rerun this entire process?” One suggestion is an interface that solicits grades from other people—possibly the end-users of the user’s LLM pipeline—to continually realign LLM-based assertions.
7.4.4 LLM 型断言更难信任。虽然参与者发现 EvalGen 建议的代码型断言比 LLM 型断言「明显错位」得更直观(第 7.3.3 节),他们也承认 LLM 型断言更难信任——因为代码型断言比 LLM 型更容易编辑(P2、P5、P6、P8、P9)。一些参与者对 LLM 型断言能否迁移到生产流水线的 LLM 输出监控持怀疑态度(P3、P6、P8)。P8 说:「我完全无法想象 LLM 当(生产中的)校验器怎么能行,我非常怀疑。」P9 问:「我怎么随时间维护我的评估?难道每次都要重跑整个过程?」一个建议是设计一个向其他人(可能是用户 LLM 流水线的终端用户)征求打分的界面,以持续重对齐 LLM 型断言。
8 讨论(Discussion)
Here, we unpack criteria drift and how it affects alignment, discuss the implications of our findings for future LLMOps evaluation assistants, and outline some open questions.
这里,我们剖析标准漂移及其对对齐的影响,讨论我们的发现对未来 LLMOps 评估助手的启示,并列出一些开放问题。
8.1 标准漂移对 LLM 评估助手的启示(Implications of Criteria Drift for LLM Evaluation Assistants)
The practice of benchmarks in ML and NLP presume a world of well-defined criteria (and well-labeled data) on which to judge LLM outputs. For instance, AutoCalibrate is a method to calibrate LLM evaluators with human preferences that requires large expert-labelled datasets with settled (i.e., established upfront) criteria [34]. However, in practice, we found that developers rapidly iterate over criteria, and furthermore that cognitively engaging with LLM outputs helps them to refine their criteria. This suggests criteria refinement and grading should happen in tandem in interactive settings, and poses challenges to alignment methods that presume settled, expert labels. In particular, the dependence of criteria on a specific prompt and LLM’s outputs means that any change to the LLM pipeline can cause criteria drift: for example, LLM APIs can be replaced with new models, resulting in prompt drift [4].
ML 与 NLP 的基准实践预设了一个存在明确定义标准(与标注良好数据)的世界,并据此评判 LLM 输出。例如 AutoCalibrate 是一种用人类偏好校准 LLM 评估器的方法,需要大规模的专家标注数据集与已敲定(即预先确立)的标准 [34]。然而在实践中,我们发现开发者会快速迭代标准,而且对 LLM 输出的认知参与恰恰帮助他们精炼标准。这表明标准精炼与打分应在交互环境中同步进行,并对那些预设「标准已敲定、标签出自专家」的对齐方法提出挑战。特别地,标准对具体提示与 LLM 输出的依赖意味着 LLM 流水线的任何变化都可能引发标准漂移:例如 LLM API 可能被替换为新模型,导致提示漂移 [4]。
Future system designs should support these requirements. For instance, an evaluator assistant might adjust criteria dynamically as the user grades and gives feedback. We might also consider per-criteria grading (while keeping the simplicity of thumbs up/down voting). Eliciting grades at the criteria level might allow evaluation assistants to more precisely adjust both the criteria for evaluation, as well as the way these criteria are implemented. Finally, including examples of both good and bad LLM outputs within the LLM-based evaluator prompts could also be beneficial (a method akin to recent work [26, 35]). However, whenever outputs change, we may need to ask users to re-grade or re-think their criteria.
未来的系统设计应支持这些需求。例如,评估助手可以在用户打分并给出反馈时动态调整标准。我们也可以考虑按标准打分(同时保持点赞/点踩的简单性)。在标准粒度上征求打分,可以让评估助手更精确地同时调整评估标准及这些标准的实现方式。最后,把好与坏 LLM 输出的示例都纳入 LLM 型评估器的提示也可能有益(类似近期工作的方法 [26, 35])。不过,每当输出变化,我们可能都需要请用户重新打分或重新思考其标准。
Our criteria drift finding echoes prior work in educational settings, where instructors often update their grading rubrics to reflect common errors as they grade more assignments [47]. Similar to how teachers adjust rubrics and regrade as they see more student submissions, making users reassess LLM outputs after updating criteria could enhance the accuracy and consistency of grades. Given the observed inconsistency in participants’ grading, evaluation assistants might also pursue crowdsourcing methods like majority voting and self or external assessments to determine accurate grades for LLM outputs [7, 9]. Another challenge for evaluation assistants is extending adapted criteria to grade both ungraded and future unseen LLM outputs. How do evaluation assistants consistently sample grades for outputs that reflect the overall distribution of LLM pipeline successes and failures?
我们的标准漂移发现呼应了教育场景中的先行工作:教师批改作业越多,越常更新评分细则(rubric)以反映常见错误 [47]。与教师随所见学生作业增多而调整细则并重新评分类似,让用户在更新标准后重新评估 LLM 输出,可以提高打分的准确性与一致性。鉴于参与者打分中观察到的不一致,评估助手还可以采用多数投票等众包方法及自评或外部评估,以确定 LLM 输出的准确打分 [7, 9]。评估助手的另一个挑战是把调整后的标准扩展到未打分与未来未见过的 LLM 输出上。评估助手该如何一致地采样打分,使其反映 LLM 流水线成功与失败的总体分布?
The reader might wonder when criteria “settle.” Perhaps there was simply not enough time in our study, and had participants graded for an hour or two, they might have solidified their criteria, and criteria drift goes away. There is reason to believe that this situation does not change with more time—as we saw, the criteria participants refined changed to adapt to the behavior of the LLM outputs being evaluated—a dependent, rather than independent assessment of quality. In the real world, similar situations exist where criteria are never “fully settled” as more inputs come in—consider the court of law. One of our participants remarked that people “know a bad output when they see it.” Their adage reflects a U.S. Supreme Court Justice’s famous opinion in a 1964 court case about obscene content [17]. As in that remark, the decisions of human validators seem at first glance “to be based on a non-rational, intuitive gut reaction, instead of reasoned analysis; it seems to be utterly subjective and personal” [17, p.1025]. However, perhaps, as the law scholar Gewirtz argues, subjectivity is not necessarily a sign of irrationality (contrasting with some imagined future AI that is entirely objective and rational, entirely “aligned” or “better” than humans). On the contrary: “There are good reasons to accept the imperfect in a judge. We should encourage judges to believe and say: This is the best I can do now; it doesn’t solve all the problems, but it’s a start, and I’ll keep thinking” [17, p.1027]. This raises a deeper epistemic question for evaluation assistants—is “alignment” an actualizable goal? To what extent does our common terminology and assumptions—e.g., that there is a “ground truth” set of labels we merely need to elicit—fail us? Is validating the validators only ever a work-in-progress?
读者也许会问:标准何时「尘埃落定」?也许只是我们研究中的时间不够——假如参与者打上一两个小时的分,他们或许就能固化标准,标准漂移随之消失。但有理由相信这种情况不会随时间改变——如我们所见,参与者所精炼的标准,改变的方向正是去适应被评估 LLM 输出的行为——这是一种依赖性、而非独立性的质量评判。真实世界中存在类似情境:随着输入增多,标准永远「不完全敲定」——想想法庭。我们的一位参与者评论说,人们「一看到坏输出就认得」。这句格言呼应了美国最高法院大法官在 1964 年一桩关于淫秽内容案件中的著名意见 [17]。正如那份意见所述,人类验证者的决定乍看「基于非理性的、直觉式的本能反应,而非推理分析;它看起来全然主观、个人化」[17, p.1025]。然而,也许正如法学者 Gewirtz 所论,主观性未必是非理性的标志(对照某种想象中完全客观理性、完全「对齐」或「优于」人类的未来 AI)。恰恰相反:「接受法官的不完美是有充分理由的。我们应鼓励法官相信并说出:这是我现在能做到的最好;它没有解决所有问题,但这是一个开始,我会继续思考」[17, p.1027]。这为评估助手提出了一个更深层的认识论问题——「对齐」是一个可实现(actualizable)的目标吗?我们的通用术语与假设——例如「存在一组我们只需去获取的『地面真值(ground truth)』标签」——在多大程度上辜负了我们?验证验证者是否永远只是进行中的工作?
8.2 断言的运营化(Operationalizing Assertions)
Participants expressed the desire to deploy their assertions in production. Some wanted to deploy assertions in the critical path of the LLM pipeline, to catch bad outputs before they are shown to end-users. Others wanted a more passive deployment—one participant requested for some form of report sent to their email inbox daily, containing results of the assertion set executed on a sample of LLM outputs from that day. In practice, assertions have different operational requirements. For example, if an assertion runs in the critical path, it should not result in many false failures, otherwise the LLM may get unnecessarily reprompted (i.e., rerun) and the overall latency might increase for our LLM pipeline. Moreover, as criteria changes over time, evaluation assistants should ask users to grade new LLM outputs observed in production and automatically adapt assertion sets.
参与者表达了把断言部署到生产的愿望。一些人想把断言部署在 LLM 流水线的关键路径(critical path)上,在坏输出到达终端用户之前拦截。另一些人想要更被动的部署——一位参与者要求某种每日发送到其邮箱的报告,内容是断言集在当日 LLM 输出样本上的执行结果。实践中,断言有不同的运营要求。例如,若断言运行在关键路径上,它不应产生太多假失败,否则 LLM 可能被不必要地重新提示(即重跑),我们 LLM 流水线的整体延迟可能上升。此外,随着标准随时间变化,评估助手应请用户为生产中观察到的新 LLM 输出打分,并自动调整断言集。
Our study confirmed the dual necessity for both code-based and LLM-based assertions among participants, but further showed that participants felt each type required distinct treatment, both in implementation selection and in how they perceive alignment. Often useful for sanity checks like output structure, code-based assertions can be used in the critical path of the LLM pipeline. In fact, a number of LLMOps tools exist for users to implement such code-based guardrails [19, 22, 42]. However, our participants highlighted the challenge of finding the right implementation for a given assertion—a decision that can depend intricately on the characteristics of LLM outputs. For example, determining the acceptable length of a response might vary significantly based on the observed output distribution. In fact, specifically for an output length criterion, P3 mentioned that they would look at a histogram of word lengths for a batch of outputs and see if there were some buckets of anomalous word lengths that exhibited other failure modes (e.g., a “rambling” response from the LLM).
我们的研究证实了参与者对代码型与 LLM 型断言的双重必要性,但进一步表明参与者感到两类断言需要区别对待——无论在实现选择还是对齐感知上。代码型断言常用于输出结构之类的健全性检查(sanity check),可用在 LLM 流水线的关键路径上。事实上,已有多款 LLMOps 工具供用户实现此类代码型护栏(guardrails)[19, 22, 42]。然而,我们的参与者强调了为给定断言找到正确实现的挑战——这一决策可能微妙地依赖 LLM 输出的特征。例如,可接受的回复长度可能随观察到的输出分布而显著变化。具体到输出长度标准,P3 提到他们会看一批输出的词长直方图,看是否存在一些异常词长的桶同时表现出其他失败模式(如 LLM 的「啰嗦」回复)。
Some participants mentioned that they wanted their collaborators, such as product managers, to grade outputs in EvalGen. Importantly, when allowing multiple users to collaborate on grading, evaluation assistants have to consider inter-rater reliability and handle disagreements, if any. Again, we can draw inspiration from techniques from the crowdsourcing literature: for example, evaluation assistants could model each individual grader’s accuracy and apply corrections if necessary [59]. Moreover, collaborators may have different experience levels with coding: those with less experience may choose LLM-based assertions for criteria that might be better evaluated with code. While EvalGen provides initial suggestions for whether criteria should be evaluated with code or another LLM, evaluation assistants may want to make sure that all users from the same team “agree” on the right implementations. One could imagine interfaces similar to creating a “pull request” for a new assertion and soliciting review from a team member, and a workflow similar to continuous integration/continuous deployment (CI/CD) that seamlessly pushes new assertions to production.
一些参与者提到,希望让协作者(如产品经理)也在 EvalGen 中打分。重要的是,允许多用户协作打分时,评估助手必须考虑评分者间信度(inter-rater reliability)并处理可能出现的分歧。同样,我们可以从众包文献的技术中汲取灵感:例如评估助手可以建模每个评分者的个体准确度,并在必要时施加校正 [59]。此外,协作者的编码经验可能不同:经验较少者可能为更适合代码评估的标准选择 LLM 型断言。虽然 EvalGen 会就「标准该用代码还是另一个 LLM 评估」给出初始建议,评估助手可能仍要确保同团队的所有用户在正确实现上「达成一致」。可以设想这样的界面:为新断言创建类似「拉取请求(pull request)」的流程并请团队成员评审,以及一条类似持续集成/持续部署(continuous integration/continuous deployment,CI/CD)的工作流,把新断言无缝推送到生产。
8.3 未来工作与局限(Future Work and Limitations)
Assertions serve as a straightforward mechanism for evaluating LLM outputs, yet the potential of evaluation assistants extends beyond EvalGen’s supported binary judgments. Consider the example of word count: rather than setting rigid thresholds, one might find it more useful to monitor variations in word count across different LLM outputs. Like in traditional ML monitoring [44], there is, of course, still imprecision in what word count counts as “bad,” given its distribution. Does the AI assistant’s definition of bad align with the user’s? Tracking finer-grained information in evaluations, beyond simple true/false conditions, can aid in debugging issues within LLM pipelines, once a user knows the output is bad. Second, many users write their LLM pipelines as chains of multiple LLM calls [1, 11, 14, 53]. Evaluation assistants can facilitate end-to-end alignment, which can be complicated not only with chains of LLM calls, but also in compound AI systems that leverage other components like retrieval-augmented generation [55].
断言是评估 LLM 输出的一种直接机制,但评估助手的潜力超越 EvalGen 所支持的二元判断。以字数为例:与其设置僵硬的阈值,追踪不同 LLM 输出间字数的变化可能更有用。如同传统 ML 监控 [44],当然,给定分布后「多少字算坏」仍不精确。AI 助手对「坏」的定义与用户的一致吗?在评估中追踪比简单真假条件更细粒度的信息,可以在用户知道输出坏了之后,帮助调试 LLM 流水线内的问题。其次,许多用户把 LLM 流水线写成多次 LLM 调用的链 [1, 11, 14, 53]。评估助手可以促成端到端对齐,这不仅对 LLM 调用链而言复杂,在使用检索增强生成(retrieval-augmented generation)等其他组件的复合 AI 系统(compound AI systems)中同样复杂 [55]。
Increasingly, user want their prompts to automatically improve based on assertion results. For instance, if an assertion fails for a group of LLM outputs, one might want to add an instruction to the prompt to solve this failure mode. Some frameworks already experiment with using feedback from assertions or user grades to refine prompts [35, 39, 48]. However, incorporating this into an evaluation assistant is not as straightforward: there may be a cyclical process where evaluation assistants not only help in adjusting assertions based on prompt performance but also use these insights to suggest further prompt modifications. This iterative cycle of evaluation and adjustment could foster a co-evolutionary environment where prompts, assertions, and even evaluative mechanisms themselves are continuously refined in a unified interface.
用户越来越希望提示能基于断言结果自动改进。例如,若一条断言在一组 LLM 输出上失败,人们可能想给提示加一条指令来解决该失败模式。一些框架已在试验用断言或用户打分的反馈精炼提示 [35, 39, 48]。但把这一点纳入评估助手并不那么直接:可能存在一个循环过程——评估助手不仅根据提示表现帮助调整断言,还用这些洞见建议进一步的提示修改。这一评估与调整的迭代循环可以催生一个共同演化的环境:提示、断言、甚至评估机制本身都在统一界面中被持续精炼。
Finally, there are two limitations of our study that are worth pointing out. First, our off-line evaluation only focused on two pipelines. Different applications may warrant different methods of sampling grades and aligning assertions. Second, our qualitative study focused on a small sample of developers, all of whom have significant experience deploying LLMs. Moreover, participants did not have enough time to iterate many times in EvalGen, and our setup did not cover the deployment phase of LLM workflows. Future work might explore best practices and pitfalls of evaluations in the broader LLMOps lifecycle.
最后,值得指出我们研究的两点局限。第一,我们的离线评估只聚焦两条流水线;不同应用可能需要不同的打分采样与断言对齐方法。第二,我们的定性研究样本是一小群开发者,且全都有部署 LLM 的丰富经验。此外,参与者没有足够时间在 EvalGen 中多次迭代,我们的设置也未覆盖 LLM 工作流的部署阶段。未来工作可以探索更广泛 LLMOps 生命周期中评估的最佳实践与陷阱。
9 结论(Conclusion)
This work presented EvalGen, a mixed-initative approach to aligning LLM-generated evaluation functions with human preferences. EvalGen assists users in developing both criteria acceptable LLM outputs and developing functions to check these standards, ensuring evaluations reflect the users’ own grading standards. In a qualitative study with 9 expert users, we observed a pattern we call criteria drift, where users refine their evaluation standards as they grade more LLM outputs. Recognizing the dependency of criteria on LLM outputs highlights new directions for designing future evaluation assistants.
本工作提出 EvalGen——一种将 LLM 生成的评估函数与人类偏好对齐的混合主动方法。EvalGen 辅助用户既制定可接受 LLM 输出的标准、又开发检查这些标准的函数,确保评估反映用户自己的评分标准。在一项与 9 位专家用户的定性研究中,我们观察到一种我们称为标准漂移的模式:用户随着给更多 LLM 输出打分而不断精炼其评估标准。认识到标准对 LLM 输出的依赖性,为设计未来评估助手指明了新方向。
(译注:结论之后为参考文献列表,按本站惯例不收录。)
附录 A 选择断言与征求打分的算法(ALGORITHMS FOR SELECTING ASSERTIONS & ELICITING GRADES)
A.1 断言选择率及其对 LLM 输出质量置信分的影响(Assertion Selectivity and Impact on LLM Output Quality Confidence)
One way to establish confidence in whether an LLM output is problematic is to assess the selectivity, or pass rate, of assertions that fail it. Intuitively, assertions that frequently fail outputs (low selectivity) provide limited insight into output quality. For example, an assertion that trivially fails every output offers no discernment and has a selectivity of 0.
判断一条 LLM 输出是否有问题,一种建立置信度的方法是考察判定它失败的断言的选择率(selectivity,即通过率)。直观上,频繁让输出失败(低选择率)的断言对输出质量的洞察有限。例如,一条对每个输出都平凡地失败的断言毫无辨别力,其选择率为 0。
EvalGen leverages selectivity estimates of assertions to assign a confidence score to each LLM output, indicating the likelihood it is of poor quality. The rationale is straightforward: an output is more likely to be problematic if failed by assertions known for their high selectivity. Concretely, for a set of assertions 𝐹 where each assertion 𝑓∈𝐹 returns 1 for a pass and 0 for a fail, we calculate the confidence score for an LLM output 𝑒 as follows:
𝜎(𝑒) = ∑_{𝑓∈𝐹} selectivity(𝑓)×𝑓(𝑒)
The score 𝜎 is always non-negative. A score of 0 means no assertions have failed the output, indicating a higher likelihood of quality, while lower scores, resulting from failures by non-selective assertions, point to uncertainty or potential issues with the output.
EvalGen 利用断言的选择率估计为每条 LLM 输出赋一个置信分,指示其为劣质的可能性。理由很直接:若一条输出被以高选择率著称的断言判失败,它更可能有问题。具体地,对断言集 F,其中每条断言 f ∈ F 通过返回 1、失败返回 0,我们如下计算 LLM 输出 e 的置信分:
σ(e) = Σ_{f∈F} selectivity(f) × f(e)
分数 σ 恒为非负。分数为 0 表示没有断言判该输出失败,指示质量较高的可能性;而较低的分数——由非选择性断言的失败所致——则表明该输出存在不确定性或潜在问题。
A.2 打分采样(Sampling Grades)
Given that users may not want to grade so many outputs in the EvalGen interface, choosing which outputs for users to grade is crucial for aligning the system’s evaluations with user expectations. Randomly selecting outputs without considering their predicted quality can lead to misalignment, especially if the selected samples aren’t representative of the entire dataset. Prior work also underscores the importance of soliciting a representative graded sample of LLM outputs [2, 45].
鉴于用户可能不想在 EvalGen 界面里给那么多输出打分,选择让用户给哪些输出打分,对于把系统评估与用户期望对齐至关重要。不考虑预测质量地随机选择输出可能导致错位,尤其当所选样本不能代表整个数据集时。先行工作也强调了征求有代表性的 LLM 输出打分样本的重要性 [2, 45]。
Given 𝜎 scores as previously defined, we consider a number of strategies to sample outputs for grading:
• Random: Sample outputs at random (uniformly)
• Highest: Sample the outputs with the highest 𝜎. This approach focuses on potentially problematic content.
• Lowest: Sample the outputs with the lowest 𝜎, prioritizing outputs that don’t fail any assertions or fail low-selectivity assertions.
• Alternating: Alternate between high and low 𝜎, aiming for a diverse sample with both bad and good outputs.
给定前述定义的 σ 分数,我们考虑若干为打分采样输出的策略:
- Random(随机):均匀随机地采样输出。
- Highest(最高):采样 σ 最高的输出。该策略聚焦于潜在的问题内容。
- Lowest(最低):采样 σ 最低的输出,优先选择不失败任何断言、或只失败低选择率断言的输出。
- Alternating(交替):在高 σ 与低 σ 之间交替,追求好坏输出兼具的多样样本。
In Section 5, we test these strategies against a random baseline on two different LLM pipelines. We employ an alternating sampling policy for the EvalGen user studies. We do not claim to have the best sampling policy; we chose an alternating policy with the hope that it would solicit a balanced sample of good and bad grades.
在第 5 节中,我们在两条不同 LLM 流水线上以随机为基线检验这些策略。在 EvalGen 用户研究中,我们采用交替采样策略。我们并不声称拥有最佳采样策略;我们选择交替策略,是希望它能征得好坏打分兼备的均衡样本。
One may wonder why we do not list a policy that ranks the outputs by score and samples the middle for grading. While this might seem akin to seeking out uncertain cases—as is common in active learning—our scores represent the likelihood of outputs being poor. They do not differentiate between good and bad per se. Therefore, outputs with low scores may still vary widely in quality, reflecting our system’s uncertainty.
有人可能奇怪,我们为何不列出「按分数排序并采样中段」的策略。这看起来类似主动学习(active learning)中常见的寻找不确定案例,但我们的分数代表输出低劣的可能性,本身并不区分好坏。因此,低分输出的质量仍可能千差万别,这反映的是系统自身的不确定性。
A.3 选择对齐的断言(Choosing Aligned Assertions)
Once the user stops grading and wants to see the alignment Report Card, as depicted in Figure 2e, for each criterion, we select the candidate assertion with the highest alignment score. We adopt notation from Shankar et al. [45] in defining alignment. Formally, let 𝐸 be a set of LLM pipeline input-output pairs and 𝑓:𝐸→{0,1} represent an assertion. Let 𝑦 be a binary vector, where 𝑦𝑖∈{0,1} represents whether the user thinks an LLM output 𝑒𝑖 is bad. Suppose 𝐹={𝑓1,𝑓2,...,𝑓𝑗} is a set of 𝑗 assertions. Concretely, the coverage and false failure rate (FFR) of 𝐹 is represented by the following equations:
Coverage(𝐹) = [∑𝑖 I[𝑦𝑖 = 0 ∧ (∃𝑓∈𝐹, 𝑓(𝑒𝑖) = 0)]] / [∑𝑖 I[𝑦𝑖 = 0]]
FFR(𝐹) = [∑𝑖 I[𝑦𝑖 = 1 ∧ (∃𝑓∈𝐹, 𝑓(𝑒𝑖) = 0)]] / [∑𝑖 I[𝑦𝑖 = 1]]
一旦用户停止打分、想查看图 2e 所示的对齐成绩单,我们就为每条标准选出对齐分数最高的候选断言。我们沿用 Shankar 等 [45] 的记号定义对齐度。形式化地,设 E 为 LLM 流水线输入-输出对的集合,f: E → {0,1} 表示一条断言。设 y 为二元向量,y_i ∈ {0,1} 表示用户是否认为 LLM 输出 e_i 为坏。设 F = {f1, f2, …, fj} 为 j 条断言的集合。具体地,F 的覆盖率与假失败率(FFR)由下列等式表示:
Coverage(F) = [Σ_i I[y_i = 0 ∧ (∃f∈F, f(e_i) = 0)]] / [Σ_i I[y_i = 0]] FFR(F) = [Σ_i I[y_i = 1 ∧ (∃f∈F, f(e_i) = 0)]] / [Σ_i I[y_i = 1]]
In both definitions, I is the indicator function. Intuitively, coverage represents the set’s true negative rate, while false failure rate represents the set’s false negative rate. An aligned set of assertions would have a high coverage and low false failure rate. We define the alignment of 𝐹 as the harmonic mean of coverage and the inverse of FFR:
Alignment(𝐹) = [2×Coverage(𝐹)×(1−FFR(𝐹))] / [Coverage(𝐹)+(1−FFR(𝐹))]
Note that alignment is very similar to F1 score; however, we are concerned with the precision and recall of failures (i.e., when 𝑓 = 0, not when 𝑓 = 1), and we are concerned with a set (i.e., when any assertion returns 0).
两个定义中,I 均为指示函数(indicator function)。直观上,覆盖率代表该集合的真负例率(true negative rate),假失败率代表该集合的假负例率(false negative rate)。一个对齐的断言集应有高覆盖率与低假失败率。我们把 F 的对齐度定义为覆盖率与 FFR 倒数的调和平均:
Alignment(F) = [2 × Coverage(F) × (1 − FFR(F))] / [Coverage(F) + (1 − FFR(F))]
注意对齐度与 F1 分数非常相似;但我们关心的是失败(即 f = 0 而非 f = 1 时)的精确率与召回率,且关心的是一个集合(即任一断言返回 0 时)。
A.4 采样策略评估(Evaluation of Sampling Policy)
We described in Appendix A.2 four options we considered to sample LLM outputs: random, highest, lowest, and alternating. Here, we compare EvalGen’s sampling policy, alternating, to these other three baselines. For this experiment, we sampled 16 outputs to grade, but in practice the user can grade more or fewer. Using the same LLM pipelines as described in Section 5.1, to assess sampling variance, we conducted 10 trials for each of the four sampling policies—where, for each trial, we kept the same set of candidate assertion functions.
我们在附录 A.2 描述了考虑过的四种 LLM 输出采样选项:随机、最高、最低与交替。这里我们把 EvalGen 的采样策略——交替——与另外三个基线比较。本实验中我们采样 16 条输出打分,但实践中用户可多可少。使用与 5.1 节相同的 LLM 流水线,为评估采样方差,我们对四种采样策略各做了 10 次试验——每次试验保持同一组候选断言函数。
The findings, shown in Figure 4, reveal that the random sampling policy exhibits a large variance in alignment. This inconsistency could lead to user frustration, particularly if the effort spent in grading outputs results in assertion sets with unpredictable relevance to their specified criteria. The alternative sampling strategies, which weight the probability for an LLM output to be graded by the selectivity (i.e., pass rate) of assertions that fail it, consistently yielded higher alignment scores across the entire datasets than the random policy. Notably, while the alternating policy didn’t consistently outperform, our results suggest that any non-random policy implemented in EvalGen may achieve satisfactory outcomes.
图 4 所示的发现表明,随机采样策略的对齐度方差很大。这种不一致可能导致用户沮丧,尤其当打分输出所付出的努力,换来的是与指定标准相关性不可预测的断言集时。另几种采样策略——按「判某输出失败的断言的选择率(即通过率)」对该输出被打分的概率加权——在整个数据集上一致取得高于随机策略的对齐分数。值得注意的是,虽然交替策略并未始终胜出,我们的结果表明,EvalGen 中实现的任何非随机策略都可能取得令人满意的结果。
[图 4: Alignments for assertion sets that result from different policies to sample grades from the user. Each policy was tested across 10 trials, with each involving a sample of 16 LLM outputs. Randomly sampling LLM outputs for grading introduces significant variance in alignment across the entire dataset.]
图 4:由不同「向用户采样打分」策略所得断言集的对齐度。每种策略测试 10 次试验,每次采样 16 条 LLM 输出。随机采样 LLM 输出打分,会在整个数据集上给对齐度引入显著方差。(图中横轴为采样策略:random、highest、lowest、alternating;纵轴为对齐度(0–0.6);按 product 与 medical 两个数据集分列。)
In this offline study, as shown in Figure 4, there’s no variation in the outcomes of the non-random policies because they are deterministic. However, in real-world use, EvalGen updates its predictions as it receives new information, so there could be some differences in results over time. Initially, when users start grading outputs in EvalGen, they might effectively be grading random outputs for the first one or two outputs, as the 𝜎 scores update and stabilize.
在这项离线研究中,如图 4 所示,非随机策略的结果没有变化,因为它们是确定性的。但在真实使用中,EvalGen 会随接收新信息更新其预测,因此结果可能随时间出现一些差异。最初,当用户开始在 EvalGen 中打分时,前一两次打的实际上可能是随机输出,因为 σ 分数需要更新和稳定。
附录 B 任务提示(TASK PROMPTS)
We prepared two prompts and corresponding datasets for tasks to present users (5.1). Both pipelines were adapted from prior work [21, 54] and correspond to medical record processing and a product description writing, respectively. The medical pipeline prompt is as follows:
```
You are extracting insights from some medical records. The records contain a medical note and a dialogue between a doctor and a patient. You need to extract values for the following: Chief complaint, History of present illness, Physical examination, Symptoms experienced by the patient, New medications prescribed or changed, including dosages (N/A if not provided), and Follow-up instructions (N/A if not provided). Your answer should not include any personal identifiable information (PII) such as name, age, gender, or ID. Use "the patient" instead of their name, for example. Return your answer as a bullet list, where each bullet is formatted like `chief complaint: xx.` If there is no value for the key, the value should be `N/A`. Keep your response around 150 words (you may have to summarize some extracted values to stay within the word limit). {transcript}
```
我们准备了两个提示及相应数据集,作为呈现给用户的任务(5.1 节)。两条流水线均改编自先行工作 [21, 54],分别对应医疗记录处理与商品描述写作。医疗流水线提示如下(附中文译文):
You are extracting insights from some medical records. The records contain a medical note and a dialogue between a doctor and a patient. You need to extract values for the following: Chief complaint, History of present illness, Physical examination, Symptoms experienced by the patient, New medications prescribed or changed, including dosages (N/A if not provided), and Follow-up instructions (N/A if not provided). Your answer should not include any personal identifiable information (PII) such as name, age, gender, or ID. Use "the patient" instead of their name, for example. Return your answer as a bullet list, where each bullet is formatted like `chief complaint: xx.` If there is no value for the key, the value should be `N/A`. Keep your response around 150 words (you may have to summarize some extracted values to stay within the word limit). {transcript}
译文:你将从一些医疗记录中提取洞见。记录包含一份医疗笔记和一段医生与患者的对话。你需要为以下各项提取取值:主诉(Chief complaint)、现病史(History of present illness)、体格检查(Physical examination)、患者经历的症状(Symptoms experienced by the patient)、新开处方或变更的药物及剂量(New medications prescribed or changed, including dosages,未提供则填 N/A)、以及随访指示(Follow-up instructions,未提供则填 N/A)。你的答案不得包含任何个人身份信息(PII),如姓名、年龄、性别或 ID。例如,应用「the patient(该患者)」代替患者姓名。以项目符号列表返回答案,每条格式如
chief complaint: xx.。若某键没有取值,取值应为N/A。回答控制在 150 词左右(为守字数限制,你可能需要概括一些提取值)。{transcript}
And the product pipeline prompt is as follows:
```
You are an expert copywriter. You need to write an e-commerce product description based on the product details and customer reviews. Your description should be SEO-optimized. It should use an active voice and include the product's features, benefits, unique selling points without overpromising, and a call to action for the buyer. Benefits describe how product features will work for the buyer, addressing exactly how the product will improve their lives. Clearly distinguish between features (e.g., lightweight, USB-chargeable) and benefits (e.g., convenience, nutritious drinks on-the-go). Don't mention weaknesses of the product or use generic or repetitive language. Don't make up review text or quotes. Don't include any links. Don't cite the reviews too heavily. Divide your description into readable chunks divided by relevant subheadings. Keep your description around 200 words, no more than 300, in Markdown format. {document}
```
商品流水线提示如下(附中文译文):
You are an expert copywriter. You need to write an e-commerce product description based on the product details and customer reviews. Your description should be SEO-optimized. It should use an active voice and include the product's features, benefits, unique selling points without overpromising, and a call to action for the buyer. Benefits describe how product features will work for the buyer, addressing exactly how the product will improve their lives. Clearly distinguish between features (e.g., lightweight, USB-chargeable) and benefits (e.g., convenience, nutritious drinks on-the-go). Don't mention weaknesses of the product or use generic or repetitive language. Don't make up review text or quotes. Don't include any links. Don't cite the reviews too heavily. Divide your description into readable chunks divided by relevant subheadings. Keep your description around 200 words, no more than 300, in Markdown format. {document}
译文:你是一位专家文案。你需要根据商品详情与顾客评论撰写一条电商商品描述。你的描述应做 SEO 优化,使用主动语态,并包含商品特性(features)、益处(benefits)、不过度承诺的独特卖点(unique selling points),以及面向买家的行动号召(call to action)。益处描述商品特性如何为买家发挥作用,确切说明商品将如何改善他们的生活。请清楚区分特性(如轻便、USB 充电)与益处(如便利、随身的营养饮品)。不要提及商品缺点,不要使用空泛或重复的语言。不要编造评论文本或引语。不要包含任何链接。不要过度引用评论。用相关小标题把描述切分为可读的段落块。描述控制在 200 词左右、不超过 300 词,采用 Markdown 格式。{document}
(译注:原文末行 “Received 03 April 2024” 为收稿日期,略去不译。)
要点速览
- 核心问题:用 LLM 评估 LLM 输出时,「验证者」自身继承 LLM 的所有缺陷(对措辞敏感、顺序偏差等),而现有 PE 工具几乎不提供验证评估器的支持,用户只能盲信。
- EvalGen 流程:嵌入 ChainForge 的向导式界面——LLM 建议自然语言标准(可增删改、可切换代码/LLM 评估器)→ GPT-4 流式生成多候选断言并立即执行 → 用户在等待期间对采样输出赞/踩打分 → 系统按对齐度选断言 → 成绩单展示每标准混淆矩阵与整体覆盖/假失败率。
- 对齐度定义:覆盖率(抓住用户认为坏的输出的比例)与假失败率(错杀好输出的比例)的调和平均;用户可设 FFR 阈值(界面默认 20%)。
- 离线对比 SPADE:医疗流水线 3 vs 5 条断言、对齐持平(48.29%);商品流水线 4 vs 9 条断言、覆盖率 0.73 vs 0.49、对齐 66.46% vs 54.35%——人工介入标准选择让断言更少且更贴用户所想;EvalGen 每条流水线仅用 16 条人工打分。
- 标准漂移(criteria drift):论文最著名的发现——打分需要标准,打分又塑造标准;标准依赖所见的具体 LLM 输出,不可能先验完全敲定,且模型/提示/上游一变就可能再次漂移。
- 用户研究(9 位从业者,NER 任务):对「断言与我的打分对齐」的 7 分制评分两极分化(1–6 分);先打分再定标准者事后被认为做法正确;看代码断言想直接核代码,LLM 断言则更难信任、对能否用于生产普遍怀疑。
- 打分采样的启示:随机采样导致对齐度方差巨大;按断言选择率加权的非随机采样(如好坏交替)稳定更优——「该让用户标哪些样本」本身就是个主动学习问题。
- 对评测方法论的挑战:对 AutoCalibrate 等「标准预先敲定+专家标签」的校准范式的适用性提出质疑;「对齐」可能永远是进行中的工作,不存在一组只需获取的地面真值标签。
- 工程启示:代码断言适合格式/计数/短语检查、可部署于关键路径;LLM 断言适合模糊质量判断但需持续维护;可设想 PR 评审 + CI/CD 式的断言协作与上线流程。
- 课程关联:与第 7 周评测单元的 LLM-as-a-Judge、tinyBenchmarks 及数据选择单元的 LIMA 互补——三者分别触及「谁当裁判」「评多少才够」「用什么数据教模型」,本文补充了「评估标准本身如何随观察演化」这一维度。
🖼 原文图表(5 张)—— 提取自原始文献,点击展开





以上图片提取自原始 PDF/网页或 arXiv 官方 HTML 版(含矢量图与原始图注),已镜像至本站;与正文的对应关系请以原文为准。