[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"blog-en-llm-golden-set-regression-protocol":3},{"id":4,"title":5,"author":6,"body":7,"cover":1739,"description":1740,"draft":1741,"extension":1742,"locale":1743,"meta":1744,"navigation":1745,"path":1746,"publishedAt":1747,"seo":1748,"seoTitle":1749,"slug":1750,"stem":1751,"tags":1752,"__hash__":1756},"blog\u002Fen\u002Fblog\u002Fllm-golden-set-regression-protocol.md","LLM regression protocol: golden sets, deterministic matchers, and update policy","Manuel Tardivo",{"type":8,"value":9,"toc":1702},"minimark",[10,15,24,28,33,45,49,64,68,92,107,111,122,128,133,156,166,170,177,181,191,195,479,487,519,727,765,769,902,906,913,1061,1073,1077,1084,1088,1118,1122,1166,1170,1213,1230,1234,1262,1266,1287,1291,1298,1302,1321,1325,1365,1369,1399,1403,1471,1482,1486,1493,1497,1527,1531,1577,1580,1630,1634,1653,1657,1674,1678,1684,1698],[11,12,14],"h2",{"id":13},"who-this-is-for","Who this is for",[16,17,18,19,23],"p",{},"Teams that already run an LLM system in production (RAG, agent, classifier) and have hit: ",[20,21,22],"em",{},"“I changed the prompt or upgraded the model and I don’t know what I broke.”"," This is not a beginner’s introduction to LLMs—it assumes you know what a prompt, an inference endpoint, and a deploy pipeline are.",[11,25,27],{"id":26},"the-problem-this-solves","The problem this solves",[29,30,32],"h3",{"id":31},"prompt-changes","Prompt changes",[16,34,35,36,40,41,44],{},"You tweak wording, add context, or tighten output format. The model family may be the same, but behavior shifts often and in small steps. You need a ",[37,38,39],"strong",{},"deterministic"," signal that, on ",[37,42,43],{},"known cases",", the system still satisfies the assertions you treat as correct (intent, JSON fields, forbidden phrases).",[29,46,48],{"id":47},"model-changes","Model changes",[16,50,51,52,55,56,59,60,63],{},"You move between versions or snapshots. The new model can look “better” on generic benchmarks and still regress on ",[37,53,54],{},"your"," inputs. These events are ",[37,57,58],{},"rare"," but ",[37,61,62],{},"high blast radius",": one deploy can change thousands of answers.",[29,65,67],{"id":66},"why-treat-them-separately","Why treat them separately",[69,70,71,82],"ul",{},[72,73,74,77,78,81],"li",{},[37,75,76],{},"Prompt changes:"," high frequency, localized impact; you want ",[37,79,80],{},"fast"," CI feedback (pre-merge).",[72,83,84,87,88,91],{},[37,85,86],{},"Model changes:"," low frequency, global impact; you want ",[37,89,90],{},"pre-deploy"," gates and often a wider review window.",[16,93,94,95,98,99,102,103,106],{},"The operational issue with “let another LLM score the output” is not ideology: it is not ",[37,96,97],{},"bit-repeatable",", it does not ",[37,100,101],{},"scale"," cleanly in CI without variance, and when it fails it rarely tells you ",[37,104,105],{},"which assertion"," broke. This article replaces that pattern with explicit golden cases, matchers, and update policy.",[11,108,110],{"id":109},"what-a-golden-set-is-and-is-not","What a golden set is (and is not)",[16,112,113,116,117,121],{},[37,114,115],{},"It is:"," a collection of ",[118,119,120],"code",{},"(input, expected assertions)"," that pin acceptable behavior on representative examples. Each case is a small executable spec.",[16,123,124,127],{},[37,125,126],{},"It is not:"," a fine-tuning dataset, an exhaustive long-tail test suite, or a vault of “perfect answers.” It does not replace production monitoring.",[16,129,130],{},[37,131,132],{},"How to pick cases:",[69,134,135,142,149],{},[72,136,137,138,141],{},"prioritize ",[37,139,140],{},"edge cases"," already seen failing in prod or staging;",[72,143,144,145,148],{},"cover the ",[37,146,147],{},"main input variants"," (not every combinatorial permutation);",[72,150,151,152,155],{},"if the system has ",[37,153,154],{},"multiple output shapes"," (e.g. tool JSON vs free text), include at least one case per shape.",[16,157,158,161,162,165],{},[37,159,160],{},"Size:"," for a medium RAG or classifier, ",[37,163,164],{},"20–50"," cases is a healthy band. Below ~20 the signal is thin; above ~200 maintenance hurts without dedicated tooling (split by domain, clear ownership).",[11,167,169],{"id":168},"file-format","File format",[16,171,172,173,176],{},"YAML works well: readable in review, ",[37,174,175],{},"Git-diff-friendly",", lintable.",[29,178,180],{"id":179},"suggested-repo-layout","Suggested repo layout",[182,183,188],"pre",{"className":184,"code":186,"language":187},[185],"language-text","eval\u002F\n  golden\u002F\n    cases\u002F\n      refund-intent-001.yaml\n      shipping-delay-002.yaml\n    README.md\n","text",[118,189,186],{"__ignoreMap":190},"",[29,192,194],{"id":193},"single-case-matchers-only-up-to-level-2","Single case (matchers only up to level 2)",[182,196,200],{"className":197,"code":198,"language":199,"meta":190,"style":190},"language-yaml shiki shiki-themes github-light github-dark","id: refund-intent-001\ndescription: \"User asks for refund with explicit order ID\"\ncreated_at: \"2026-05-01\"\nupdated_at: \"2026-05-01\"\nupdate_reason: \"Added from ticket #4412\"\ndeprecated: false\ninput:\n  user_message: \"I want a refund for order #1234\"\n  context: \"Verified customer; last order shipped 10 days ago.\"\nmatchers:\n  - type: json_field\n    field: intent\n    op: eq\n    value: refund\n  - type: json_field\n    field: confidence\n    op: gte\n    value: 0.85\n  - type: contains\n    value: \"1234\"\n  - type: not_contains\n    value: \"I cannot help you\"\nlevel: 2\ntags:\n  - refund\n  - entity-extraction\n","yaml",[118,201,202,219,230,241,251,262,274,283,294,305,313,327,338,349,360,371,381,391,401,413,423,435,445,456,464,471],{"__ignoreMap":190},[203,204,207,211,215],"span",{"class":205,"line":206},"line",1,[203,208,210],{"class":209},"s9eBZ","id",[203,212,214],{"class":213},"sVt8B",": ",[203,216,218],{"class":217},"sZZnC","refund-intent-001\n",[203,220,222,225,227],{"class":205,"line":221},2,[203,223,224],{"class":209},"description",[203,226,214],{"class":213},[203,228,229],{"class":217},"\"User asks for refund with explicit order ID\"\n",[203,231,233,236,238],{"class":205,"line":232},3,[203,234,235],{"class":209},"created_at",[203,237,214],{"class":213},[203,239,240],{"class":217},"\"2026-05-01\"\n",[203,242,244,247,249],{"class":205,"line":243},4,[203,245,246],{"class":209},"updated_at",[203,248,214],{"class":213},[203,250,240],{"class":217},[203,252,254,257,259],{"class":205,"line":253},5,[203,255,256],{"class":209},"update_reason",[203,258,214],{"class":213},[203,260,261],{"class":217},"\"Added from ticket #4412\"\n",[203,263,265,268,270],{"class":205,"line":264},6,[203,266,267],{"class":209},"deprecated",[203,269,214],{"class":213},[203,271,273],{"class":272},"sj4cs","false\n",[203,275,277,280],{"class":205,"line":276},7,[203,278,279],{"class":209},"input",[203,281,282],{"class":213},":\n",[203,284,286,289,291],{"class":205,"line":285},8,[203,287,288],{"class":209},"  user_message",[203,290,214],{"class":213},[203,292,293],{"class":217},"\"I want a refund for order #1234\"\n",[203,295,297,300,302],{"class":205,"line":296},9,[203,298,299],{"class":209},"  context",[203,301,214],{"class":213},[203,303,304],{"class":217},"\"Verified customer; last order shipped 10 days ago.\"\n",[203,306,308,311],{"class":205,"line":307},10,[203,309,310],{"class":209},"matchers",[203,312,282],{"class":213},[203,314,316,319,322,324],{"class":205,"line":315},11,[203,317,318],{"class":213},"  - ",[203,320,321],{"class":209},"type",[203,323,214],{"class":213},[203,325,326],{"class":217},"json_field\n",[203,328,330,333,335],{"class":205,"line":329},12,[203,331,332],{"class":209},"    field",[203,334,214],{"class":213},[203,336,337],{"class":217},"intent\n",[203,339,341,344,346],{"class":205,"line":340},13,[203,342,343],{"class":209},"    op",[203,345,214],{"class":213},[203,347,348],{"class":217},"eq\n",[203,350,352,355,357],{"class":205,"line":351},14,[203,353,354],{"class":209},"    value",[203,356,214],{"class":213},[203,358,359],{"class":217},"refund\n",[203,361,363,365,367,369],{"class":205,"line":362},15,[203,364,318],{"class":213},[203,366,321],{"class":209},[203,368,214],{"class":213},[203,370,326],{"class":217},[203,372,374,376,378],{"class":205,"line":373},16,[203,375,332],{"class":209},[203,377,214],{"class":213},[203,379,380],{"class":217},"confidence\n",[203,382,384,386,388],{"class":205,"line":383},17,[203,385,343],{"class":209},[203,387,214],{"class":213},[203,389,390],{"class":217},"gte\n",[203,392,394,396,398],{"class":205,"line":393},18,[203,395,354],{"class":209},[203,397,214],{"class":213},[203,399,400],{"class":272},"0.85\n",[203,402,404,406,408,410],{"class":205,"line":403},19,[203,405,318],{"class":213},[203,407,321],{"class":209},[203,409,214],{"class":213},[203,411,412],{"class":217},"contains\n",[203,414,416,418,420],{"class":205,"line":415},20,[203,417,354],{"class":209},[203,419,214],{"class":213},[203,421,422],{"class":217},"\"1234\"\n",[203,424,426,428,430,432],{"class":205,"line":425},21,[203,427,318],{"class":213},[203,429,321],{"class":209},[203,431,214],{"class":213},[203,433,434],{"class":217},"not_contains\n",[203,436,438,440,442],{"class":205,"line":437},22,[203,439,354],{"class":209},[203,441,214],{"class":213},[203,443,444],{"class":217},"\"I cannot help you\"\n",[203,446,448,451,453],{"class":205,"line":447},23,[203,449,450],{"class":209},"level",[203,452,214],{"class":213},[203,454,455],{"class":272},"2\n",[203,457,459,462],{"class":205,"line":458},24,[203,460,461],{"class":209},"tags",[203,463,282],{"class":213},[203,465,467,469],{"class":205,"line":466},25,[203,468,318],{"class":213},[203,470,359],{"class":217},[203,472,474,476],{"class":205,"line":473},26,[203,475,318],{"class":213},[203,477,478],{"class":217},"entity-extraction\n",[29,480,482,483,486],{"id":481},"example-with-tier-3-json_subset-and-extra-keys-across-model-versions","Example with tier 3 (",[118,484,485],{},"json_subset"," and extra keys across model versions)",[16,488,489,490,493,494,497,498,501,502,505,506,509,510,514,515,518],{},"The API (or tool) returns JSON with ",[37,491,492],{},"extra keys"," that differ by model version or snapshot (",[118,495,496],{},"model_reasoning",", ",[118,499,500],{},"debug_trace",", experimental fields). You only care that a ",[37,503,504],{},"subset"," of key\u002Fvalue pairs holds: intent, operational status, etc. ",[37,507,508],{},"Tier 2 alone"," forces you either to assert every irrelevant new key away or to churn the golden constantly. Use ",[37,511,512],{},[118,513,485],{},": the parsed output must contain ",[37,516,517],{},"at least"," the expected object; everything else is ignored.",[182,520,522],{"className":197,"code":521,"language":199,"meta":190,"style":190},"id: tool-output-contract-015\ndescription: \"Tracking tool: model adds extra keys; client contract is intent + status + confidence only\"\ncreated_at: \"2026-05-12\"\nupdated_at: \"2026-05-12\"\nupdate_reason: \"After model upgrade undocumented optional keys appear in payload\"\ndeprecated: false\ninput:\n  user_message: \"Where is my package?\"\nmatchers:\n  - type: json_subset\n    expected:\n      intent: tracking\n      status: ok\n  - type: json_field\n    field: confidence\n    op: gte\n    value: 0.72\n  - type: normalized_text_equal\n    field: summary\n    expected: \"The parcel is in transit to the operations hub.\"\nlevel: 3\ntags:\n  - tool-json\n  - tracking\n",[118,523,524,533,542,551,559,568,576,582,591,597,608,615,625,635,645,653,661,670,681,690,699,708,714,721],{"__ignoreMap":190},[203,525,526,528,530],{"class":205,"line":206},[203,527,210],{"class":209},[203,529,214],{"class":213},[203,531,532],{"class":217},"tool-output-contract-015\n",[203,534,535,537,539],{"class":205,"line":221},[203,536,224],{"class":209},[203,538,214],{"class":213},[203,540,541],{"class":217},"\"Tracking tool: model adds extra keys; client contract is intent + status + confidence only\"\n",[203,543,544,546,548],{"class":205,"line":232},[203,545,235],{"class":209},[203,547,214],{"class":213},[203,549,550],{"class":217},"\"2026-05-12\"\n",[203,552,553,555,557],{"class":205,"line":243},[203,554,246],{"class":209},[203,556,214],{"class":213},[203,558,550],{"class":217},[203,560,561,563,565],{"class":205,"line":253},[203,562,256],{"class":209},[203,564,214],{"class":213},[203,566,567],{"class":217},"\"After model upgrade undocumented optional keys appear in payload\"\n",[203,569,570,572,574],{"class":205,"line":264},[203,571,267],{"class":209},[203,573,214],{"class":213},[203,575,273],{"class":272},[203,577,578,580],{"class":205,"line":276},[203,579,279],{"class":209},[203,581,282],{"class":213},[203,583,584,586,588],{"class":205,"line":285},[203,585,288],{"class":209},[203,587,214],{"class":213},[203,589,590],{"class":217},"\"Where is my package?\"\n",[203,592,593,595],{"class":205,"line":296},[203,594,310],{"class":209},[203,596,282],{"class":213},[203,598,599,601,603,605],{"class":205,"line":307},[203,600,318],{"class":213},[203,602,321],{"class":209},[203,604,214],{"class":213},[203,606,607],{"class":217},"json_subset\n",[203,609,610,613],{"class":205,"line":315},[203,611,612],{"class":209},"    expected",[203,614,282],{"class":213},[203,616,617,620,622],{"class":205,"line":329},[203,618,619],{"class":209},"      intent",[203,621,214],{"class":213},[203,623,624],{"class":217},"tracking\n",[203,626,627,630,632],{"class":205,"line":340},[203,628,629],{"class":209},"      status",[203,631,214],{"class":213},[203,633,634],{"class":217},"ok\n",[203,636,637,639,641,643],{"class":205,"line":351},[203,638,318],{"class":213},[203,640,321],{"class":209},[203,642,214],{"class":213},[203,644,326],{"class":217},[203,646,647,649,651],{"class":205,"line":362},[203,648,332],{"class":209},[203,650,214],{"class":213},[203,652,380],{"class":217},[203,654,655,657,659],{"class":205,"line":373},[203,656,343],{"class":209},[203,658,214],{"class":213},[203,660,390],{"class":217},[203,662,663,665,667],{"class":205,"line":383},[203,664,354],{"class":209},[203,666,214],{"class":213},[203,668,669],{"class":272},"0.72\n",[203,671,672,674,676,678],{"class":205,"line":393},[203,673,318],{"class":213},[203,675,321],{"class":209},[203,677,214],{"class":213},[203,679,680],{"class":217},"normalized_text_equal\n",[203,682,683,685,687],{"class":205,"line":403},[203,684,332],{"class":209},[203,686,214],{"class":213},[203,688,689],{"class":217},"summary\n",[203,691,692,694,696],{"class":205,"line":415},[203,693,612],{"class":209},[203,695,214],{"class":213},[203,697,698],{"class":217},"\"The parcel is in transit to the operations hub.\"\n",[203,700,701,703,705],{"class":205,"line":425},[203,702,450],{"class":209},[203,704,214],{"class":213},[203,706,707],{"class":272},"3\n",[203,709,710,712],{"class":205,"line":437},[203,711,461],{"class":209},[203,713,282],{"class":213},[203,715,716,718],{"class":205,"line":447},[203,717,318],{"class":213},[203,719,720],{"class":217},"tool-json\n",[203,722,723,725],{"class":205,"line":458},[203,724,318],{"class":213},[203,726,624],{"class":217},[16,728,729,731,732,735,736,739,740,743,744,749,750,753,754,757,758,760,761,764],{},[118,730,485],{}," pins the ",[37,733,734],{},"minimal structure","; ",[118,737,738],{},"json_field"," on ",[118,741,742],{},"confidence"," keeps a local threshold check. ",[37,745,746],{},[118,747,748],{},"normalized_text_equal"," (still tier 3 here) applies ",[37,751,752],{},"runner-defined, fixed"," normalization (e.g. NFC, lowercase, collapsed whitespace) before comparing the ",[118,755,756],{},"summary"," field—useful when the model tweaks punctuation or spacing but meaning must stay equivalent. Case ",[118,759,450],{}," is ",[37,762,763],{},"3"," because the weakest structural matcher (not byte-for-byte on the whole JSON) is subset\u002Fnormalization.",[29,766,768],{"id":767},"field-reference","Field reference",[770,771,772,785],"table",{},[773,774,775],"thead",{},[776,777,778,782],"tr",{},[779,780,781],"th",{},"Field",[779,783,784],{},"Role",[786,787,788,801,810,828,837,849,858,870,893],"tbody",{},[776,789,790,795],{},[791,792,793],"td",{},[118,794,210],{},[791,796,797,798,800],{},"Stable key for the CI runner and reports. Do not rename casually—prefer ",[118,799,267],{}," plus a new file.",[776,802,803,807],{},[791,804,805],{},[118,806,224],{},[791,808,809],{},"Human context for review: why this case exists.",[776,811,812,819],{},[791,813,814,816,817],{},[118,815,235],{}," \u002F ",[118,818,246],{},[791,820,821,822,824,825,827],{},"Audit trail; any contract change updates ",[118,823,246],{}," and ",[118,826,256],{},".",[776,829,830,834],{},[791,831,832],{},[118,833,256],{},[791,835,836],{},"Required when assertions or inputs change: explains the decision.",[776,838,839,843],{},[791,840,841],{},[118,842,267],{},[791,844,845,848],{},[118,846,847],{},"true"," when the case must no longer gate CI but stays in repo for history.",[776,850,851,855],{},[791,852,853],{},[118,854,279],{},[791,856,857],{},"Payload the runner passes into the function or endpoint under test.",[776,859,860,864],{},[791,861,862],{},[118,863,310],{},[791,865,866,867,869],{},"Ordered list of ",[37,868,39],{}," checks (see tier table).",[776,871,872,876],{},[791,873,874],{},[118,875,450],{},[791,877,878,881,882,885,886,889,890,892],{},[37,879,880],{},"Maximum"," matcher tier used in the case (",[118,883,884],{},"1"," … ",[118,887,888],{},"5","). Used to aggregate reports (“how many cases touch tier 4?”). Must match the ",[118,891,321],{}," list.",[776,894,895,899],{},[791,896,897],{},[118,898,461],{},[791,900,901],{},"Filters in dashboards or product ownership.",[29,903,905],{"id":904},"example-with-tier-4-embedding","Example with tier 4 (embedding)",[16,907,908,909,912],{},"Use when answers are ",[37,910,911],{},"free-form"," but you still want an automated signal before human review:",[182,914,916],{"className":197,"code":915,"language":199,"meta":190,"style":190},"id: tone-paraphrase-010\ndescription: \"Empathetic reply equivalent, not literal\"\ncreated_at: \"2026-05-10\"\nupdated_at: \"2026-05-10\"\nupdate_reason: \"Threshold tuned on 30 human-labeled pairs\"\ndeprecated: false\ninput:\n  user_message: \"I am really upset about the delay.\"\nmatchers:\n  - type: embedding_similarity\n    golden_text: \"I understand the frustration about the delay; here is the latest on your shipment.\"\n    threshold: 0.88\n    model: text-embedding-3-small\nlevel: 4\ntags:\n  - tone\n  - shipping\n",[118,917,918,927,936,945,953,962,970,976,985,991,1002,1012,1022,1032,1041,1047,1054],{"__ignoreMap":190},[203,919,920,922,924],{"class":205,"line":206},[203,921,210],{"class":209},[203,923,214],{"class":213},[203,925,926],{"class":217},"tone-paraphrase-010\n",[203,928,929,931,933],{"class":205,"line":221},[203,930,224],{"class":209},[203,932,214],{"class":213},[203,934,935],{"class":217},"\"Empathetic reply equivalent, not literal\"\n",[203,937,938,940,942],{"class":205,"line":232},[203,939,235],{"class":209},[203,941,214],{"class":213},[203,943,944],{"class":217},"\"2026-05-10\"\n",[203,946,947,949,951],{"class":205,"line":243},[203,948,246],{"class":209},[203,950,214],{"class":213},[203,952,944],{"class":217},[203,954,955,957,959],{"class":205,"line":253},[203,956,256],{"class":209},[203,958,214],{"class":213},[203,960,961],{"class":217},"\"Threshold tuned on 30 human-labeled pairs\"\n",[203,963,964,966,968],{"class":205,"line":264},[203,965,267],{"class":209},[203,967,214],{"class":213},[203,969,273],{"class":272},[203,971,972,974],{"class":205,"line":276},[203,973,279],{"class":209},[203,975,282],{"class":213},[203,977,978,980,982],{"class":205,"line":285},[203,979,288],{"class":209},[203,981,214],{"class":213},[203,983,984],{"class":217},"\"I am really upset about the delay.\"\n",[203,986,987,989],{"class":205,"line":296},[203,988,310],{"class":209},[203,990,282],{"class":213},[203,992,993,995,997,999],{"class":205,"line":307},[203,994,318],{"class":213},[203,996,321],{"class":209},[203,998,214],{"class":213},[203,1000,1001],{"class":217},"embedding_similarity\n",[203,1003,1004,1007,1009],{"class":205,"line":315},[203,1005,1006],{"class":209},"    golden_text",[203,1008,214],{"class":213},[203,1010,1011],{"class":217},"\"I understand the frustration about the delay; here is the latest on your shipment.\"\n",[203,1013,1014,1017,1019],{"class":205,"line":329},[203,1015,1016],{"class":209},"    threshold",[203,1018,214],{"class":213},[203,1020,1021],{"class":272},"0.88\n",[203,1023,1024,1027,1029],{"class":205,"line":340},[203,1025,1026],{"class":209},"    model",[203,1028,214],{"class":213},[203,1030,1031],{"class":217},"text-embedding-3-small\n",[203,1033,1034,1036,1038],{"class":205,"line":351},[203,1035,450],{"class":209},[203,1037,214],{"class":213},[203,1039,1040],{"class":272},"4\n",[203,1042,1043,1045],{"class":205,"line":362},[203,1044,461],{"class":209},[203,1046,282],{"class":213},[203,1048,1049,1051],{"class":205,"line":373},[203,1050,318],{"class":213},[203,1052,1053],{"class":217},"tone\n",[203,1055,1056,1058],{"class":205,"line":383},[203,1057,318],{"class":213},[203,1059,1060],{"class":217},"shipping\n",[16,1062,1063,1064,760,1066,1069,1070,827],{},"Here ",[118,1065,450],{},[37,1067,1068],{},"4"," because the top matcher is semantic; do not label such a case ",[118,1071,1072],{},"level: 2",[11,1074,1076],{"id":1075},"matcher-tiers-1-5","Matcher tiers (1 → 5)",[16,1078,1079,1080,1083],{},"Standing rule: ",[37,1081,1082],{},"pick the lowest tier that works."," If most cases land at 4 or 5, your output contract is probably too open—not your testers’ fault.",[29,1085,1087],{"id":1086},"tier-1-exact-equality","Tier 1 — Exact equality",[69,1089,1090,1096,1106,1112],{},[72,1091,1092,1095],{},[37,1093,1094],{},"What:"," byte-for-byte string equality on the canonical serialized output (or strict rules you document as tier 1).",[72,1097,1098,1101,1102,1105],{},[37,1099,1100],{},"Example type:"," ",[118,1103,1104],{},"exact_text"," on deterministic JSON serialization (sorted keys if your runner always emits the same shape).",[72,1107,1108,1111],{},[37,1109,1110],{},"When:"," rigid templates, temperature 0, no acceptable surface variation.",[72,1113,1114,1117],{},[37,1115,1116],{},"Cost:"," negligible.",[29,1119,1121],{"id":1120},"tier-2-structure-or-substring-predicates","Tier 2 — Structure or substring predicates",[69,1123,1124,1156,1161],{},[72,1125,1126,1101,1128,1130,1131,497,1134,497,1137,497,1140,497,1143,1146,1147,497,1150,497,1153,827],{},[37,1127,1094],{},[118,1129,738],{}," (",[118,1132,1133],{},"eq",[118,1135,1136],{},"ne",[118,1138,1139],{},"gte",[118,1141,1142],{},"lte",[118,1144,1145],{},"in","), ",[118,1148,1149],{},"contains",[118,1151,1152],{},"not_contains",[118,1154,1155],{},"regex",[72,1157,1158,1160],{},[37,1159,1110],{}," structured tool output, intent + confidence, lexical bans.",[72,1162,1163,1165],{},[37,1164,1116],{}," linear in output length, all local.",[29,1167,1169],{"id":1168},"tier-3-structural-contract-or-controlled-normalization","Tier 3 — Structural contract or controlled normalization",[69,1171,1172,1192,1208],{},[72,1173,1174,1101,1176,1178,1179,1146,1182,1185,1186,1130,1188,1191],{},[37,1175,1094],{},[118,1177,485],{}," (parsed output contains at least the nested pairs under ",[118,1180,1181],{},"expected",[118,1183,1184],{},"json_schema"," validation, ",[118,1187,748],{},[37,1189,1190],{},"runner-fixed"," normalization rules, e.g. NFC + lowercase + collapsed whitespace, then equality on the named field).",[72,1193,1194,1101,1196,1199,1200,1203,1204,1207],{},[37,1195,1110],{},[37,1197,1198],{},"Extra"," keys are acceptable or vary by model version (see ",[118,1201,1202],{},"tool-output-contract-015"," above); free text in one field where only the ",[37,1205,1206],{},"normalized surface"," must match.",[72,1209,1210,1212],{},[37,1211,1116],{}," JSON parse + optional schema validation; all local.",[16,1214,1215,1218,1219,1221,1222,731,1226,1229],{},[37,1216,1217],{},"Tier 2 vs 3:"," at tier 2 you drive ",[118,1220,738],{}," for every field you care about and keep updating the golden whenever a new irrelevant key appears. At tier 3 ",[37,1223,1224],{},[118,1225,485],{},[37,1227,1228],{},"sub-contract"," that matters and ignores structural noise outside it.",[29,1231,1233],{"id":1232},"tier-4-embedding-similarity","Tier 4 — Embedding similarity",[69,1235,1236,1252,1257],{},[72,1237,1238,1101,1240,1243,1244,1247,1248,1251],{},[37,1239,1094],{},[118,1241,1242],{},"embedding_similarity"," between output and ",[118,1245,1246],{},"golden_text"," (or cached golden embedding vs actual), with a ",[37,1249,1250],{},"threshold"," calibrated on a human validation slice.",[72,1253,1254,1256],{},[37,1255,1110],{}," paraphrases are OK; you need a measurable semantic band.",[72,1258,1259,1261],{},[37,1260,1116],{}," one or two embedding calls per case (cache the golden side in CI when possible).",[29,1263,1265],{"id":1264},"tier-5-human-review-only","Tier 5 — Human review only",[69,1267,1268,1273,1278],{},[72,1269,1270,1272],{},[37,1271,1094],{}," the case documents inputs and qualitative criteria (brand, compliance, legal edge) you will not encode as a reliable binary matcher.",[72,1274,1275,1277],{},[37,1276,1110],{}," no generic automation should gate pass\u002Ffail; humans or a bespoke process decide.",[72,1279,1280,1282,1283,1286],{},[37,1281,1116],{}," human time; ",[37,1284,1285],{},"never"," treat as automatic green in CI.",[11,1288,1290],{"id":1289},"ci-integration-contract","CI integration contract",[16,1292,1293,1294,1297],{},"No mandate for Vitest, GitHub Actions, etc.—define a ",[37,1295,1296],{},"contract"," any runner can implement.",[29,1299,1301],{"id":1300},"inputs","Inputs",[69,1303,1304,1307,1314],{},[72,1305,1306],{},"Directory or manifest of golden YAML files.",[72,1308,1309,1310,1313],{},"Function or endpoint under test (prompt and model version ",[37,1311,1312],{},"pinned"," or identified by pipeline env vars).",[72,1315,1316,1317,1320],{},"Runner config: timeouts, whether retries are allowed (usually ",[37,1318,1319],{},"no"," for regression checks), secrets path for embeddings if tier 4 is used.",[29,1322,1324],{"id":1323},"outputs-machine-human","Outputs (machine + human)",[69,1326,1327,1335,1346,1362],{},[72,1328,1329,1330,827],{},"Pass\u002Ffail counts overall and ",[37,1331,1332,1333],{},"by max case ",[118,1334,450],{},[72,1336,1337,1338,1341,1342,1345],{},"For each failure: which ",[118,1339,1340],{},"matcher"," broke, ",[37,1343,1344],{},"expected vs actual"," (truncated strings, JSON paths, similarity scores).",[72,1347,1348,1353,1354,1357,1358,1361],{},[37,1349,1350],{},[118,1351,1352],{},"review_required: true"," on every ",[37,1355,1356],{},"tier 4"," failure (evidence for review tied to a possible override) and on all ",[37,1359,1360],{},"tier 5"," cases.",[72,1363,1364],{},"Optional: textual or pretty JSON diff—mask PII such as order IDs in logs.",[29,1366,1368],{"id":1367},"when-to-run","When to run",[69,1370,1371,1381,1389],{},[72,1372,1373,1376,1377,1380],{},[37,1374,1375],{},"Every prompt change"," touching the system: mandatory run ",[37,1378,1379],{},"pre-merge"," (or pre-push for tiny teams).",[72,1382,1383,1386,1387,827],{},[37,1384,1385],{},"Every model change"," (API version, snapshot, production-default temperature): mandatory ",[37,1388,90],{},[72,1390,1391,1394,1395,1398],{},[37,1392,1393],{},"Periodic runs"," (e.g. weekly) with no code changes: providers ship silent updates—",[37,1396,1397],{},"model drift"," happens even if your lockfile did not move.",[29,1400,1402],{"id":1401},"failure-policy-recommended","Failure policy (recommended)",[770,1404,1405,1415],{},[773,1406,1407],{},[776,1408,1409,1412],{},[779,1410,1411],{},"Tier involved in the failure",[779,1413,1414],{},"Pipeline behavior",[786,1416,1417,1428,1461],{},[776,1418,1419,1422],{},[791,1420,1421],{},"1–3",[791,1423,1424,1427],{},[37,1425,1426],{},"Fail"," blocks merge\u002Fdeploy.",[776,1429,1430,1432],{},[791,1431,1068],{},[791,1433,1434,1437,1438,1441,1442,1445,1446,1449,1450,1453,1454,1456,1457,1460],{},[37,1435,1436],{},"Default under this protocol:"," same as 1–3 — ",[37,1439,1440],{},"fail blocks"," merge\u002Fdeploy. ",[37,1443,1444],{},"Overrides"," are allowed only with ",[37,1447,1448],{},"explicit, traceable approval"," (linked ticket, PR label, or mandatory signed field in the CI report from an owner) after reading the report (similarity score, actual snippet). Do not clear red without a trail: tier 4 is inherently ",[37,1451,1452],{},"less binary"," than ",[118,1455,1133],{}," and would otherwise hollow out CI. Larger orgs may soften this ",[37,1458,1459],{},"only"," by replacing the default in written policy.",[776,1462,1463,1465],{},[791,1464,888],{},[791,1466,1467,1470],{},[37,1468,1469],{},"Never"," a final automatic pass\u002Ffail; human queue only.",[16,1472,1473,1474,1477,1478,1481],{},"Blocking tier 4 by default stops merges where the only signal was “embedding below threshold”: thresholds have ",[37,1475,1476],{},"variance"," (embedding model, text, tokens). Review must be a ",[37,1479,1480],{},"deliberate step",", not a hole in the pipeline.",[11,1483,1485],{"id":1484},"golden-set-update-policy","Golden set update policy",[16,1487,1488,1489,1492],{},"A stale golden set is worse than none: it creates ",[37,1490,1491],{},"false confidence",". This section is the social contract between engineering and product.",[29,1494,1496],{"id":1495},"when-to-update","When to update",[69,1498,1499,1510,1517],{},[72,1500,1501,1502,1505,1506,816,1508,827],{},"A ",[37,1503,1504],{},"new production input shape"," appears → add a case with ",[118,1507,235],{},[118,1509,256],{},[72,1511,1512,1513,1516],{},"A case fails but the new output is ",[37,1514,1515],{},"correct under an updated spec"," → fix the golden, document why the old expectation was wrong.",[72,1518,1519,1522,1523,1526],{},[37,1520,1521],{},"Intentional behavior change"," (refund policy, tone) → golden updates ship in the ",[37,1524,1525],{},"same"," change request as code or prompt.",[29,1528,1530],{"id":1529},"how-to-update-operational-rules","How to update (operational rules)",[69,1532,1533,1549,1559],{},[72,1534,1535,1538,1539,1541,1542,1544,1545,1548],{},[37,1536,1537],{},"No silent auto-updates"," from CI (“overwrite expected with actual”). Every change to ",[118,1540,310],{}," or ",[118,1543,279],{}," is a ",[37,1546,1547],{},"human"," decision with a trail.",[72,1550,1551,1552,824,1554,1558],{},"Every edit bumps ",[118,1553,246],{},[37,1555,1556],{},[118,1557,256],{}," (a single sentence in YAML is enough).",[72,1560,1561,1562,1565,1566,816,1569,1572,1573,1576],{},"Do not hard-delete cases overnight: set ",[118,1563,1564],{},"deprecated: true",", optionally ",[118,1567,1568],{},"deprecated_at",[118,1570,1571],{},"deprecated_reason",", and keep at least ",[37,1574,1575],{},"one release cycle"," for grep and audit.",[16,1578,1579],{},"Deprecation example:",[182,1581,1583],{"className":197,"code":1582,"language":199,"meta":190,"style":190},"id: refund-intent-legacy-000\ndescription: \"Legacy wording pre April 2026 spec\"\ndeprecated: true\ndeprecated_at: \"2026-05-15\"\ndeprecated_reason: \"Replaced by refund-intent-001 after order-ID policy change\"\n",[118,1584,1585,1594,1603,1612,1621],{"__ignoreMap":190},[203,1586,1587,1589,1591],{"class":205,"line":206},[203,1588,210],{"class":209},[203,1590,214],{"class":213},[203,1592,1593],{"class":217},"refund-intent-legacy-000\n",[203,1595,1596,1598,1600],{"class":205,"line":221},[203,1597,224],{"class":209},[203,1599,214],{"class":213},[203,1601,1602],{"class":217},"\"Legacy wording pre April 2026 spec\"\n",[203,1604,1605,1607,1609],{"class":205,"line":232},[203,1606,267],{"class":209},[203,1608,214],{"class":213},[203,1610,1611],{"class":272},"true\n",[203,1613,1614,1616,1618],{"class":205,"line":243},[203,1615,1568],{"class":209},[203,1617,214],{"class":213},[203,1619,1620],{"class":217},"\"2026-05-15\"\n",[203,1622,1623,1625,1627],{"class":205,"line":253},[203,1624,1571],{"class":209},[203,1626,214],{"class":213},[203,1628,1629],{"class":217},"\"Replaced by refund-intent-001 after order-ID policy change\"\n",[29,1631,1633],{"id":1632},"degraded-set-signal","Degraded set signal",[16,1635,1636,1637,1640,1641,1644,1645,1648,1649,1652],{},"If a ",[37,1638,1639],{},"single"," model upgrade forces you to rewrite assertions for ",[37,1642,1643],{},"more than ~30%"," of cases, the set is too coupled to ",[37,1646,1647],{},"surface form"," (verbatim punctuation) instead of ",[37,1650,1651],{},"semantic contract",". Fix by pushing matchers down to tiers 2–3 (structured fields), tightening schemas, and reducing unnecessary free text.",[11,1654,1656],{"id":1655},"out-of-scope","Out of scope",[69,1658,1659,1665],{},[72,1660,1661,1664],{},[37,1662,1663],{},"LLM-as-judge evaluators"," as the primary gate: different pattern, not deterministic in the sense used here.",[72,1666,1667,497,1670,1673],{},[37,1668,1669],{},"Live A\u002FB traffic",[37,1671,1672],{},"fine-tuning evals",", commercial frameworks (LangSmith, RAGAS, …): useful, but not required to implement the standard above with an internal script.",[11,1675,1677],{"id":1676},"closing","Closing",[16,1679,1680,1681],{},"The takeaway question: ",[20,1682,1683],{},"“If I change the model tomorrow morning, how long until I know whether I broke my real cases?”",[16,1685,1686,1687,1541,1690,1693,1694,1697],{},"If the answer is ",[20,1688,1689],{},"I don’t know",[20,1691,1692],{},"about a day",", you need an operational golden set. If it becomes ",[20,1695,1696],{},"ten minutes after the pipeline finishes",", this article did its job.",[1699,1700,1701],"style",{},"html pre.shiki code .s9eBZ, html code.shiki .s9eBZ{--shiki-default:#22863A;--shiki-dark:#85E89D}html pre.shiki code .sVt8B, html code.shiki .sVt8B{--shiki-default:#24292E;--shiki-dark:#E1E4E8}html pre.shiki code .sZZnC, html code.shiki .sZZnC{--shiki-default:#032F62;--shiki-dark:#9ECBFF}html pre.shiki code .sj4cs, html code.shiki .sj4cs{--shiki-default:#005CC5;--shiki-dark:#79B8FF}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":190,"searchDepth":221,"depth":221,"links":1703},[1704,1705,1710,1711,1719,1726,1732,1737,1738],{"id":13,"depth":221,"text":14},{"id":26,"depth":221,"text":27,"children":1706},[1707,1708,1709],{"id":31,"depth":232,"text":32},{"id":47,"depth":232,"text":48},{"id":66,"depth":232,"text":67},{"id":109,"depth":221,"text":110},{"id":168,"depth":221,"text":169,"children":1712},[1713,1714,1715,1717,1718],{"id":179,"depth":232,"text":180},{"id":193,"depth":232,"text":194},{"id":481,"depth":232,"text":1716},"Example with tier 3 (json_subset and extra keys across model versions)",{"id":767,"depth":232,"text":768},{"id":904,"depth":232,"text":905},{"id":1075,"depth":221,"text":1076,"children":1720},[1721,1722,1723,1724,1725],{"id":1086,"depth":232,"text":1087},{"id":1120,"depth":232,"text":1121},{"id":1168,"depth":232,"text":1169},{"id":1232,"depth":232,"text":1233},{"id":1264,"depth":232,"text":1265},{"id":1289,"depth":221,"text":1290,"children":1727},[1728,1729,1730,1731],{"id":1300,"depth":232,"text":1301},{"id":1323,"depth":232,"text":1324},{"id":1367,"depth":232,"text":1368},{"id":1401,"depth":232,"text":1402},{"id":1484,"depth":221,"text":1485,"children":1733},[1734,1735,1736],{"id":1495,"depth":232,"text":1496},{"id":1529,"depth":232,"text":1530},{"id":1632,"depth":232,"text":1633},{"id":1655,"depth":221,"text":1656},{"id":1676,"depth":221,"text":1677},"\u002Fblog\u002F22_llm_golden_set_regression_protocol_1x.webp","A golden set for LLM regressions on prompt and model changes, without LLM-as-judge: file format, matcher tiers, CI policy, and update rules.",false,"md","en",{},true,"\u002Fen\u002Fblog\u002Fllm-golden-set-regression-protocol","2026-06-16",{"title":5,"description":1740},"LLM golden set regression without LLM-as-judge","llm-golden-set-regression-protocol","en\u002Fblog\u002Fllm-golden-set-regression-protocol",[1753,1754,1755],"mvp","llm","automation","wv_mmZANK0utx2mvaqLfnnfI1M8pYSmypufBVjmklxo"]