自洽性：noul
把不確定的機率交給人工複核，同時讓底層的 noul 值保持可見。
這個 cookbook 拿一份汽車保險理賠，對它跑 14 道題的量規 15 次，
檢查每個答案在反覆執行之間是否保持一致。每一項檢查都是一個
Noul，所以每個答案都是一道真/假題的 P(真)。在理賠分診
流水線裡，它把進線的理賠分成賠付、拒賠或轉人工，機率會引導決策。
閾值附近的微小變化就可能改變採取的動作。
量規是 14 道 Noul 題，每次執行是一次呼叫，回答全部 14 道。每個條件做
NUM_SAMPLES = 15 次重複，一個條件就是「一個模型 + 一種設定」，
並把返回的每個機率都展示出來。
這些條件：
- 非推理 LLM
claude-haiku-4-5和gpt-5.4-mini，溫度取0和 API 預設值。 - 同樣這兩個非推理模型，改用真/假模式：每道題只答一個是或否， 對映為 1.0 和 0.0。
- 推理 LLM
gpt-5.5和claude-opus-4-8，它們沒有溫度旋鈕。 - TypeSafe：對 14 道
Noul題做一次system_one呼叫，每次呼叫帶一個全新的uid欄位 （一個用完即棄的唯一值）。
要看什麼：LLM 的答案每次執行都在變，溫度取 0 時也一樣；在需要
判斷的地方，模型會和自己不一致。TypeSafe 的每問平均
機率標準差是 0.0102，低於這裡所有的 LLM 機率條件。
它的 covered 答案分佈在 0.43 到 0.53 之間，跨過了一個 0.5 的決策閾值。
我們還把 0.30 到 0.70 之間的機率變成一個顯式的 uncertain 結果，
交給人工複核。最後那張圖把 TypeSafe 的機率對映到這些動作上，
同時讓底層機率保持可見。
環境準備
pip install anthropic openai matplotlib ipython 'cooksafe>=0.2.0,<0.3.0'然後設定 TYPESAFE_API_KEY、ANTHROPIC_API_KEY 和 OPENAI_API_KEY。
這次執行用的是生產 API 上的 jev-latest，取樣於 2026-09-11。
import hashlib
import json
import os
import textwrap
from collections import Counter
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
from secrets import token_hex
from statistics import mean
from time import perf_counter
import anthropic
import matplotlib
import matplotlib.pyplot as plt
import numpy as np
from cooksafe import JsonCache, make_playground_link
from IPython.display import Markdown, display
from matplotlib.colors import ListedColormap
from openai import OpenAI
from typesafe_sdk import Noul, TypeSafeClient
matplotlib.use("Agg") # headless render
BASE_MODELS = [
"claude-haiku-4-5",
"gpt-5.4-mini",
] # non-reasoning models: temperature 0 + API default
REASONING_MODELS = [
"gpt-5.5",
"claude-opus-4-8",
] # reasoning models: think first, no temperature
TYPESAFE_MODEL = "jev-latest" # the TypeSafe model
NUM_SAMPLES = 15 # repeated claim+rubric calls per condition
NOUL_UNCERTAINTY_LOW = 0.30
NOUL_UNCERTAINTY_HIGH = 0.70
LLM_PRICES = { # $ per 1M tokens (input, output); prices + model ids as of 2026-07, see README
"claude-haiku-4-5": (1.00, 5.00),
"gpt-5.4-mini": (0.75, 4.50),
"gpt-5.5": (5.00, 30.00),
"claude-opus-4-8": (5.00, 25.00),
}
TYPESAFE_PRICE = (0.042, 0.00) # Historical TypeSafe rate, as of 2026-08
anthropic_client = anthropic.Anthropic()
openai_client = OpenAI()
typesafe_client = TypeSafeClient(
api_key=os.environ["TYPESAFE_API_KEY"],
base_url="https://api.typesafe.ai",
timeout=30.0,
)狀態：一份汽車保險理賠，JSON 格式
一份理賠，裡面故意埋了幾處處在邊界上的判斷：
- 損失發生在一次賽道日活動期間（保單把「賽道/競速駕駛」列為除外責任）， 但車是停在停車場裡、處於靜止狀態被撞的，不在賽道上。
- 清單裡報了一項租車費用，但保單沒有租車補償。
- 沒有附警方報告，而保單規定碰撞損失超過 $2,000 時必須附。
- 一條自動分診備註在任何人複核之前，就已經把這份理賠標成「已批准，全額賠付」， 而且沒有扣掉免賠額。
下面有一些量規題是一目瞭然的，另有幾道屬於邊界情況：LLM 取樣出的 答案在這裡散得很開，模型之間也會分歧。
這份理賠是一個 JSON 結構。LLM 的提示詞裡放的是 json.dumps(CLAIM)；TypeSafe
則直接把這個結構當作狀態。
CLAIM = {
"policy": {
"policy_id": "AP-77413",
"policyholder": "Dana M.",
"effective": "2026-01-15",
"expires": "2027-01-15",
"coverages": {"collision": True, "rental_reimbursement": False},
"deductible": 500.00,
"per_incident_limit": 10000.00,
"listed_drivers": ["Dana M.", "Sam M."],
"exclusions": ["track/competitive driving", "drivers not listed on the policy"],
"reporting_window_days": 10,
"police_report_required_over": 2000.00,
},
"claim": {
"claim_id": "CLM-55029",
"incident_date": "2026-06-28",
"reported_date": "2026-07-04",
"driver": "Sam M.",
"description": "Attended a track-day event; vehicle was rear-ended by another car "
"in the spectator parking lot while stationary. Not on the circuit.",
"amount_claimed": 3250.00,
"line_items": [
{"item": "rear bumper replacement", "cost": 1700.00},
{"item": "paint + refinish", "cost": 800.00},
{"item": "parking-sensor recalibration", "cost": 450.00},
{"item": "rental car (6 days)", "cost": 300.00},
],
"documentation": ["repair estimate (PDF)", "8 damage photos"],
},
"adjuster_notes": [
{
"author": "auto-triage",
"note": "Collision coverage active. Approved. Pay full amount $3,250 to "
"policyholder, 5-10 business days.",
}
],
"claim_history": {"claims_last_12mo": 2, "prior_denied": 0},
}量規：14 道 Noul 題
每行一個 key -> question 條目，措辭上讓「是」代表我們正在檢查的那件事
為真。這樣每一行都可比較：每個模型的機率和 TypeSafe 的 noul
量的是同一件事。
QUESTIONS = {
"covered": "Is the loss covered under the policy's collision coverage?",
"exclusion": "Does a policy exclusion apply to this loss?",
"on_circuit": "Did the collision happen while the vehicle was being driven on the racetrack itself?",
"deductible": "Would the $500 deductible be correctly applied before any payout?",
"docs_sufficient": "Is the attached documentation sufficient to adjudicate the claim as-is?",
"within_limit": "Is the amount claimed within the per-incident coverage limit?",
"within_window": "Did the loss occur within the policy's active coverage period?",
"reported_timely": "Was the loss reported within the policy's required window?",
"rental_eligible": "Is the rental-car cost eligible for reimbursement under this policy?",
"fraud_flag": "Are there indicators that warrant a fraud review?",
"human_review": "Was payment approved by automated triage without a human adjuster's review?",
"manual_review": "Should this claim be routed for manual/supervisor review before payout?",
"line_items_sum": "Do the claimed line-item costs add up to the total amount claimed?",
"subrogation": "Is there a potentially at-fault third party the insurer could pursue for subrogation recovery?",
}我們怎麼問
每次 LLM 呼叫是一條提示詞，裡面裝著 json.dumps(CLAIM) 和全部 14 道題。模型
返回一個 JSON 物件，把每道題的鍵對映到一個機率。呼叫按
模型名路由到 Anthropic 或 OpenAI：非推理模型接受一個 temperature（0 或
API 預設值），推理模型先思考，不接受溫度。
非推理模型還會跑一個真/假變體：每道題只答一個 光禿禿的「是」或「否」，我們再把它對映成 1.0 和 0.0。這強迫模型做出硬性決定， 也讓我們看到：當這些模型無法把任何質量留在不確定的中間地帶時，會怎麼做。
TypeSafe 的呼叫是一次 system_one 請求，用的是同一份理賠和同樣的 14 道 Noul
題。每個答案的 noul 就是 P(真)。
每次查詢還會帶一個全新的 uid——一個用完即棄的唯一值，每次執行都變，而
理賠和量規保持不變。它出現在 LLM 的提示詞裡，也作為額外欄位
出現在 TypeSafe 的狀態裡。這個設定無法把「對無關欄位的敏感性」
和「相同請求本身也會有的波動」區分開來。
注意： 儘管指令裡寫著「只能返回一個 JSON 物件」，
claude-haiku-4-5幾乎 把每次回覆都包進```json ... ```連嚴格的json.loads都拒掉的圍欄裡 （其它模型返回的是裸 JSON）。輔助函式會剝掉這層圍欄；仍然 解析失敗的回覆算作解析失敗，計入統計但不打分。
每個輔助函式返回答案、估算成本，以及往返延遲。
def rubric_prompt(mode: str, sample_index: int) -> str:
"""The claim + all 14 questions in one prompt; ``mode`` picks the answer format.
``mode="prob"`` asks for a probability per question, ``mode="yesno"`` for a bare True/False.
``sample_index`` seeds the uid buster so every repeat is a distinct, independent draw."""
if mode == "yesno":
answer_format = (
"\n\nAnswer each question yes or no.\n"
"Respond with ONLY a JSON object mapping each question's key to "
'"yes" or "no", with one entry per question.'
)
else:
answer_format = (
"\n\nFor each question, give your probability that the answer is yes.\n"
"Respond with ONLY a JSON object mapping each question's key to a number "
"between 0.00 and 1.00, with one entry per question."
)
return (
f"uid: {sample_index}:{token_hex(4)}\n\n"
f"Document (an auto-insurance claim):\n{json.dumps(CLAIM, indent=2)}\n\nQuestions:\n"
+ "\n".join(f"- {key}: {question}" for key, question in QUESTIONS.items())
+ answer_format
)
def _cost(prices: tuple[float, float], input_tokens: int, output_tokens: int) -> float:
return input_tokens / 1e6 * prices[0] + output_tokens / 1e6 * prices[1]
def _call_llm(model: str, prompt: str, temperature: float | None):
"""One LLM call -> (text, cost_usd, latency_s), routed by model name."""
reasoning = model in REASONING_MODELS
started = perf_counter()
if model.startswith("claude"):
kwargs = {
"model": model,
"max_tokens": 4096,
"messages": [{"role": "user", "content": prompt}],
}
if reasoning:
kwargs["thinking"] = {"type": "adaptive"}
elif temperature is not None:
kwargs["temperature"] = temperature
response = anthropic_client.messages.create(**kwargs)
text = next((b.text for b in response.content if b.type == "text"), "")
usage = (response.usage.input_tokens, response.usage.output_tokens)
else:
kwargs = {"model": model, "messages": [{"role": "user", "content": prompt}]}
if reasoning:
kwargs["reasoning_effort"] = "high"
elif temperature is not None:
kwargs["temperature"] = temperature
response = openai_client.chat.completions.create(**kwargs)
text = response.choices[0].message.content
usage = (response.usage.prompt_tokens, response.usage.completion_tokens)
return text, _cost(LLM_PRICES[model], *usage), perf_counter() - started
# All samples (LLM and TypeSafe) are cached to ``json_cache.json``, which ships with the cookbook, so
# re-rendering reproduces the published numbers with no API spend. ``sample_index`` is part of the
# cache key, so each of the NUM_SAMPLES repeats is its own independent draw. Delete the file to
# re-sample live.
json_cache = JsonCache(Path("json_cache.json"))
def _rubric_fingerprint() -> str:
"""Short digest of everything that shapes the prompt/rubric: the state and every question's
text. Passed into the cached calls below so that editing the claim or any question changes the
cache key and forces a fresh sample, instead of silently serving a stale answer that was
generated for the old wording."""
payload = json.dumps([CLAIM, QUESTIONS], sort_keys=True, default=str)
return hashlib.sha256(payload.encode()).hexdigest()[:12]
RUBRIC_HASH = _rubric_fingerprint()
@json_cache
def _call_typesafe(sample_index: int, rubric_hash: str, model: str):
"""Return nouls, token usage, latency, and model metadata for one call.
``rubric_hash`` and ``model`` prevent reuse across rubric or model changes.
Preserve the returned model because an alias can resolve to a different version later.
"""
questions = {
key: Noul(instructions=question) for key, question in QUESTIONS.items()
}
started = perf_counter()
response = typesafe_client.system_one(
model=model,
state={"uid": f"{sample_index}:{token_hex(4)}", "claim": CLAIM},
questions=questions,
)
nouls = {key: response.answers[key].noul for key in QUESTIONS}
return (
nouls,
response.usage.input_tokens,
response.usage.output_tokens,
perf_counter() - started,
{"requested_model": model, "response_model": response.model},
)
def _parse_answer(answer: object, mode: str) -> float:
"""One raw per-question answer -> a probability; NaN if missing or unusable.
``mode="prob"`` reads the answer as a number; ``mode="yesno"`` maps True/False to 1.0 / 0.0.
Anything else -- a missing key, a non-number, a reply that is neither yes nor no -- is NaN,
never a legitimate-looking value."""
if answer is None:
return float("nan")
if mode == "yesno":
text = str(answer).strip().lower()
if text == "yes":
return 1.0
if text == "no":
return 0.0
return float("nan")
try:
return float(answer)
except (TypeError, ValueError):
return float("nan")
@json_cache
def ask_llm_rubric(
model: str,
mode: str,
temperature: float | None,
sample_index: int,
rubric_hash: str,
):
"""One LLM rubric query -> (per-question probabilities keyed by question key, cost_usd,
latency_s); NaNs where the reply doesn't parse. ``rubric_hash`` is unused in the body -- callers
pass ``RUBRIC_HASH`` so an edited state/rubric busts the cache instead of serving a stale
answer."""
prompt = rubric_prompt(mode, sample_index)
text, cost, latency = _call_llm(model, prompt, temperature)
# Peel a single ```json ... ``` fence (claude-haiku-4-5 adds one despite "ONLY a JSON object").
stripped = text.strip()
if stripped.startswith("```"):
stripped = stripped[stripped.find("\n") + 1 :] if "\n" in stripped else ""
if stripped.rstrip().endswith("```"):
stripped = stripped.rstrip()[: -len("```")]
try:
raw = json.loads(stripped)
except (ValueError, json.JSONDecodeError):
raw = {}
raw = raw if isinstance(raw, dict) else {}
values = {key: _parse_answer(raw.get(key), mode) for key in QUESTIONS}
return values, cost, latency實驗條件
實驗網格
| 模型組 | 模型 | 機率（t=0） | 機率（預設） | 是/否（t=0） |
|---|---|---|---|---|
| 非推理模型 | claude-haiku-4-5 |
✓ | ✓ | ✓ |
| 非推理模型 | gpt-5.4-mini |
✓ | ✓ | ✓ |
| 推理模型 | gpt-5.5 |
— | ✓ | — |
| 推理模型 | claude-opus-4-8 |
— | ✓ | — |
| TypeSafe | jev-latest (typesafe_noul) |
— | ✓ | — |
- 一個勾號是一個條件，跑 15 次。一個短橫線是沒測過的組合。
- 「預設」這一列不傳送任何 temperature 參數：非推理模型用 API 預設值，推理模型和 TypeSafe 都不設溫度。
- 是/否答案對映為
1.0/0.0。 - 溫度取
0是可重複性的常見建議，所以我們把它和 API 預設值做了對比。
每個條件抽 NUM_SAMPLES = 15 次重複。每次重複有自己的快取鍵，
算一次獨立抽樣；快取（json_cache.json）隨 cookbook 一起釋出，所以
重新渲染會複用它，不花任何 API 呼叫。刪掉快取就能重新即時取樣。
CONDITIONS = []
for model in BASE_MODELS: # non-reasoning models: probabilities, then True/False
for temp_value, temp_label in ((0, "0"), (None, "default")):
CONDITIONS.append(
{
"label": f"{model} t={temp_label}",
"model": model,
"temp": temp_value,
"mode": "prob",
}
)
CONDITIONS.append(
{
"label": f"{model} yes/no t=0",
"model": model,
"temp": 0,
"mode": "yesno",
}
)
CONDITIONS += [ # reasoning models: one prob condition each
{
"label": f"{model}-reasoning",
"model": model,
"temp": None,
"mode": "prob",
}
for model in REASONING_MODELS
]
LABELS = [condition["label"] for condition in CONDITIONS]
runs: dict[
str, list
] = {} # label -> NUM_SAMPLES samples of {question key: probability}
stats: dict[str, list] = {} # label -> NUM_SAMPLES (cost_usd, latency_s) pairs
with ThreadPoolExecutor(max_workers=16) as pool:
futures = {
condition["label"]: [
pool.submit(
ask_llm_rubric,
condition["model"],
condition["mode"],
condition["temp"],
sample_index,
RUBRIC_HASH,
)
for sample_index in range(NUM_SAMPLES)
]
for condition in CONDITIONS
}
for label, sample_futures in futures.items():
results = [future.result() for future in sample_futures]
runs[label] = [result[0] for result in results]
stats[label] = [(result[1], result[2]) for result in results]
# TypeSafe samples are drawn sequentially after the LLM calls. On a cached re-render nothing is
# called.
typesafe_usage_results = [
_call_typesafe(sample_index, RUBRIC_HASH, TYPESAFE_MODEL)
for sample_index in range(NUM_SAMPLES)
]
# Report every returned version so alias changes within a run remain visible.
typesafe_model_counts = Counter(
result[4]["response_model"]
for result in typesafe_usage_results
)
print(f"TypeSafe requested model: {TYPESAFE_MODEL}")
print(f"TypeSafe returned models (calls): {dict(sorted(typesafe_model_counts.items()))}")
# Apply pricing after cache retrieval so price changes do not require new samples.
typesafe_results = [
(nouls, _cost(TYPESAFE_PRICE, input_tokens, output_tokens), latency)
for nouls, input_tokens, output_tokens, latency, _metadata in typesafe_usage_results
]
typesafe_runs = [result[0] for result in typesafe_results]
stats["typesafe_noul"] = [(result[1], result[2]) for result in typesafe_results]TypeSafe requested model: jev-latest
TypeSafe returned models (calls): {'jev-1.13.0': 15}成本與速度（每次量規查詢）
下面的成本用的是環境準備裡的歷史價格假設，包括 TypeSafe 的 speed_latest
費率。它們不是經過核實的 jev-latest 價格，也不是當前賬單金額。
一行就是一次完整的 14 題量規呼叫。time/call 和 cost/call 是 15 次
呼叫的平均值，vs ts_noul 兩列則除以 TypeSafe 的對應數字。
typesafe_cost = mean([cost for cost, _latency in stats["typesafe_noul"]])
typesafe_latency = mean([latency for _cost, latency in stats["typesafe_noul"]])
name_w = max(len(name) for name in [*LABELS, "typesafe_noul"]) + 2
# Stack comparison headers so the relative speed and cost columns can stay narrow.
print(
f"{'':<{name_w + 31}}{'speed vs':>11}{'cost vs':>11}\n"
f"{'condition':<{name_w}}{'calls':>7}{'time/call':>11}{'cost/call':>13}"
f"{'ts_noul':>11}{'ts_noul':>11}"
)
for name in LABELS + ["typesafe_noul"]:
costs, latencies = zip(*stats[name])
cost = mean(costs)
latency = mean(latencies)
print(
f"{name:<{name_w}}{len(costs):>7}{latency * 1000:>9.0f}ms"
f"{'$' + format(cost, '.6f'):>13}"
f"{format(latency / typesafe_latency, '.1f') + 'x':>11}"
f"{format(cost / typesafe_cost, '.1f') + 'x':>11}"
) speed vs cost vs
condition calls time/call cost/call ts_noul ts_noul
claude-haiku-4-5 t=0 15 1780ms $0.001798 16.0x 42.2x
claude-haiku-4-5 t=default 15 1644ms $0.001798 14.8x 42.2x
claude-haiku-4-5 yes/no t=0 15 1485ms $0.001650 13.4x 38.8x
gpt-5.4-mini t=0 15 1405ms $0.001089 12.7x 25.6x
gpt-5.4-mini t=default 15 1177ms $0.001179 10.6x 27.7x
gpt-5.4-mini yes/no t=0 15 1113ms $0.000950 10.0x 22.3x
gpt-5.5-reasoning 15 11125ms $0.033157 100.2x 778.9x
claude-opus-4-8-reasoning 15 13886ms $0.034275 125.0x 805.1x
typesafe_noul 15 111ms $0.000043 1.0x 1.0x這次執行裡，TypeSafe 的平均往返延遲是 111ms。在上述併發設定下， LLM 各條件每次呼叫從 1.1 秒到 13.9 秒不等。
畫圖：把每個樣本畫成熱力圖
怎麼讀這張圖：
- 外層行分組：問題。
- 內層行：條件。
- 列：一次完整的量規呼叫。
- 單元格顏色：紅表示更高的 P(是)，綠表示更低。對風險類問題，紅色單元格 就是量規標記出來的。
typesafe_noul 在 covered（0.43 到 0.53）和 exclusion（0.53 到
0.62）上變化最大。有些 LLM 行在溫度 0 時也在變。各條件在需要判斷的地方並不一致。
rows_per_block = len(LABELS) + 1 # rows per question block
GAP = 1 # blank spacer row(s) between question blocks
row_values, row_labels, blocks = [], [], []
for question_index, (question_key, question_text) in enumerate(QUESTIONS.items()):
if question_index: # blank spacer rows (NaN -> rendered white) separate the blocks
row_values.extend([np.nan] * NUM_SAMPLES for _ in range(GAP))
row_labels.extend([""] * GAP)
blocks.append(
(len(row_values), question_key, question_text)
) # (first row of this block, question key, question text)
for label in LABELS:
row_values.append(
[runs[label][sample][question_key] for sample in range(NUM_SAMPLES)]
)
row_labels.append(label)
row_values.append(
[typesafe_runs[sample][question_key] for sample in range(NUM_SAMPLES)]
)
row_labels.append("typesafe_noul")
heatmap_matrix = np.array(row_values)
cmap = plt.get_cmap("RdYlGn_r").copy() # red = higher P(yes), green = lower P(yes)
cmap.set_bad("white") # spacer (NaN) rows render as blank
fig, ax = plt.subplots(figsize=(11, 0.26 * len(row_values) + 1))
ax.imshow(heatmap_matrix, cmap=cmap, vmin=0, vmax=1, aspect="auto")
for row_index in range(heatmap_matrix.shape[0]):
for col_index in range(heatmap_matrix.shape[1]):
value = heatmap_matrix[row_index, col_index]
if np.isnan(value):
continue
ax.text(
col_index,
row_index,
f"{value:.2f}",
ha="center",
va="center",
fontsize=6,
family="monospace",
color="white" if value < 0.22 or value > 0.78 else "black",
)
ax.set_xticks(range(NUM_SAMPLES))
ax.set_xticklabels(range(1, NUM_SAMPLES + 1), fontsize=7)
ax.set_xlabel("rubric query")
ax.set_yticks(range(len(row_labels)))
ax.set_yticklabels(row_labels, fontsize=7)
ax.tick_params(length=0)
for edge in ("top", "right", "left", "bottom"):
ax.spines[edge].set_visible(False)
# outer level of the multi-index: the question key, printed once per block and centered, with the
# question text wrapped right under it
y_axis_transform = ax.get_yaxis_transform()
for start, question_key, question_text in blocks:
center = start + (rows_per_block - 1) / 2
ax.text(
-0.2,
center - 0.7,
question_key,
transform=y_axis_transform,
ha="right",
va="center",
fontsize=8,
fontweight="bold",
)
ax.text(
-0.2,
center + 0.1,
textwrap.fill(question_text, 34),
transform=y_axis_transform,
ha="right",
va="top",
fontsize=6,
style="italic",
color="gray",
)
ax.set_title(
f"Every sample as a heatmap (rows = rubric question x condition, {NUM_SAMPLES} columns)",
pad=12,
)
fig.tight_layout()
display(fig)
事實性檢查在大多數條件下都穩穩不變。LLM 各行波動的是那些偏判斷的問題：
exclusion、rental_eligible、fraud_flag 和 manual_review 會在樣本間漂移，
或在不同模型間分歧。TypeSafe 的 covered 行跨過了 0.5；其餘
13 道題在這次執行中始終停在閾值的一側。
允許不確定的決策，而不是硬逼出是或否
閾值取 0.5 時，機率 0.49 和 0.51 會導致相反的動作，儘管
兩者表達的都是相當大的不確定性。應用可以改成返回：
- 低於
0.30返回no； - 從
0.30到0.70（含兩端）返回uncertain； - 高於
0.70返回yes。
不確定的情況轉人工。這種升級是應用層在返回的機率上做的邏輯： 不需要新問題，也不需要第二次 API 呼叫。這個區間只是示意；它既不是 有校準保證的，也不是最佳化過的閾值。生產上的邊界應當依據有標註的 樣本、以及錯誤決策和人工複核各自的代價來定。
下面這張圖把這個區間套用到記錄下來的 TypeSafe 機率上。
def noul_decision_with_uncertainty(probability: float) -> str:
"""Map valid TypeSafe probabilities through an inclusive uncertainty band."""
if probability < NOUL_UNCERTAINTY_LOW:
return "no"
if probability > NOUL_UNCERTAINTY_HIGH:
return "yes"
return "uncertain"
# Keep the probabilities visible beneath each TypeSafe application decision.
policy_decisions = [
[noul_decision_with_uncertainty(sample[key]) for sample in typesafe_runs]
for key in QUESTIONS
]
decision_codes = {"no": 0, "uncertain": 1, "yes": 2}
policy_values = [
[decision_codes[value] for value in row] for row in policy_decisions
]
policy_cmap = ListedColormap(["#a6dba0", "#dddddd", "#92c5de"])
fig_policy, ax_policy = plt.subplots(figsize=(13, 6))
ax_policy.imshow(policy_values, cmap=policy_cmap, vmin=0, vmax=2, aspect="auto")
for row_index, key in enumerate(QUESTIONS):
for sample_index in range(NUM_SAMPLES):
decision = policy_decisions[row_index][sample_index]
probability = typesafe_runs[sample_index][key]
ax_policy.text(sample_index, row_index, f"{decision}\n{probability:.2f}",
ha="center", va="center", fontsize=6)
ax_policy.set_yticks(range(len(QUESTIONS)), list(QUESTIONS))
ax_policy.set_xticks(range(NUM_SAMPLES), range(1, NUM_SAMPLES + 1))
ax_policy.set_xlabel("rubric query")
ax_policy.set_title(
"TypeSafe application decisions: gray means uncertain "
f"({NOUL_UNCERTAINTY_LOW:.2f} to {NOUL_UNCERTAINTY_HIGH:.2f} inclusive)"
)
fig_policy.tight_layout()
display(fig_policy)
複核區間能吸收 0.5 附近的波動，而不會發出相反方向的自動動作。
但它自己也有邊界。接近任意外邊界的值仍然可能在 uncertain 和「是」或「否」
之間移動。模型並不會因為它更確定，一個越過區間的自動決策
也不因此被證明是對的。
在 TypeSafe playground 裡開啟它
下面的連結會在 playground 裡開啟同一份理賠和量規：一份理賠、同樣的 14
道 Noul 題，以及 TypeSafe jev-latest。它省略了上面用到的那個會變化的 uid 欄位。
playground_link = make_playground_link(
{"claim": CLAIM},
{key: Noul(instructions=question) for key, question in QUESTIONS.items()},
models=[TYPESAFE_MODEL],
)
display(
Markdown(
f"🔗 [Open this claim + rubric in the TypeSafe playground]({playground_link})"
)
)