用置信度做分类
用置信度做分类
每份 SEC 年报用一个 Choice 归入 75 个行业组之一，再读答案自身的置信度，决定是报这个组还是它上面更宽的部门。
每一家向 SEC 提交年报的公司，都会在年报里描述自己的业务。我们把这些描述归入标准产业分类（Standard Industrial Classification）：75 个行业组，每个文件问一个 Choice 问题。
大多数文件的分类很轻松。一家地区银行就是一家地区银行。有些则不然：比如一家刚卖掉自己两个业务板块之一的公司，或者一家描述的是一门打算进入、而非正在经营的业务的初创公司。不论如何，模型都得挑出一个组，而难题的答案和简单题的答案看起来没什么两样。要在难题和简单题之间分个高下，通常就得付出成本：再上一个模型、额外的调用、人工复核。
一个 Choice 本身就能告诉你。除了胜出的那个选项，它还会返回 confidence：当几乎全部概率都落在一个选项上时它高，概率分散在几个选项上时它低。就这一个数字，把你能信的答案和不能信的答案分开了。
拿一个不可信的答案怎么办，取决于你的标签体系。SIC 的标签构成一个层级：行业组向上归并成更宽的部门。这让同一次响应几乎免费。当模型对是哪个组没把握时，就报它所属的部门。宽标签可以从窄标签推出来，所以不用再调一次。
在 60 份文件上，用 0.9 的置信度切一刀，刚好把它们分成两半。有把握的那一半，90% 是对的；另一半只有 40%。把它们往上报一个层级，那 40% 就变成了 70%。最后我们得到一个 classify() 函数，每个文件一次请求，返回一个标签以及它有多具体。
flowchart LR
doc["Item 1 'Business'<br/>from one 10-K"]
subgraph request["one request"]
q["Choice<br/>75 industry groups"]
end
sure{"confidence<br/>≥ 0.9?"}
grp["report the industry group<br/><i>e.g. 28</i>"]
div["report its division<br/><i>e.g. manufacturing</i>"]
doc --> request --> sure
%% both branches leave the test, so they share a rank and stack on their own
sure -- "yes" --> grp
sure -- "no" --> div准备工作
pip install ipython matplotlib 'cooksafe>=0.2.0,<0.3.0'然后设置 TYPESAFE_API_KEY。每次 API 调用都缓存到 json_cache.json，该文件随 cookbook 一起提供，所以重新渲染会回放已发布的数字，不会调用 API。删掉该文件就能全部真实重跑。
下面的数字来自 2026-08-12 的 jev-1.12。
import json
from collections import defaultdict
from pathlib import Path
import matplotlib
import matplotlib.pyplot as plt
from cooksafe import JsonCache, make_playground_link
from IPython.display import Markdown, display
from typesafe_sdk import Choice, TypeSafeClient
matplotlib.use("Agg") # headless render
import os # noqa: E402
TYPESAFE_MODEL = "jev-1.12"
CONFIDENT = 0.9 # above this the group is reported; below it, the division
client = TypeSafeClient(
api_key=os.environ.get(
"TYPESAFE_API_KEY", "cache-only"
), # keyless kernels replay the cache
base_url=os.environ.get("TYPESAFE_ENDPOINT"),
timeout=120.0,
)
json_cache = JsonCache(Path("json_cache.json"))搭出分类法的两个层级
sic_codes.tsv 是 SEC 发布、供申报人挑选自用代码的行业清单，抓取于 2026-08-10：444 个四位代码，每个都配一个行业名称。这些数字本身就是一个层级。前两位是大类（这里是 75 个，从 01 农业生产到 99 无法分类），而大类的固定区间构成了 10 个部门，这是 SIC 最粗的一层划分。
这两个层级都从这一个文件里算出来，不涉及任何模型：按前两位给代码分组，再把这些两位数字映射到一个部门。
DIVISIONS = [
(1, 9, "agriculture, forestry and fishing"),
(10, 14, "mining"),
(15, 17, "construction"),
(20, 39, "manufacturing"),
(40, 49, "transportation, communications and utilities"),
(50, 51, "wholesale trade"),
(52, 59, "retail trade"),
(60, 67, "finance, insurance and real estate"),
(70, 89, "services"),
(91, 99, "public administration"),
]
INDUSTRIES: dict[str, str] = {}
for line in Path("sic_codes.tsv").read_text().splitlines()[1:]:
code, _office, title = line.split("\t")
INDUSTRIES[code] = title.lower()
GROUPS: dict[str, list[str]] = defaultdict(list)
for code in sorted(INDUSTRIES):
GROUPS[code[:2]].append(code)
def division(group: str) -> str:
number = int(group)
return next(name for low, high, name in DIVISIONS if low <= number <= high)
print(
f"{len(INDUSTRIES)} industries -> {len(GROUPS)} major groups -> {len(DIVISIONS)} divisions"
)
print(
f" group 35 = {division('35')} / {', '.join(INDUSTRIES[c] for c in GROUPS['35'][:3])} ..."
)444 industries -> 75 major groups -> 10 divisions
group 35 = manufacturing / engines & turbines, farm machinery & equipment, lawn & garden tractors & home lawn & gardens equip ...Choice 问题需要给每个选项配上描述，而组本身的名字并不总是存在：75 个组里有 42 个在 SEC 的清单里带一个总括性的名称，其余什么都没有。所以每个组都用它包含的行业来描述 —— 读文件的人本来也是拿这些来对照的。
MAX_NAMED = (
8 # industries listed per group; enough to characterise it without a wall of text
)
def describe(group: str) -> str:
umbrella = INDUSTRIES.get(f"{group}00")
inside = [INDUSTRIES[c] for c in GROUPS[group] if c != f"{group}00"][:MAX_NAMED]
listed = "; ".join(inside)
return (
f"{umbrella} — includes: {listed}"
if umbrella and listed
else (umbrella or listed)
)
print(f"group 20: {describe('20')[:150]}")
print(f"\ngroup 65: {describe('65')[:150]}")group 20: food and kindred products — includes: meat packing plants; sausages & other prepared meat products; poultry slaughtering and processing; dairy product
group 65: real estate — includes: real estate operators (no developers) & lessors; operators of nonresidential buildings; operators of apartment buildings; less这些文件
filings.jsonl 装着 60 份年报（10-K），每份都裁剪到 Item 1 “Business” —— 公司描述自己做什么的那一节，也是行业代码唯一涉及的部分。它们跨越 1993–2024 年，篇幅从 700 到 2200 词不等。每份都带着申报人自己选的 SIC 代码，以及用来在 EDGAR 上查到它的 accession number。
在看任何准确率数字之前，得先说清楚这个标签是怎么来的。它是自报的：准备文件的人当初挑一次就定下了，而一旦公司卖掉了代码所指的业务却还留着这个代码，它就过时了。这 60 份被筛选到「自身文本支撑其所带代码」的文件，所以这里的数字衡量的是这套做法，而不是 EDGAR 元数据的现状。
FILINGS = [json.loads(line) for line in Path("filings.jsonl").read_text().splitlines()]
example = FILINGS[7]
print(
f"{len(FILINGS)} filings, {sum(f['words'] for f in FILINGS) // len(FILINGS)} words on average"
)
print(f"\n{example['id']} (filed {example['year']}, accession {example['accession']}):")
print(f" {example['text'][:230]}...")
print(f" filer's code: {example['sic']} {INDUSTRIES[example['sic']]}")60 filings, 1438 words on average
1389870_2008 (filed 2008, accession 0001079974-09-000155):
Item 1. DESCRIPTION OF BUSINESS. NARRATIVE DESCRIPTION OF THE BUSINESS Across America Financial Services, Inc. is a corporation which was formed under the laws of the State of Colorado on December 1, 2005. Until March 23, 2007, we...
filer's code: 6163 loan brokers问一个 Choice 问题，读它的置信度
一个 Choice 问题，选项就是那 75 个组。整棵分类法装得下一次请求：Choice 在约 240 个选项以内都能稳定工作，75 完全在范围内。
答案里带着 choice，也就是胜出的组；probabilities，75 个选项各自的权重；以及 confidence，它说明这份分布有多集中。这套做法读的是 confidence，而不是胜出者自己的概率。一个 0.45 的胜出者配上 0.44 的第二名，和一个 0.45 的胜出者配上稀稀拉拉散开的其余权重，是两种不同的情形，而 confidence 正是把它们区分开的东西。
QUESTION = (
"Which broad industry does this company operate in? Judge the company's own operations "
"as this filing describes them."
)
def questions() -> dict:
return {
"group": Choice(
instructions=QUESTION,
criteria={group: describe(group) for group in sorted(GROUPS)},
)
}
@json_cache
def ask(filing_id: str, text: str) -> dict:
response = client.system_one(
state=text, questions=questions(), model=TYPESAFE_MODEL
)
answer = response.answers["group"]
return {
"group": answer.choice,
"confidence": answer.confidence,
"probabilities": dict(answer.probabilities),
}有把握时报组，没把握时报部门
下面这四行就是整套做法。置信度达到 0.9 及以上，答案就作为行业组报出；低于这个值，同一个答案就作为该组所属的部门报出。
每份文件依然会返回一个可用的标签。模型没能有把握地分类的那一份，不会是被丢掉或转走，而是往上报一个层级。如果某个部门粗到你的应用没法据此行动，这个分支就是你把它交给人的地方。
def classify(filing: dict) -> dict:
answer = ask(filing["id"], filing["text"])
sure = answer["confidence"] >= CONFIDENT
return {
"level": "group" if sure else "division",
"label": answer["group"] if sure else division(answer["group"]),
"confidence": answer["confidence"],
"group": answer["group"],
}
def show(filing: dict) -> None:
result = classify(filing)
named = describe(result["group"]).split(" — ")[0][:46]
print(
f" {filing['id']:>13} conf {result['confidence']:.2f} -> {result['level']:<8} "
f"{result['label']:<14} (group {result['group']}: {named})"
)
print("three filings the model was sure about:")
for f in sorted(FILINGS, key=lambda f: -ask(f["id"], f["text"])["confidence"])[:3]:
show(f)
print("\nthree it was not:")
for f in sorted(FILINGS, key=lambda f: ask(f["id"], f["text"])["confidence"])[:3]:
show(f)three filings the model was sure about:
310158_1996 conf 1.00 -> group 28 (group 28: chemicals & allied products)
33416_1998 conf 1.00 -> group 63 (group 63: life insurance; accident & health insurance; h)
352541_1996 conf 1.00 -> group 49 (group 49: electric, gas & sanitary services)
three it was not:
1372167_2013 conf 0.22 -> division manufacturing (group 38: search, detection, navagation, guidance, aeron)
1398633_2009 conf 0.23 -> division wholesale trade (group 50: wholesale-durable goods)
46653_1999 conf 0.29 -> division services (group 87: services-engineering, accounting, research, ma)这些置信度和每份文件分类起来有多难是吻合的。三个 1.00 的分别是制药商、寿险公司和公用事业公司；三家在纸面上都是控股公司，但每家都有一个主导业务，文件里明明白白地点了出来。垫底的三个难在哪，从文本里能读出来。两家是处于开发阶段的公司，描述的是它们打算开办的业务（Nevaeh “intends to operate as a software developer”，Barricode “organized to enter into the computer security software industry”），第三家则有两个业务板块，在申报前几周卖掉了其中一个。这三份最后以部门而非组的形式返回。
classify() 就是整套做法。把 ask() 指向你自己的文档，再为你的分类法重写 describe()，其余部分照搬即可。
报得更宽能换来什么
全部 60 份文件，都以每个申报人自己选的代码为基准打分，两种策略各跑一遍：每次都报一个组，或者只要置信度落到 0.9 以下就报部门。
def correct(filing: dict, result: dict) -> bool:
gold_group = filing["sic"][:2]
if result["level"] == "group":
return result["label"] == gold_group
return result["label"] == division(gold_group)
results = [(f, classify(f)) for f in FILINGS]
sure = [(f, r) for f, r in results if r["level"] == "group"]
unsure = [(f, r) for f, r in results if r["level"] == "division"]
forced = sum(r["group"] == f["sic"][:2] for f, r in results)
broadened = sum(correct(f, r) for f, r in results)
print(f"forced to name a group every time {forced}/{len(results)} right")
print(
f" of those, the {len(sure)} it was sure about "
f"{sum(r['group'] == f['sic'][:2] for f, r in sure)}/{len(sure)} right"
)
print(
f" and the {len(unsure)} it was not "
f"{sum(r['group'] == f['sic'][:2] for f, r in unsure)}/{len(unsure)} right"
)
print(
f"\nletting it answer coarsely when unsure {broadened}/{len(results)} useful answers"
)forced to name a group every time 39/60 right
of those, the 30 it was sure about 27/30 right
and the 30 it was not 12/30 right
letting it answer coarsely when unsure 48/60 useful answers模型有把握的地方，它报出的组十次里有九次是对的。没把握的地方，报组是错多于对，只有 40%。把这些同样的答案改报成部门，就升到了 70%。
下图把两种策略并排放在一起，并按模型是否有把握做了拆分。
labels = ["sure\n(group reported)", "unsure\n(division reported)"]
forced_split = [
sum(r["group"] == f["sic"][:2] for f, r in sure) / len(sure),
sum(r["group"] == f["sic"][:2] for f, r in unsure) / len(unsure),
]
broad_split = [
sum(correct(f, r) for f, r in sure) / len(sure),
sum(correct(f, r) for f, r in unsure) / len(unsure),
]
fig, ax = plt.subplots(figsize=(7, 3.6))
x = range(len(labels))
ax.bar(
[i - 0.19 for i in x],
forced_split,
0.38,
label="always name a group",
color="#c8ccd4",
)
ax.bar(
[i + 0.19 for i in x],
broad_split,
0.38,
label="answer broadly when unsure",
color="#3b6ea5",
)
for i, (a, b) in enumerate(zip(forced_split, broad_split)):
ax.text(i - 0.19, a + 0.02, f"{a:.0%}", ha="center", fontsize=9)
ax.text(i + 0.19, b + 0.02, f"{b:.0%}", ha="center", fontsize=9)
ax.set_xticks(list(x))
ax.set_xticklabels(
[f"{lab}\nn={n}" for lab, n in zip(labels, [len(sure), len(unsure)])]
)
ax.set_ylabel("labels that are right")
ax.set_ylim(0, 1.12)
ax.set_title("Where the broader answer helps: the filings it was unsure about")
ax.legend(frameon=False, loc="upper right")
ax.spines[["top", "right"]].set_visible(False)
plt.tight_layout()
display(fig)
在 Playground 里打开
这个分享链接里装着一份文件和那个 75 选项的问题，不用写任何代码，就能看到它给出的分布和置信度。
playground_link = make_playground_link(
example["text"], questions(), models=[TYPESAFE_MODEL]
)
display(
Markdown(
f"🔗 [Open the filing + question in the TypeSafe playground]({playground_link})"
)
)