Helpers
Language detection
laya.detect_language is laya.lang.analyse.
Names, types, defaults and code stay in English; the rest is translated (entries not translated yet are shown in the original English).
analyse
analyse(state: Union[str, bytes, Mapping, list, None]) -> Dict[str, object]Full detection result for a state.
Returns script, script_profile, language (best effort, may be None),
is_english, non_latin_fraction and mixed_segment (the line or field that made a mostly
English state non-English, else None).
String values are what get read. When a state has several of them, one non-English value is enough: joining every value into one window let a long English note fill the 4000 characters, or outvote a short German message, and that message was then sent to the English checkpoint (#384). The segment scan still stops at 4000 characters, which is what keeps a huge field cheap; a value it did not reach is read on its own afterwards.
Parameters
stateUnion[str, bytes, Mapping, list, None]
detect_script
detect_script(text: str) -> strDominant script of text: 'latin', 'han', 'devanagari', ... or 'unknown' if there are no letters.
Parameters
textstr
is_english
is_english(state: Union[str, bytes, Mapping, list, None]) -> boolTrue when the English checkpoint can be expected to read this state.
Parameters
stateUnion[str, bytes, Mapping, list, None]
clean_email_body
clean_email_body(body: str, max_chars: int = 3000) -> strRemove quoted email history, signatures and disclaimers to keep input focused.
max_chars is the length the result is cut to, 3000 characters unless raised -- see
email_state, which takes the same budget and passes it through.
Parameters
bodystrmax_charsint=3000
email_state
email_state(
subject: str,
body: str,
sender: Optional[str] = None,
clean: bool = True,
max_chars: int = 3000,
extra,
) -> DictConstruct a clean state dictionary for email classification.
max_chars is the budget clean_email_body cuts the body to, and it is worth raising for a
long message: at the default the body stops after 3000 characters, so a request that arrives in
the last paragraphs never reaches the model -- including through predict_long, which scans a
state in windows precisely so it can read past one window's worth. Ignored when clean=False,
which passes the body through whole.
Any other keyword becomes a field of the state, so it is read by the model; a typo here is an
input mutation, not an error. The exception is a keyword whose value is exactly None: it is
dropped rather than added, so email_state(..., thread_id=None) gives the same state as
leaving the keyword out. The key is absent, not null -- and the distinction reaches the model,
because laya.common.serialize_state turns the state into JSON before it is tokenized, so a
retained None would put "thread_id": null in the prompt. sender is dropped on the
same call by a different rule, falsiness rather than None, so sender="" leaves no
from field either.
Parameters
subjectstrbodystrsenderOptional[str]=Nonecleanbool=Truemax_charsint=3000extra
Question presets
triage_questions
triage_questions() -> DictPreset questions for customer support ticket triage.
email_questions
email_questions(categories: Optional[Dict[str, str]] = None) -> DictPreset questions for inbound email triage and threat filtering.
Parameters
categoriesOptional[Dict[str, str]]=None
guard_questions
guard_questions() -> DictPreset questions for real-time LLM input guardrails.
moderation_questions
moderation_questions() -> DictPreset questions for content safety and moderation.
router_questions
router_questions() -> DictPreset questions for intelligent model routing.
Pre-run request sizing
What a request will fit before any forward pass. These are the helpers predict_long itself
reads, so a question can be measured without holding an Agent: build_head is the question half
of a sequence, state_room is what is left for the state after it, and build_sequence reports
what the head budget did to the options and to the state.
from transformers import AutoTokenizer
from laya.common import build_sequence, collapsed_options, state_room, window_budget
tok = AutoTokenizer.from_pretrained("convaiinnovations/laya", subfolder="tokenizer")
max_len, head_max_len = 512, 192 # from the checkpoint's rl_agent_config.json
question = {"type": "choice", "instructions": "What does the customer want?",
"criteria": {"refund": "money back", "cancel": "stop the service"}}
# The short-key internal shape the sizing helpers read; inline the mapping as
# questions-and-answers.md shows rather than reaching into Agent._to_internal.
q = {"t": question["type"], "ins": question["instructions"], "crit": question["criteria"]}
state = "I was charged twice for the same invoice."
state_room(tok, q, max_len, head_max_len) # state tokens this question leaves room for
seq, markers, stats, trunc = build_sequence(
tok, state, q, max_len, head_max_len, return_stats=True, return_truncation_stats=True)
stats # how many options the head budget kept distinct, and the cap it applied
trunc # state tokens, used, dropped, and whether the state was truncated
collapsed_options(["q"], [{"options": stats}]) # {} until an option loses its own span
window_budget(tok, [q], max_len, head_max_len) # (window, stride, room) a scan may use
The same numbers come back after a call in usage: state_tokens, state_tokens_dropped,
truncated, truncated_questions, and options when an option lost its span. That block is
the record of what a run did; the helpers above are how to find out before paying for one.
predict_long hands a state straight to a single pass when its tokens fit the room its
questions leave, and a scan window is capped at that room with a 50%-of-window stride – a cap
large enough to multiply the passes warns before the scan runs.
None of this picks a strategy: tournament rounds come back as tournament[qid]["rounds"]
after predict_tournament, and shortlist ranking quality stays with whatever embedder
supplies it.
state_room
state_room(tok, q: Dict, max_len: int = 512, head_max_len: int = 192) -> intHow many state tokens q leaves inside max_len, which is what build_sequence keeps.
The head is the question's own -- its instructions plus one [MASK]-prefixed span per option --
so a question with many options leaves less room for the state than one with two, and two
questions in the same request do not have to leave the same amount. Anything past the return
value is cut off (the start is kept, or the end for a conversation turn list).
Parameters
tokqDictmax_lenint=512head_max_lenint=192
window_budget
window_budget(
tok,
questions,
max_len: int = 512,
head_max_len: int = 192,
window: Optional[int] = None,
stride: Optional[int] = None,
)Window size and stride for scanning a state that is longer than one sequence.
predict_long decodes each token window back to text and scores it as an ordinary state, so a
window wider than the room the questions leave is re-truncated by build_sequence on the way
in: the tail of every window reaches no model, while the reported span says it did. The window
is therefore capped at state_room, and at the smallest room of questions, because the
windows are one list of states scored for every question in shared forward passes -- a window
sized for the roomiest question would be cut short for the tightest one, and the offsets
reported on its answers would mean something different per question. With no questions there is
nothing to fit, so the caller's window (or the checkpoint default) stands. That default is
max(64, max_len - head_max_len - 8): the state budget the config leaves, with a 64-token
floor under it, so a widened head_max_len stops shrinking it there.
questions are internal question dicts, as Agent._to_internal returns them.
An explicit window wider than the room is clamped to it with a RuntimeWarning, since a
scan at the requested size cannot read what it claims to. A stride past the effective window
is refused: the tokens between two windows would be read by neither, which is the failure
predict_long exists to prevent. The default stride keeps its 50% overlap of the effective
window, so a span near a boundary still lands whole inside some window; an explicit stride
equal to the window still reads every token, with no overlap to catch a span that straddles a
boundary.
Returns (window, stride, room).
Parameters
tokquestionsmax_lenint=512head_max_lenint=192windowOptional[int]=NonestrideOptional[int]=None
collapsed_options
collapsed_options(qids, items) -> Dict[str, Dict[str, Optional[int]]]The questions whose options no longer have a token span each, from per-item stats.
total is the number of options the question defines, not the number of markers that
reached the sequence: a report counted from the markers would say "43/58" about a request
where 28 options never made it into the input at all.
Parameters
qidsitems
Shortlisting
shortlist_choice
shortlist_choice(
state: Any,
criteria: Any,
embed_fn: Callable[[Sequence[str]], Any],
k: int = DEFAULT_SHORTLIST_K,
DEFAULT_SHORTLIST_K,
instructions: Optional[str] = None,
return_scores: bool = False,
) -> AnyReturn the top-k choice labels for state.
embed_fn maps a list of strings to an array of shape (len(texts), dim).
It is called once, with the query text first and then one string per option in
criteria order. Option strings match render_options for a choice question.
When k is at least the number of labels, every label is returned in its
original order and embed_fn is not called.
Ties keep the earlier label. Ranking is a signed cosine, not a similarity floor:
a label that scores 0 -- no signal at all, or a non-finite vector treated as one
-- does outrank an earlier label that scored negative, and k drops the
negative labels first.
With return_scores=True the return is the (labels, scores) pair, where
scores holds the signed cosine per kept label in rank order -- the same
values predict_shortlist reports in its shortlist metadata. scores
is None when nothing was dropped, exactly as in that metadata.
Parameters
stateAnycriteriaAnyembed_fnCallable[[Sequence[str]], Any]kint=DEFAULT_SHORTLIST_KDEFAULT_SHORTLIST_KinstructionsOptional[str]=Nonereturn_scoresbool=False
predict_shortlist
predict_shortlist(
agent: Any,
state: Any,
questions: Dict[str, Dict[str, Any]],
embed_fn: Callable[[Sequence[str]], Any],
k: int = DEFAULT_SHORTLIST_K,
DEFAULT_SHORTLIST_K,
predict_kwargs: Any,
) -> Dict[str, Any]Shortlist each choice question, then call predict or system_one once.
Non-choice questions are forwarded unchanged. A choice whose label count is
<= k is forwarded unchanged and does not call embed_fn. The caller's
questions dict is not mutated.
A narrowed choice that carried an option_order has it dropped, with one
RuntimeWarning naming the question: the order has one slot per option, so an
order that fitted the caller's label set no longer fits the kept labels and the
agent refuses the question. Remapping the order onto the kept labels would
silently move options between slots, so it is dropped instead. A passthrough
choice and non-choice questions keep theirs.
The returned dict is the model result plus a shortlist entry. Probabilities
on a shortlisted choice are over the kept labels only. shortlist[qid] holds
labels, scores, k, n, and passthrough. labels is the rank
order a shortlist produced, or the criteria order itself when passthrough is
set and no ranking ran; scores is the signed cosine of each kept label in that
order -- negative included, never clamped to 0 -- or None when nothing was
dropped.
Extra keyword arguments are forwarded to predict / system_one (for
example model= on a Router).
Parameters
agentAnystateAnyquestionsDict[str, Dict[str, Any]]embed_fnCallable[[Sequence[str]], Any]kint=DEFAULT_SHORTLIST_KDEFAULT_SHORTLIST_Kpredict_kwargsAny
predict_tournament
predict_tournament(
agent: Any,
state: Any,
questions: Dict[str, Dict[str, Any]],
group_size: int = DEFAULT_TOURNAMENT_GROUP,
DEFAULT_TOURNAMENT_GROUP,
predict_kwargs: Any,
) -> Dict[str, Any]Narrow each large choice question by elimination, then call predict once more.
A choice with more than group_size labels is cut, in criteria order, into groups of
near-equal size and at most group_size. One predict call answers every group of
every such question -- the groups go in as separate questions, so they share one forward
pass -- and each group's answer goes through to the next round. Rounds repeat until no
choice has more than group_size labels left; at the default 16, up to 256 labels take
one round. Unlike predict_shortlist this needs no embedder, and each label is read
with the option budget of a group_size-label question, not of the whole label set.
The final call answers every question of the request, with each choice that went through
a round cut to its finalists. Non-choice questions and choices of at most group_size
labels go to it unchanged, so when nothing needs a round it is the only call. The
caller's questions dict is not mutated.
A choice that went through a round loses an option_order it carried, with one
RuntimeWarning naming the question: the group and finalist questions hold fewer
labels than the caller's, and an order no longer fits the kept labels. Remapping it
onto them would silently move options between slots, so it is dropped instead. A
choice that needed no round keeps its order.
The returned dict is the final call's result plus a tournament entry.
tournament[qid] holds labels (the finalists, in criteria order), n (the
label count) and rounds for each choice question. Probabilities, confidences and
usage come from the final call, so a tournament choice's probabilities are over its
finalists only.
Extra keyword arguments are forwarded to every call (for example model= on a Router).
Parameters
agentAnystateAnyquestionsDict[str, Dict[str, Any]]group_sizeint=DEFAULT_TOURNAMENT_GROUPDEFAULT_TOURNAMENT_GROUPpredict_kwargsAny
embed_fn_from_agent
embed_fn_from_agent(
agent: Any,
max_length: int = 512,
batch_size: int = 32,
) -> Callable[[Sequence[str]], np.ndarray]Mean-pool the checkpoint encoder already loaded on agent.
The callable embeds a list of strings with agent.tok and agent.model.encoder.
It does not run the decision head and does not download weights. A dedicated
bi-encoder passed as embed_fn will usually shortlist better; this helper is
for callers who only have the Laya checkpoint in memory.
Padding positions are excluded from the mean. The encoder's train/eval flag is
left as the caller set it (a loaded Agent is already in eval).
Each call uses the current agent.device, including after CPU fallback.
Parameters
agentAnymax_lengthint=512batch_sizeint=32
cached_embed_fn
cached_embed_fn(
embed_fn: Callable[[Sequence[str]], Any],
maxsize: int = 4096,
) -> Callable[[Sequence[str]], np.ndarray]Cache embed_fn output per input string, under an LRU bound.
predict_shortlist embeds the query plus every option text on each call. When the
same option set is shortlisted on every request -- a fixed intent or label list, as
in the README's BANKING77 example -- the option rows do not change between calls,
yet they are re-embedded every time. Wrapping the embedder once::
embed_fn = cached_embed_fn(embed_fn_from_agent(agent))
leaves the first call unchanged and reduces each repeat call to embedding the new query alone.
Lookups are exact string matches. Texts missing from the cache are deduplicated and
embedded in a single embed_fn call, so a cold cache costs the same number of
batched calls as the unwrapped function. Rows are stored as float32; the cache holds
at most maxsize strings and then evicts the least recently used entry, bounding
memory at about maxsize * dim * 4 bytes. Nothing is cached when embed_fn
raises or returns a bad shape.
The wrapper is safe to share between threads: the lock covers only cache reads and
writes, never the embedding call. The returned callable carries cache_info() --
a dict with size, maxsize, hits and misses -- and cache_clear().
Clear the cache if the model or weights behind embed_fn change.
Parameters
embed_fnCallable[[Sequence[str]], Any]maxsizeint=4096
Abstention
check_min_confidence
check_min_confidence(v: Any)Validate opt-in abstention threshold min_confidence (#361, #394).
Either a real number in [0.0, 1.0] (one threshold for every answer; booleans rejected even
though isinstance(True, int)), or a per-bucket mapping (see :func:check_min_confidence_map)
so the threshold can differ by option count. Returns the value in its validated form -- a
float for the scalar case, a dict[str, float] for the mapping case -- which the gate
functions below both accept.
Parameters
vAny
check_min_confidence_map
check_min_confidence_map(m: Dict[Any, Any]) -> Dict[str, float]Validate a per-bucket abstention-threshold map (#394).
Keys are the option-count buckets in common.temp_bucket's spelling -- every
:func:bucket_keys member ("choice:2", "choice:3-5", "score:6-10", "noul:2"
and so on) -- plus the optional "default" used for any bucket the map does not name.
Anything else raises: a key no answer can produce is not an unused entry, it is a bucket the
caller believes is gated and is not, because :func:resolve_min_confidence falls past it to
"default" and then to 0.0 -- gate nothing. Values are floats in [0.0, 1.0]. One confidence
threshold does not transfer across option counts (#394); this lets a caller gate each bucket
at the level its calibration actually earns. Fit one with
:func:laya.calibrate.fit_abstention_thresholds, whose keys are this vocabulary by
construction.
Parameters
mDict[Any, Any]
resolve_min_confidence
resolve_min_confidence(
answer: Dict[str, Any],
thresholds: Dict[str, float],
default: float = 0.0,
) -> floatThe threshold this answer's option-count bucket is gated at, under a per-bucket map.
Falls back to the map's "default" entry, then to default (0.0 -- gate nothing), for a
bucket the map does not name, so an unconfigured bucket never abstains by surprise. A map that
went through :func:check_min_confidence_map can only miss here by omitting a bucket: a key
mis-spelled well enough that no answer produces it is refused at validation, because it would
otherwise reach this fall-through and leave that bucket gated at nothing while its caller
believes it is gated.
Parameters
answerDict[str, Any]thresholdsDict[str, float]defaultfloat=0.0
flag_low_confidence
flag_low_confidence(results: List[Dict[str, Any]], min_confidence: float) -> NoneOpt-in abstention marker (#361): flag answers whose confidence falls below min_confidence.
Reads answer_confidence (max(p), the quantity the calibration figures describe and the one
that does not drift with the number of options), falling back to confidence if
answer_confidence is absent.
The raw answer and confidence stay intact; low_confidence: True is added when the answer
falls below the threshold, and removed if a previously-flagged answer now clears it (e.g.
when a result dict is reused or re-evaluated with a different threshold).
min_confidence is either a float (one threshold for every answer) or a per-bucket mapping
(#394), in which case each answer is gated at the threshold of its own option-count bucket via
:func:resolve_min_confidence.
Parameters
resultsList[Dict[str, Any]]min_confidencefloat
apply_confidence_gate
apply_confidence_gate(
results: List[Dict[str, Any]],
min_confidence: Optional[float] = None,
) -> NoneReport the confidence gate's state, on the answers a gate was actually applied to.
A gate is a policy, and a policy whose application cannot be observed is not one. With
min_confidence set, this writes abstention -- one of :data:GATE_STATES -- onto every
answer, plus abstention_threshold, so a caller can answer three questions it otherwise
cannot:
- what fraction of decisions abstained, rather than inferring it from whether
low_confidencehappened to be set; - how many answers the gate could not decide on, which a boolean cannot express at all;
- what threshold produced these results --
flag_low_confidenceconsumes the threshold and drops it, so without this a batch run with per-class thresholds cannot be re-split.
GATE_UNEVALUATED is the case a boolean cannot express: the gate ran and the answer carried
no usable confidence, so the gate could not decide. Reporting that as a pass is the same lie
as reporting it as a flag.
With min_confidence unset, this writes nothing. No abstention, no
abstention_threshold, no flag. That is the whole contract: an ungated call returns exactly
the payload it returned before, and the presence of the field -- not a fourth value read out
of it -- is what tells a caller the gate ran. Call it unconditionally, once per call, in place
of an if min_confidence is not None: guard: that guard is what leaves a path reporting
nothing at all, which is the state this function exists to distinguish.
The flag itself stays :func:flag_low_confidence's -- this delegates rather than
re-implementing the rule, so the boolean and the reported state cannot drift apart.
A min_confidence of exactly 0.0 was set, so states are reported, and
:func:flag_low_confidence treats 0.0 as a no-op because nothing can fall below it. Every
answer carrying a usable confidence therefore reads passed, and the threshold echo is what
distinguishes that from a real pass at a real threshold.
Parameters
resultsList[Dict[str, Any]]min_confidenceOptional[float]=None
GATE_STATES
GATE_STATES = (GATE_PASSED, GATE_ABSTAINED, GATE_UNEVALUATED)Calibration and training
answer_confidence
answer_confidence(p: np.ndarray, k: int) -> floatProbability mass on the answer being reported: max(p).
This is the quantity temperature scaling fits, and the quantity every calibration figure in
this repository is computed on -- both benchmark harnesses take conf = max(probs) before
calling ece_score. The README's gating section relies on the property that goes with it:
of the answers returned at confidence c, about c of them are right. That property is
conditional, and the condition is not met by default -- it holds only after the temperatures
have been fitted and validated on held-out data for this checkpoint and this option count.
The shipped checkpoints are over-confident: choice:11+ is fitted at 0.1006, a ~10x
sharpener that would publish a genuine 0.24 top probability as 0.9999. clamp_temperature
refuses it and applies TEMP_MIN (0.5) instead, which still sharpens 2x and publishes that
same 0.24 as 0.4993 -- so a threshold applied to these numbers still selects below model
accuracy (issue #394). The clamp bounds the damage; it does not calibrate anything.
confidence_from_probs below reports a different quantity on a different scale and carries
no such guarantee, so the two must not be compared against the same threshold.
Parameters
pnp.ndarraykint
confidence_from_probs
confidence_from_probs(p: np.ndarray, k: int) -> floatNormalized Shannon entropy confidence: 1 - H(p) / log(k).
How concentrated the whole distribution is. Useful, but not calibrated: it is not what
temperature scaling fits and not what the reported ECE measures. See answer_confidence.
Parameters
pnp.ndarraykint
ece_score
ece_score(conf: np.ndarray, correct: np.ndarray, bins: int = 15) -> floatExpected Calibration Error across confidence bins.
Parameters
confnp.ndarraycorrectnp.ndarraybinsint=15
fit_temperatures
fit_temperatures = fit_temperature_mapfit_one_temperature
fit_one_temperature(pairs: Sequence, min_n: Optional[int] = None) -> floatFit one scalar T by NLL + LBFGS on log T.
The result is clamp_temperature of the optimised scale, so it lies in
[TEMP_MIN, TEMP_MAX] (or is the neutral 1.0 when the value is not a number).
Returns 1.0 when fewer than min_n pairs are given. min_n defaults to
MIN_BUCKET_N (the per-bucket floor). Type-level fits pass MIN_TYPE_N, which
is lower, so a dataset that fills no bucket still gets a scalar instead of
staying at 1.0.
Parameters
pairsSequencemin_nOptional[int]=None
fit_temperature_map
fit_temperature_map(
records: Iterable,
compute_ece: bool = False,
seed: int = 0,
) -> Dict[str, Any]Fit type-level scalars and per-bucket temperatures.
MIN_BUCKET_N is the per-bucket floor: smaller buckets are omitted from
temperature_by_options and the type-level scalar covers them. MIN_TYPE_N is
the separate, lower floor for that scalar only.
compute_ece=False (the default, and the path Agent.fit_temperatures stores)
fits on every record and returns no report key. seed is ignored on this path.
compute_ece=True holds out ECE_HOLDOUT_FRAC of each bucket, stratified by
temp_bucket, using seed so the same records always split the same way. Temperatures
are fit on the remainder only and ECE is scored only on the held-out records.
report["n"] is the number of records passed in; report["n_eval"] is the held-out
count the ECE rests on. A bucket that would drop below MIN_BUCKET_N after the
holdout is fit on all of its records, left out of the eval set, and named in
report["buckets_excluded_from_eval"] instead of being dropped. n_by_bucket
always counts the full input, including when the fit itself used a subset.
Parameters
recordsIterablecompute_ecebool=Falseseedint=0
fit_abstention_thresholds
fit_abstention_thresholds(
records: Iterable,
temperature: Sequence[float],
temperature_by_options: Dict[str, float],
binning_map: Optional[Dict[str, Dict[str, Any]]] = None,
target_error: float = 0.10,
min_bucket_n: int = MIN_ABSTAIN_BUCKET_N,
MIN_ABSTAIN_BUCKET_N,
conservative: bool = True,
) -> Dict[str, float]Fit a per-temp_bucket abstention threshold so a gate keeps a target error in every bucket.
A single min_confidence does not transfer across option counts (#394): the calibrated
confidence of a 2-option and a 12-option answer live on different scales, so one cut over- or
under-abstains depending on the question. This fits one cut per bucket instead, keyed exactly
like temperature_by_options (common.temp_bucket, e.g. "choice:3-5"), and the result is a
min_confidence map that :func:laya.confidence.check_min_confidence /
:func:laya.confidence.apply_confidence_gate accept directly.
records are the same (qtype, logits, target[, k]) tuples fit_temperature_map consumes
(records_from_labeled builds them). Confidence is the calibrated max(p) -- the logits are
scaled by the fitted temperature / temperature_by_options first, so thresholds and the
numbers the runtime reports are on the same scale. target_error is the tolerated error among
accepted answers; min_bucket_n omits buckets too small to fit, and conservative adds a
one-sample margin. The thresholds are empirical cuts on the calibration set, not a formal
coverage guarantee -- validate on held-out data (fit_temperature_map(..., compute_ece=True)
gives a held-out split) for a production gate.
Pass binning_map when the agent that will serve these thresholds has one installed -- by
Agent.fit_binning, or by a calibration payload that carries binning_map -- because the
runtime recalibrates answer_confidence through that map before anything reads it, so a cut
fitted without it is a cut on a scale the gate never sees. The thresholds are then on the binned
scale, and the order the two were fitted in stops mattering. Measured on 1,200 synthetic
12-option records at target_error=0.10: the cut fitted without a map holds 9.8% error over 50%
coverage on un-binned confidences, and admits 94.5% of answers at 25.6% error once the same
number is compared against binned ones.
Parameters
recordsIterabletemperatureSequence[float]temperature_by_optionsDict[str, float]binning_mapOptional[Dict[str, Dict[str, Any]]]=Nonetarget_errorfloat=0.10min_bucket_nint=MIN_ABSTAIN_BUCKET_NMIN_ABSTAIN_BUCKET_Nconservativebool=True
fit_binning_map
fit_binning_map(
records: Iterable,
temperature: Sequence[float],
temperature_by_options: Dict[str, float],
bins: int = 15,
min_bucket_n: int = MIN_BINNING_BUCKET_N,
MIN_BINNING_BUCKET_N,
) -> Dict[str, Dict[str, Any]]Fit a per-temp_bucket histogram-binning recalibration map for answer_confidence.
Temperature scaling applies one scalar per bucket; it cannot fix a bucket whose reliability
curve is not a simple sharpening/softening (the pathological choice:11+ the shipped English
checkpoint carries is one). Histogram binning is the non-parametric alternative: split the
calibrated confidences of a bucket into bins equal-width bins over [0, 1], and map every
confidence that lands in a bin to that bin's empirical accuracy. It needs no monotonicity
assumption and no extra dependency (NumPy only; isotonic regression would pull in scikit-learn).
records are the same (qtype, logits, target[, k]) tuples fit_temperature_map consumes;
confidence is the calibrated max(p) (logits scaled by the fitted temperature /
temperature_by_options first), so a binning map composes on top of a temperature map rather
than replacing it. Returns {bucket: {"bins": N, "values": [recalibrated confidence per bin]}};
buckets below min_bucket_n are omitted. Apply it with :func:apply_binning_map. An empty bin
(a confidence range the calibration set never produced) maps to its own midpoint, i.e. leaves
that region unchanged, so an unseen value is never recalibrated to a fabricated 0.
Parameters
recordsIterabletemperatureSequence[float]temperature_by_optionsDict[str, float]binsint=15min_bucket_nint=MIN_BINNING_BUCKET_NMIN_BINNING_BUCKET_N
apply_binning_map
apply_binning_map(
confidence: float,
bucket: str,
binning_map: Dict[str, Dict[str, Any]],
) -> floatRecalibrate one answer_confidence for its option-count bucket (common.temp_bucket).
Returns the confidence unchanged when the map has no entry for the bucket, so a bucket the map was not fit for passes through rather than being forced to a wrong value.
Parameters
confidencefloatbucketstrbinning_mapDict[str, Dict[str, Any]]
fit_binning
fit_binning(
records,
min_bucket_n: int = MIN_BINNING_BUCKET_N,
MIN_BINNING_BUCKET_N,
) -> Dict[str, Any]Fit a histogram-binning map on top of this agent's fitted temperatures and store it.
records are the same (qtype, logits, target[, k]) tuples as fit_temperatures
consumed. The map is keyed exactly like temperature_by_options, composes on top of
the current temperatures, and is written out by save_calibration as binning_map.
Parameters
recordsmin_bucket_nint=MIN_BINNING_BUCKET_NMIN_BINNING_BUCKET_N
render_options
render_options(q: Dict) -> List[str]Render option texts in label-index order. Noul semantic order is always [false, true].
Parameters
qDict
proper_reward
proper_reward(
q: torch.Tensor,
target: torch.Tensor,
qtype: torch.Tensor,
mask: torch.Tensor,
w_sph: float = 0.5,
w_rps: float = 1.0,
log_floor: float = -9.21,
) -> torch.TensorStrictly proper scoring rule reward: log score + spherical score + ranked probability score.
q: [..., N, K] reported distributions target: [N, K] (one-hot or soft target distributions)
Parameters
qtorch.Tensortargettorch.Tensorqtypetorch.Tensormasktorch.Tensorw_sphfloat=0.5w_rpsfloat=1.0log_floorfloat=-9.21
td_lambda_targets
td_lambda_targets(p_true: torch.Tensor, batch: Dict, lam: float = 1.0) -> torch.TensorTD(lambda) targets for multi-turn conversation trajectories.
Parameters
p_truetorch.TensorbatchDictlamfloat=1.0
QTYPES
QTYPES = {"choice": 0, "score": 1, "noul": 2}QTYPE_NAMES
QTYPE_NAMES = {v: k for k, v in QTYPES.items()}