Docs

Helpers

Language detection

laya.detect_language is laya.lang.analyse.

Names, types, defaults and code stay in English; the rest is translated (entries not translated yet are shown in the original English).

analyse

analyse(state: Union[str, bytes, Mapping, list, None]) -> Dict[str, object]

Full detection result for a state.

Returns script, script_profile, language (best effort, may be None), is_english, non_latin_fraction and mixed_segment (the line or field that made a mostly English state non-English, else None).

String values are what get read. When a state has several of them, one non-English value is enough: joining every value into one window let a long English note fill the 4000 characters, or outvote a short German message, and that message was then sent to the English checkpoint (#384). The segment scan still stops at 4000 characters, which is what keeps a huge field cheap; a value it did not reach is read on its own afterwards.

Parameters

stateUnion[str, bytes, Mapping, list, None]

detect_script

detect_script(text: str) -> str

Dominant script of text: 'latin', 'han', 'devanagari', ... or 'unknown' if there are no letters.

Parameters

textstr

is_english

is_english(state: Union[str, bytes, Mapping, list, None]) -> bool

True when the English checkpoint can be expected to read this state.

Parameters

stateUnion[str, bytes, Mapping, list, None]

Email

clean_email_body

clean_email_body(body: str, max_chars: int = 3000) -> str

Remove quoted email history, signatures and disclaimers to keep input focused.

max_chars is the length the result is cut to, 3000 characters unless raised -- see email_state, which takes the same budget and passes it through.

Parameters

bodystr
max_charsint= 3000

email_state

email_state(
    subject: str,
    body: str,
    sender: Optional[str] = None,
    clean: bool = True,
    max_chars: int = 3000,
    extra,
) -> Dict

Construct a clean state dictionary for email classification.

max_chars is the budget clean_email_body cuts the body to, and it is worth raising for a long message: at the default the body stops after 3000 characters, so a request that arrives in the last paragraphs never reaches the model -- including through predict_long, which scans a state in windows precisely so it can read past one window's worth. Ignored when clean=False, which passes the body through whole.

Any other keyword becomes a field of the state, so it is read by the model; a typo here is an input mutation, not an error. The exception is a keyword whose value is exactly None: it is dropped rather than added, so email_state(..., thread_id=None) gives the same state as leaving the keyword out. The key is absent, not null -- and the distinction reaches the model, because laya.common.serialize_state turns the state into JSON before it is tokenized, so a retained None would put "thread_id": null in the prompt. sender is dropped on the same call by a different rule, falsiness rather than None, so sender="" leaves no from field either.

Parameters

subjectstr
bodystr
senderOptional[str]= None
cleanbool= True
max_charsint= 3000
extra

Question presets

triage_questions

triage_questions() -> Dict

Preset questions for customer support ticket triage.

email_questions

email_questions(categories: Optional[Dict[str, str]] = None) -> Dict

Preset questions for inbound email triage and threat filtering.

Parameters

categoriesOptional[Dict[str, str]]= None

guard_questions

guard_questions() -> Dict

Preset questions for real-time LLM input guardrails.

moderation_questions

moderation_questions() -> Dict

Preset questions for content safety and moderation.

router_questions

router_questions() -> Dict

Preset questions for intelligent model routing.

Pre-run request sizing

What a request will fit before any forward pass. These are the helpers predict_long itself reads, so a question can be measured without holding an Agent: build_head is the question half of a sequence, state_room is what is left for the state after it, and build_sequence reports what the head budget did to the options and to the state.

from transformers import AutoTokenizer
from laya.common import build_sequence, collapsed_options, state_room, window_budget

tok = AutoTokenizer.from_pretrained("convaiinnovations/laya", subfolder="tokenizer")
max_len, head_max_len = 512, 192      # from the checkpoint's rl_agent_config.json

question = {"type": "choice", "instructions": "What does the customer want?",
            "criteria": {"refund": "money back", "cancel": "stop the service"}}
# The short-key internal shape the sizing helpers read; inline the mapping as
# questions-and-answers.md shows rather than reaching into Agent._to_internal.
q = {"t": question["type"], "ins": question["instructions"], "crit": question["criteria"]}
state = "I was charged twice for the same invoice."

state_room(tok, q, max_len, head_max_len)   # state tokens this question leaves room for
seq, markers, stats, trunc = build_sequence(
    tok, state, q, max_len, head_max_len, return_stats=True, return_truncation_stats=True)
stats       # how many options the head budget kept distinct, and the cap it applied
trunc       # state tokens, used, dropped, and whether the state was truncated
collapsed_options(["q"], [{"options": stats}])   # {} until an option loses its own span
window_budget(tok, [q], max_len, head_max_len)   # (window, stride, room) a scan may use

The same numbers come back after a call in usage: state_tokens, state_tokens_dropped, truncated, truncated_questions, and options when an option lost its span. That block is the record of what a run did; the helpers above are how to find out before paying for one. predict_long hands a state straight to a single pass when its tokens fit the room its questions leave, and a scan window is capped at that room with a 50%-of-window stride – a cap large enough to multiply the passes warns before the scan runs.

None of this picks a strategy: tournament rounds come back as tournament[qid]["rounds"] after predict_tournament, and shortlist ranking quality stays with whatever embedder supplies it.

state_room

state_room(tok, q: Dict, max_len: int = 512, head_max_len: int = 192) -> int

How many state tokens q leaves inside max_len, which is what build_sequence keeps.

The head is the question's own -- its instructions plus one [MASK]-prefixed span per option -- so a question with many options leaves less room for the state than one with two, and two questions in the same request do not have to leave the same amount. Anything past the return value is cut off (the start is kept, or the end for a conversation turn list).

Parameters

tok
qDict
max_lenint= 512
head_max_lenint= 192

window_budget

window_budget(
    tok,
    questions,
    max_len: int = 512,
    head_max_len: int = 192,
    window: Optional[int] = None,
    stride: Optional[int] = None,
)

Window size and stride for scanning a state that is longer than one sequence.

predict_long decodes each token window back to text and scores it as an ordinary state, so a window wider than the room the questions leave is re-truncated by build_sequence on the way in: the tail of every window reaches no model, while the reported span says it did. The window is therefore capped at state_room, and at the smallest room of questions, because the windows are one list of states scored for every question in shared forward passes -- a window sized for the roomiest question would be cut short for the tightest one, and the offsets reported on its answers would mean something different per question. With no questions there is nothing to fit, so the caller's window (or the checkpoint default) stands. That default is max(64, max_len - head_max_len - 8): the state budget the config leaves, with a 64-token floor under it, so a widened head_max_len stops shrinking it there.

questions are internal question dicts, as Agent._to_internal returns them.

An explicit window wider than the room is clamped to it with a RuntimeWarning, since a scan at the requested size cannot read what it claims to. A stride past the effective window is refused: the tokens between two windows would be read by neither, which is the failure predict_long exists to prevent. The default stride keeps its 50% overlap of the effective window, so a span near a boundary still lands whole inside some window; an explicit stride equal to the window still reads every token, with no overlap to catch a span that straddles a boundary.

Returns (window, stride, room).

Parameters

tok
questions
max_lenint= 512
head_max_lenint= 192
windowOptional[int]= None
strideOptional[int]= None

collapsed_options

collapsed_options(qids, items) -> Dict[str, Dict[str, Optional[int]]]

The questions whose options no longer have a token span each, from per-item stats.

total is the number of options the question defines, not the number of markers that reached the sequence: a report counted from the markers would say "43/58" about a request where 28 options never made it into the input at all.

Parameters

qids
items

Shortlisting

shortlist_choice

shortlist_choice(
    state: Any,
    criteria: Any,
    embed_fn: Callable[[Sequence[str]], Any],
    k: int = DEFAULT_SHORTLIST_K,
    DEFAULT_SHORTLIST_K,
    instructions: Optional[str] = None,
    return_scores: bool = False,
) -> Any

Return the top-k choice labels for state.

embed_fn maps a list of strings to an array of shape (len(texts), dim). It is called once, with the query text first and then one string per option in criteria order. Option strings match render_options for a choice question.

When k is at least the number of labels, every label is returned in its original order and embed_fn is not called.

Ties keep the earlier label. Ranking is a signed cosine, not a similarity floor: a label that scores 0 -- no signal at all, or a non-finite vector treated as one -- does outrank an earlier label that scored negative, and k drops the negative labels first.

With return_scores=True the return is the (labels, scores) pair, where scores holds the signed cosine per kept label in rank order -- the same values predict_shortlist reports in its shortlist metadata. scores is None when nothing was dropped, exactly as in that metadata.

Parameters

stateAny
criteriaAny
embed_fnCallable[[Sequence[str]], Any]
kint= DEFAULT_SHORTLIST_K
DEFAULT_SHORTLIST_K
instructionsOptional[str]= None
return_scoresbool= False

predict_shortlist

predict_shortlist(
    agent: Any,
    state: Any,
    questions: Dict[str, Dict[str, Any]],
    embed_fn: Callable[[Sequence[str]], Any],
    k: int = DEFAULT_SHORTLIST_K,
    DEFAULT_SHORTLIST_K,
    predict_kwargs: Any,
) -> Dict[str, Any]

Shortlist each choice question, then call predict or system_one once.

Non-choice questions are forwarded unchanged. A choice whose label count is <= k is forwarded unchanged and does not call embed_fn. The caller's questions dict is not mutated.

A narrowed choice that carried an option_order has it dropped, with one RuntimeWarning naming the question: the order has one slot per option, so an order that fitted the caller's label set no longer fits the kept labels and the agent refuses the question. Remapping the order onto the kept labels would silently move options between slots, so it is dropped instead. A passthrough choice and non-choice questions keep theirs.

The returned dict is the model result plus a shortlist entry. Probabilities on a shortlisted choice are over the kept labels only. shortlist[qid] holds labels, scores, k, n, and passthrough. labels is the rank order a shortlist produced, or the criteria order itself when passthrough is set and no ranking ran; scores is the signed cosine of each kept label in that order -- negative included, never clamped to 0 -- or None when nothing was dropped.

Extra keyword arguments are forwarded to predict / system_one (for example model= on a Router).

Parameters

agentAny
stateAny
questionsDict[str, Dict[str, Any]]
embed_fnCallable[[Sequence[str]], Any]
kint= DEFAULT_SHORTLIST_K
DEFAULT_SHORTLIST_K
predict_kwargsAny

predict_tournament

predict_tournament(
    agent: Any,
    state: Any,
    questions: Dict[str, Dict[str, Any]],
    group_size: int = DEFAULT_TOURNAMENT_GROUP,
    DEFAULT_TOURNAMENT_GROUP,
    predict_kwargs: Any,
) -> Dict[str, Any]

Narrow each large choice question by elimination, then call predict once more.

A choice with more than group_size labels is cut, in criteria order, into groups of near-equal size and at most group_size. One predict call answers every group of every such question -- the groups go in as separate questions, so they share one forward pass -- and each group's answer goes through to the next round. Rounds repeat until no choice has more than group_size labels left; at the default 16, up to 256 labels take one round. Unlike predict_shortlist this needs no embedder, and each label is read with the option budget of a group_size-label question, not of the whole label set.

The final call answers every question of the request, with each choice that went through a round cut to its finalists. Non-choice questions and choices of at most group_size labels go to it unchanged, so when nothing needs a round it is the only call. The caller's questions dict is not mutated.

A choice that went through a round loses an option_order it carried, with one RuntimeWarning naming the question: the group and finalist questions hold fewer labels than the caller's, and an order no longer fits the kept labels. Remapping it onto them would silently move options between slots, so it is dropped instead. A choice that needed no round keeps its order.

The returned dict is the final call's result plus a tournament entry. tournament[qid] holds labels (the finalists, in criteria order), n (the label count) and rounds for each choice question. Probabilities, confidences and usage come from the final call, so a tournament choice's probabilities are over its finalists only.

Extra keyword arguments are forwarded to every call (for example model= on a Router).

Parameters

agentAny
stateAny
questionsDict[str, Dict[str, Any]]
group_sizeint= DEFAULT_TOURNAMENT_GROUP
DEFAULT_TOURNAMENT_GROUP
predict_kwargsAny

embed_fn_from_agent

embed_fn_from_agent(
    agent: Any,
    max_length: int = 512,
    batch_size: int = 32,
) -> Callable[[Sequence[str]], np.ndarray]

Mean-pool the checkpoint encoder already loaded on agent.

The callable embeds a list of strings with agent.tok and agent.model.encoder. It does not run the decision head and does not download weights. A dedicated bi-encoder passed as embed_fn will usually shortlist better; this helper is for callers who only have the Laya checkpoint in memory.

Padding positions are excluded from the mean. The encoder's train/eval flag is left as the caller set it (a loaded Agent is already in eval). Each call uses the current agent.device, including after CPU fallback.

Parameters

agentAny
max_lengthint= 512
batch_sizeint= 32

cached_embed_fn

cached_embed_fn(
    embed_fn: Callable[[Sequence[str]], Any],
    maxsize: int = 4096,
) -> Callable[[Sequence[str]], np.ndarray]

Cache embed_fn output per input string, under an LRU bound.

predict_shortlist embeds the query plus every option text on each call. When the same option set is shortlisted on every request -- a fixed intent or label list, as in the README's BANKING77 example -- the option rows do not change between calls, yet they are re-embedded every time. Wrapping the embedder once::

embed_fn = cached_embed_fn(embed_fn_from_agent(agent))

leaves the first call unchanged and reduces each repeat call to embedding the new query alone.

Lookups are exact string matches. Texts missing from the cache are deduplicated and embedded in a single embed_fn call, so a cold cache costs the same number of batched calls as the unwrapped function. Rows are stored as float32; the cache holds at most maxsize strings and then evicts the least recently used entry, bounding memory at about maxsize * dim * 4 bytes. Nothing is cached when embed_fn raises or returns a bad shape.

The wrapper is safe to share between threads: the lock covers only cache reads and writes, never the embedding call. The returned callable carries cache_info() -- a dict with size, maxsize, hits and misses -- and cache_clear(). Clear the cache if the model or weights behind embed_fn change.

Parameters

embed_fnCallable[[Sequence[str]], Any]
maxsizeint= 4096

Abstention

check_min_confidence

check_min_confidence(v: Any)

Validate opt-in abstention threshold min_confidence (#361, #394).

Either a real number in [0.0, 1.0] (one threshold for every answer; booleans rejected even though isinstance(True, int)), or a per-bucket mapping (see :func:check_min_confidence_map) so the threshold can differ by option count. Returns the value in its validated form -- a float for the scalar case, a dict[str, float] for the mapping case -- which the gate functions below both accept.

Parameters

vAny

check_min_confidence_map

check_min_confidence_map(m: Dict[Any, Any]) -> Dict[str, float]

Validate a per-bucket abstention-threshold map (#394).

Keys are the option-count buckets in common.temp_bucket's spelling -- every :func:bucket_keys member ("choice:2", "choice:3-5", "score:6-10", "noul:2" and so on) -- plus the optional "default" used for any bucket the map does not name. Anything else raises: a key no answer can produce is not an unused entry, it is a bucket the caller believes is gated and is not, because :func:resolve_min_confidence falls past it to "default" and then to 0.0 -- gate nothing. Values are floats in [0.0, 1.0]. One confidence threshold does not transfer across option counts (#394); this lets a caller gate each bucket at the level its calibration actually earns. Fit one with :func:laya.calibrate.fit_abstention_thresholds, whose keys are this vocabulary by construction.

Parameters

mDict[Any, Any]

resolve_min_confidence

resolve_min_confidence(
    answer: Dict[str, Any],
    thresholds: Dict[str, float],
    default: float = 0.0,
) -> float

The threshold this answer's option-count bucket is gated at, under a per-bucket map.

Falls back to the map's "default" entry, then to default (0.0 -- gate nothing), for a bucket the map does not name, so an unconfigured bucket never abstains by surprise. A map that went through :func:check_min_confidence_map can only miss here by omitting a bucket: a key mis-spelled well enough that no answer produces it is refused at validation, because it would otherwise reach this fall-through and leave that bucket gated at nothing while its caller believes it is gated.

Parameters

answerDict[str, Any]
thresholdsDict[str, float]
defaultfloat= 0.0

flag_low_confidence

flag_low_confidence(results: List[Dict[str, Any]], min_confidence: float) -> None

Opt-in abstention marker (#361): flag answers whose confidence falls below min_confidence.

Reads answer_confidence (max(p), the quantity the calibration figures describe and the one that does not drift with the number of options), falling back to confidence if answer_confidence is absent. The raw answer and confidence stay intact; low_confidence: True is added when the answer falls below the threshold, and removed if a previously-flagged answer now clears it (e.g. when a result dict is reused or re-evaluated with a different threshold).

min_confidence is either a float (one threshold for every answer) or a per-bucket mapping (#394), in which case each answer is gated at the threshold of its own option-count bucket via :func:resolve_min_confidence.

Parameters

resultsList[Dict[str, Any]]
min_confidencefloat

apply_confidence_gate

apply_confidence_gate(
    results: List[Dict[str, Any]],
    min_confidence: Optional[float] = None,
) -> None

Report the confidence gate's state, on the answers a gate was actually applied to.

A gate is a policy, and a policy whose application cannot be observed is not one. With min_confidence set, this writes abstention -- one of :data:GATE_STATES -- onto every answer, plus abstention_threshold, so a caller can answer three questions it otherwise cannot:

  • what fraction of decisions abstained, rather than inferring it from whether low_confidence happened to be set;
  • how many answers the gate could not decide on, which a boolean cannot express at all;
  • what threshold produced these results -- flag_low_confidence consumes the threshold and drops it, so without this a batch run with per-class thresholds cannot be re-split.

GATE_UNEVALUATED is the case a boolean cannot express: the gate ran and the answer carried no usable confidence, so the gate could not decide. Reporting that as a pass is the same lie as reporting it as a flag.

With min_confidence unset, this writes nothing. No abstention, no abstention_threshold, no flag. That is the whole contract: an ungated call returns exactly the payload it returned before, and the presence of the field -- not a fourth value read out of it -- is what tells a caller the gate ran. Call it unconditionally, once per call, in place of an if min_confidence is not None: guard: that guard is what leaves a path reporting nothing at all, which is the state this function exists to distinguish.

The flag itself stays :func:flag_low_confidence's -- this delegates rather than re-implementing the rule, so the boolean and the reported state cannot drift apart.

A min_confidence of exactly 0.0 was set, so states are reported, and :func:flag_low_confidence treats 0.0 as a no-op because nothing can fall below it. Every answer carrying a usable confidence therefore reads passed, and the threshold echo is what distinguishes that from a real pass at a real threshold.

Parameters

resultsList[Dict[str, Any]]
min_confidenceOptional[float]= None

GATE_STATES

GATE_STATES = (GATE_PASSED, GATE_ABSTAINED, GATE_UNEVALUATED)

Calibration and training

answer_confidence

answer_confidence(p: np.ndarray, k: int) -> float

Probability mass on the answer being reported: max(p).

This is the quantity temperature scaling fits, and the quantity every calibration figure in this repository is computed on -- both benchmark harnesses take conf = max(probs) before calling ece_score. The README's gating section relies on the property that goes with it: of the answers returned at confidence c, about c of them are right. That property is conditional, and the condition is not met by default -- it holds only after the temperatures have been fitted and validated on held-out data for this checkpoint and this option count. The shipped checkpoints are over-confident: choice:11+ is fitted at 0.1006, a ~10x sharpener that would publish a genuine 0.24 top probability as 0.9999. clamp_temperature refuses it and applies TEMP_MIN (0.5) instead, which still sharpens 2x and publishes that same 0.24 as 0.4993 -- so a threshold applied to these numbers still selects below model accuracy (issue #394). The clamp bounds the damage; it does not calibrate anything.

confidence_from_probs below reports a different quantity on a different scale and carries no such guarantee, so the two must not be compared against the same threshold.

Parameters

pnp.ndarray
kint

confidence_from_probs

confidence_from_probs(p: np.ndarray, k: int) -> float

Normalized Shannon entropy confidence: 1 - H(p) / log(k).

How concentrated the whole distribution is. Useful, but not calibrated: it is not what temperature scaling fits and not what the reported ECE measures. See answer_confidence.

Parameters

pnp.ndarray
kint

ece_score

ece_score(conf: np.ndarray, correct: np.ndarray, bins: int = 15) -> float

Expected Calibration Error across confidence bins.

Parameters

confnp.ndarray
correctnp.ndarray
binsint= 15

fit_temperatures

fit_temperatures = fit_temperature_map

fit_one_temperature

fit_one_temperature(pairs: Sequence, min_n: Optional[int] = None) -> float

Fit one scalar T by NLL + LBFGS on log T.

The result is clamp_temperature of the optimised scale, so it lies in [TEMP_MIN, TEMP_MAX] (or is the neutral 1.0 when the value is not a number). Returns 1.0 when fewer than min_n pairs are given. min_n defaults to MIN_BUCKET_N (the per-bucket floor). Type-level fits pass MIN_TYPE_N, which is lower, so a dataset that fills no bucket still gets a scalar instead of staying at 1.0.

Parameters

pairsSequence
min_nOptional[int]= None

fit_temperature_map

fit_temperature_map(
    records: Iterable,
    compute_ece: bool = False,
    seed: int = 0,
) -> Dict[str, Any]

Fit type-level scalars and per-bucket temperatures.

MIN_BUCKET_N is the per-bucket floor: smaller buckets are omitted from temperature_by_options and the type-level scalar covers them. MIN_TYPE_N is the separate, lower floor for that scalar only.

compute_ece=False (the default, and the path Agent.fit_temperatures stores) fits on every record and returns no report key. seed is ignored on this path.

compute_ece=True holds out ECE_HOLDOUT_FRAC of each bucket, stratified by temp_bucket, using seed so the same records always split the same way. Temperatures are fit on the remainder only and ECE is scored only on the held-out records. report["n"] is the number of records passed in; report["n_eval"] is the held-out count the ECE rests on. A bucket that would drop below MIN_BUCKET_N after the holdout is fit on all of its records, left out of the eval set, and named in report["buckets_excluded_from_eval"] instead of being dropped. n_by_bucket always counts the full input, including when the fit itself used a subset.

Parameters

recordsIterable
compute_ecebool= False
seedint= 0

fit_abstention_thresholds

fit_abstention_thresholds(
    records: Iterable,
    temperature: Sequence[float],
    temperature_by_options: Dict[str, float],
    binning_map: Optional[Dict[str, Dict[str, Any]]] = None,
    target_error: float = 0.10,
    min_bucket_n: int = MIN_ABSTAIN_BUCKET_N,
    MIN_ABSTAIN_BUCKET_N,
    conservative: bool = True,
) -> Dict[str, float]

Fit a per-temp_bucket abstention threshold so a gate keeps a target error in every bucket.

A single min_confidence does not transfer across option counts (#394): the calibrated confidence of a 2-option and a 12-option answer live on different scales, so one cut over- or under-abstains depending on the question. This fits one cut per bucket instead, keyed exactly like temperature_by_options (common.temp_bucket, e.g. "choice:3-5"), and the result is a min_confidence map that :func:laya.confidence.check_min_confidence / :func:laya.confidence.apply_confidence_gate accept directly.

records are the same (qtype, logits, target[, k]) tuples fit_temperature_map consumes (records_from_labeled builds them). Confidence is the calibrated max(p) -- the logits are scaled by the fitted temperature / temperature_by_options first, so thresholds and the numbers the runtime reports are on the same scale. target_error is the tolerated error among accepted answers; min_bucket_n omits buckets too small to fit, and conservative adds a one-sample margin. The thresholds are empirical cuts on the calibration set, not a formal coverage guarantee -- validate on held-out data (fit_temperature_map(..., compute_ece=True) gives a held-out split) for a production gate.

Pass binning_map when the agent that will serve these thresholds has one installed -- by Agent.fit_binning, or by a calibration payload that carries binning_map -- because the runtime recalibrates answer_confidence through that map before anything reads it, so a cut fitted without it is a cut on a scale the gate never sees. The thresholds are then on the binned scale, and the order the two were fitted in stops mattering. Measured on 1,200 synthetic 12-option records at target_error=0.10: the cut fitted without a map holds 9.8% error over 50% coverage on un-binned confidences, and admits 94.5% of answers at 25.6% error once the same number is compared against binned ones.

Parameters

recordsIterable
temperatureSequence[float]
temperature_by_optionsDict[str, float]
binning_mapOptional[Dict[str, Dict[str, Any]]]= None
target_errorfloat= 0.10
min_bucket_nint= MIN_ABSTAIN_BUCKET_N
MIN_ABSTAIN_BUCKET_N
conservativebool= True

fit_binning_map

fit_binning_map(
    records: Iterable,
    temperature: Sequence[float],
    temperature_by_options: Dict[str, float],
    bins: int = 15,
    min_bucket_n: int = MIN_BINNING_BUCKET_N,
    MIN_BINNING_BUCKET_N,
) -> Dict[str, Dict[str, Any]]

Fit a per-temp_bucket histogram-binning recalibration map for answer_confidence.

Temperature scaling applies one scalar per bucket; it cannot fix a bucket whose reliability curve is not a simple sharpening/softening (the pathological choice:11+ the shipped English checkpoint carries is one). Histogram binning is the non-parametric alternative: split the calibrated confidences of a bucket into bins equal-width bins over [0, 1], and map every confidence that lands in a bin to that bin's empirical accuracy. It needs no monotonicity assumption and no extra dependency (NumPy only; isotonic regression would pull in scikit-learn).

records are the same (qtype, logits, target[, k]) tuples fit_temperature_map consumes; confidence is the calibrated max(p) (logits scaled by the fitted temperature / temperature_by_options first), so a binning map composes on top of a temperature map rather than replacing it. Returns {bucket: {"bins": N, "values": [recalibrated confidence per bin]}}; buckets below min_bucket_n are omitted. Apply it with :func:apply_binning_map. An empty bin (a confidence range the calibration set never produced) maps to its own midpoint, i.e. leaves that region unchanged, so an unseen value is never recalibrated to a fabricated 0.

Parameters

recordsIterable
temperatureSequence[float]
temperature_by_optionsDict[str, float]
binsint= 15
min_bucket_nint= MIN_BINNING_BUCKET_N
MIN_BINNING_BUCKET_N

apply_binning_map

apply_binning_map(
    confidence: float,
    bucket: str,
    binning_map: Dict[str, Dict[str, Any]],
) -> float

Recalibrate one answer_confidence for its option-count bucket (common.temp_bucket).

Returns the confidence unchanged when the map has no entry for the bucket, so a bucket the map was not fit for passes through rather than being forced to a wrong value.

Parameters

confidencefloat
bucketstr
binning_mapDict[str, Dict[str, Any]]

fit_binning

fit_binning(
    records,
    min_bucket_n: int = MIN_BINNING_BUCKET_N,
    MIN_BINNING_BUCKET_N,
) -> Dict[str, Any]

Fit a histogram-binning map on top of this agent's fitted temperatures and store it.

records are the same (qtype, logits, target[, k]) tuples as fit_temperatures consumed. The map is keyed exactly like temperature_by_options, composes on top of the current temperatures, and is written out by save_calibration as binning_map.

Parameters

records
min_bucket_nint= MIN_BINNING_BUCKET_N
MIN_BINNING_BUCKET_N

render_options

render_options(q: Dict) -> List[str]

Render option texts in label-index order. Noul semantic order is always [false, true].

Parameters

qDict

proper_reward

proper_reward(
    q: torch.Tensor,
    target: torch.Tensor,
    qtype: torch.Tensor,
    mask: torch.Tensor,
    w_sph: float = 0.5,
    w_rps: float = 1.0,
    log_floor: float = -9.21,
) -> torch.Tensor

Strictly proper scoring rule reward: log score + spherical score + ranked probability score.

q: [..., N, K] reported distributions target: [N, K] (one-hot or soft target distributions)

Parameters

qtorch.Tensor
targettorch.Tensor
qtypetorch.Tensor
masktorch.Tensor
w_sphfloat= 0.5
w_rpsfloat= 1.0
log_floorfloat= -9.21

td_lambda_targets

td_lambda_targets(p_true: torch.Tensor, batch: Dict, lam: float = 1.0) -> torch.Tensor

TD(lambda) targets for multi-turn conversation trajectories.

Parameters

p_truetorch.Tensor
batchDict
lamfloat= 1.0

QTYPES

QTYPES = {"choice": 0, "score": 1, "noul": 2}

QTYPE_NAMES

QTYPE_NAMES = {v: k for k, v in QTYPES.items()}