SLCMB

The Supersonic Labs Classification Models Benchmark measures what a model decides, how confident it is, and whether that decision holds when the options move.

What it measures

The model receives a context, a question and the permitted answers. A question can ask for a choice from a closed set, an ordered score, or a yes/no decision. The scorer checks the chosen answer and the probability distribution. For example, a ticket routed to Billing should still go to Billing when that option moves from first to last. A shift in probabilities can reveal sensitivity even when the selected answer stays the same.

SetQuestionsWhat it covers
Core scenarios200Constructed scenarios for routing, triage, written rules, constraints and near-miss alternatives
Public-source questions690News, emotion, banking intent, spam, science, exams and selected typed decisions
Ethical dilemmas30Descriptive comparison of choices and order sensitivity, with no correct answer assigned
Total920890 questions with an answer key and 30 descriptive cases

The core is balanced between English and Brazilian Portuguese. The public-source expansion contains 583 English items and 107 Portuguese items. The 16 reporting areas include operational domains and public-source tasks. Some areas have limited samples.

Evaluation protocol

  1. Freeze the question set, option-order schedule and scorer. Record the model checkpoint, adapter, settings and environment.
  2. Present each question 100 times, keeping the context and question fixed. Reorder the options using the same schedule for every model. The full run has 92,000 presentations per model, including repeated orderings used as controls.
  3. Preserve option keys when moving options. For a binary question, each option appears first 50 times. For a four-level rubric, each level appears in each position 25 times. With larger option sets the schedule balances positions as closely as 100 presentations allow.
  4. Report exact accuracy, agreement between repetitions, changes to the full probability vector, sensitivity to position and coverage. Invalid responses count against the score. Report latency separately with its measurement conditions.

The 100 presentations of a question are repeated measurements of one item. Uncertainty estimates resample questions with their repetitions kept together.

How scoring works

A local score combines 80% decision quality and 20% reliability. Decision quality combines 65% exact accuracy with 35% probability scoring, using Brier loss for categorical or binary outcomes and ranked probability score for ordered levels. Reliability considers accuracy on the hardest presentations, agreement across repetitions, stability of probabilities, order sensitivity and valid-response coverage.

Version 3 aggregates the covered languages within each area and the 16 areas overall using (mean of the square roots of the scores)². On two equally weighted areas, scores of 50 and 50 combine to 50, while scores of 100 and 0 combine to 25. This gives weak areas greater influence than a simple average would. The resulting SLCMB points are a composite score out of 100.

Model results

ModelFull-set SLCMB 3.0 pointsSource
Julia 133.3Julia 1.5 model card comparison
Julia 1.546.2Julia 1.5 model card
Julia 264.7Julia 2 preliminary internal evaluation

The model-card scores shown here use the full 920-question set reported by each release. Julia 2 also reports 67.7 on the development split. Flash’s paired evaluation uses a separate held-out subset of 222 questions. Keep the dataset revision and split fixed when comparing runs. Julia 1.5 results · Julia 2 and Flash results

Scope and next steps

The benchmark measures performance on a defined set. The operational scenarios are constructed for evaluation, and the public-source labels have varying levels of review. For a production decision, also evaluate with representative data and independently reviewed labels. The ethical cases describe choices under the stated conditions. Latency belongs to the tested hardware and adapter.

As Julia gains capabilities, we’ll add tasks and publish the scoring and split used for each new evaluation.