SLCMB
The Supersonic Labs Classification Models Benchmark measures what a model decides, how confident it is, and whether that decision holds when the options move.
What it measures
The model receives a context, a question and the permitted answers. A question can ask for a choice from a closed set, an ordered score, or a yes/no decision. The scorer checks the chosen answer and the probability distribution. For example, a ticket routed to Billing should still go to Billing when that option moves from first to last. A shift in probabilities can reveal sensitivity even when the selected answer stays the same.
| Set | Questions | What it covers |
|---|---|---|
| Core scenarios | 200 | Constructed scenarios for routing, triage, written rules, constraints and near-miss alternatives |
| Public-source questions | 690 | News, emotion, banking intent, spam, science, exams and selected typed decisions |
| Ethical dilemmas | 30 | Descriptive comparison of choices and order sensitivity, with no correct answer assigned |
| Total | 920 | 890 questions with an answer key and 30 descriptive cases |
The core is balanced between English and Brazilian Portuguese. The public-source expansion contains 583 English items and 107 Portuguese items. The 16 reporting areas include operational domains and public-source tasks. Some areas have limited samples.
Evaluation protocol
- Freeze the question set, option-order schedule and scorer. Record the model checkpoint, adapter, settings and environment.
- Present each question 100 times, keeping the context and question fixed. Reorder the options using the same schedule for every model. The full run has 92,000 presentations per model, including repeated orderings used as controls.
- Preserve option keys when moving options. For a binary question, each option appears first 50 times. For a four-level rubric, each level appears in each position 25 times. With larger option sets the schedule balances positions as closely as 100 presentations allow.
- Report exact accuracy, agreement between repetitions, changes to the full probability vector, sensitivity to position and coverage. Invalid responses count against the score. Report latency separately with its measurement conditions.
The 100 presentations of a question are repeated measurements of one item. Uncertainty estimates resample questions with their repetitions kept together.
How scoring works
A local score combines 80% decision quality and 20% reliability. Decision quality combines 65% exact accuracy with 35% probability scoring, using Brier loss for categorical or binary outcomes and ranked probability score for ordered levels. Reliability considers accuracy on the hardest presentations, agreement across repetitions, stability of probabilities, order sensitivity and valid-response coverage.
Version 3 aggregates the covered languages within each area and the 16 areas overall using (mean of the square roots of the scores)². On two equally weighted areas, scores of 50 and 50 combine to 50, while scores of 100 and 0 combine to 25. This gives weak areas greater influence than a simple average would. The resulting SLCMB points are a composite score out of 100.
Model results
| Model | Full-set SLCMB 3.0 points | Source |
|---|---|---|
| Julia 1 | 33.3 | Julia 1.5 model card comparison |
| Julia 1.5 | 46.2 | Julia 1.5 model card |
| Julia 2 | 64.7 | Julia 2 preliminary internal evaluation |
The model-card scores shown here use the full 920-question set reported by each release. Julia 2 also reports 67.7 on the development split. Flash’s paired evaluation uses a separate held-out subset of 222 questions. Keep the dataset revision and split fixed when comparing runs. Julia 1.5 results · Julia 2 and Flash results
Scope and next steps
The benchmark measures performance on a defined set. The operational scenarios are constructed for evaluation, and the public-source labels have varying levels of review. For a production decision, also evaluate with representative data and independently reviewed labels. The ethical cases describe choices under the stated conditions. Latency belongs to the tested hardware and adapter.
As Julia gains capabilities, we’ll add tasks and publish the scoring and split used for each new evaluation.