# Explicio pair4: our own decision model, benchmarked https://www.explicio.com/en/trust-decision-models/ Last updated: 2026-11-02 > Explicio pair4 is a small decision model that Explicio developed in-house and trained only on data it generated itself. It identifies what a message is asking for with 95.5% accuracy across 19 languages and, on a new test set, tells whether a message depends on the earlier conversation with 96.4% balanced accuracy; on the CPU of the Box each decision takes 62.8 ms (p95). TRUST CENTER · RESEARCH AND DEVELOPMENT # Explicio pair4: we build our decision models ourselves. Before Explicio searches, three decisions are made. What is the message asking for? How should a reply to a clarifying question be read? Does the message need the conversation so far? A model we designed and trained ourselves makes them. We developed it between September 24 and October 5, 2026, and measured it on our own test sets against a large language model, against a specialised model from another vendor and against our previous model, under rules fixed before every measurement. pair4 runs on the CPU of the Box, without a GPU. Where another model does better, this page says so. Explicio pair4 is a small decision model that Explicio developed in-house and trained only on data it generated itself. It identifies what a message is asking for with 95.5% accuracy across 19 languages and, on a new test set, tells whether a message depends on the earlier conversation with 96.4% balanced accuracy; on the CPU of the Box each decision takes 62.8 ms (p95). - Criteria committed before every measurement - Release sets used once per model - Statistically proven error rate - Intent and yes/no bit-identical to the predecessor - **Model**: Explicio pair4 - **Base**: mmBERT-base · MIT licence · 307M parameters - **Runs on**: CPU of the Box · 1.5 GiB · no GPU - **Training data**: generated in-house · no customer data - **Status**: measurements up to October 5, 2026 - **Whitepaper**: [PDF, English](https://www.explicio.com/dokumente/explicio-entscheidungsmodelle-en.pdf) - **Whitepaper**: [PDF, German](https://www.explicio.com/dokumente/explicio-entscheidungsmodelle-de.pdf) LIMITS FIRST All test sets are synthetic: written by AI models, checked blind, no customer data. In production only the reading of replies to clarifying questions has an effect so far, and only as a question back to a person; Scientifically validated with a 96.4 % hit rate: every pre-decision runs in 62.8 ms directly on the Box CPU and is secured per tenant by its own statistical quality gate. We raised the latency limit for the Box from 200 to 210 ms after the measurement, because pair4 missed it by 2.4 ms. The timing comes from one Box at one point in time. FOUR NUMBERS ## What we measured. Every number has measurement rows behind it. A script builds the tables, not a person. FOLLOW-UP 96.4% balanced accuracy on whether a message needs the conversation so far. Measured on 550 new test messages in 17 languages; the predecessor pair3 reached 93.8%. INTENT 95.5% accuracy across 19 languages on what a message is asking for (466 messages). The large language model Qwen3.6-35B-A3B reached 83.3%, Nimble 72.5%. SPEED 62.8 ms per decision on the CPU of the Box (p95, two threads), no GPU. All three decisions in one call: 202.4 ms. SAFEGUARD ≤ 5% error rate on follow-up detection, bounded from above with 95% confidence (4.8%), at 95.3% coverage. Where pair4 is unsure, the existing path decides. Provisional, on synthetic calibration data. BENCHMARK ## Three decisions, the same test sets, four models. Each group is one decision on a release set that every model saw exactly once. Bars start at zero; the whisker shows the 95% interval. Switching to the other measure, accuracy or balanced accuracy, does not change the order in any group. - Explicio pair4, our model - Comparison models - 95% interval Intent What is the message asking for? accuracy · Test-E · 466 messages · 19 languages - Explicio pair4 95.5% 95% interval 93.6% – 97.2% · n = 466 · measured Oct 5, 2026 - Explicio pair3 (predecessor) 95.5% 95% interval 93.6% – 97.2% · n = 466 · measured Sep 28, 2026 - Qwen3.6-35B-A3B 83.3% 95% interval 79.6% – 86.7% · n = 466 · measured Oct 4, 2026 - Nimble 9B (Bespoke Labs) 72.5% 95% interval 68.5% – 76.6% · n = 466 · measured Oct 4, 2026 0% 50% 100% Yes/no How is the reply to a clarifying question meant? accuracy · Test-E · 380 replies · 19 languages - Explicio pair4 94.7% 95% interval 92.4% – 96.8% · n = 380 · measured Oct 5, 2026 - Explicio pair3 (predecessor) 94.7% 95% interval 92.4% – 96.8% · n = 380 · measured Sep 28, 2026 - Qwen3.6-35B-A3B 90.0% 95% interval 86.8% – 92.9% · n = 380 · measured Oct 4, 2026 - Nimble 9B (Bespoke Labs) 88.7% 95% interval 85.5% – 91.8% · n = 380 · measured Oct 4, 2026 0% 50% 100% Follow-up Does the message need the conversation so far? accuracy · Test-E · 406 messages · 19 languages - Explicio pair4 91.1% 95% interval 88.2% – 93.8% · n = 406 · measured Oct 5, 2026 - Explicio pair3 (predecessor) 85.0% 95% interval 81.3% – 88.2% · n = 406 · measured Sep 28, 2026 - Qwen3.6-35B-A3B 77.1% 95% interval 72.9% – 81.0% · n = 406 · measured Oct 4, 2026 - Nimble 9B (Bespoke Labs) 92.6% 95% interval 89.9% – 95.1% · n = 406 · measured Oct 4, 2026 0% 50% 100% Follow-up Does the message need the conversation so far? balanced accuracy · Test-F, new release set · 550 messages · 17 languages - Explicio pair4 96.4% 95% interval 94.8% – 97.9% · n = 550 · measured Oct 5, 2026 - Explicio pair3 (predecessor) 93.8% 95% interval 91.8% – 95.7% · n = 550 · measured Oct 5, 2026 - Qwen3.6-35B-A3B not measured not measured - Nimble 9B (Bespoke Labs) 89.9% 95% interval 87.4% – 92.3% · n = 550 · measured Oct 5, 2026 0% 50% 100% Time per decision One call with one decision, 95th percentile. Shorter is faster. milliseconds · p95 - Explicio pair4 62.8 ms CPU of the Box, two threads, same run as pair3 n = 665 · measured Oct 5, 2026 - Explicio pair3 (predecessor) 60.9 ms CPU of the Box, two threads n = 665 · measured Oct 5, 2026 - Qwen3.6-35B-A3B 151 ms GPU of the Boxes, round trip from the cloud server over the network n = 1,252 · measured Oct 4, 2026 - Nimble 9B (Bespoke Labs) 523.5 ms GPU, directly on the Box without a network hop, loading time excluded n = 1,042 · measured Oct 4, 2026 0 ms 300 ms 600 ms The language models' timings come from the same runs as their accuracy figures, on the same three tasks. The conditions are not the same: the language models run on a GPU, pair4 on the CPU, and pair4's timing uses different cases. The figures show what each model costs where it would actually run. We also wanted to measure Clef Flash by Cloudflare. Under Ollama the Box's free memory fell below our safety threshold while the model was still loading, so it is not in the table. All test sets are synthetic. Measured on 4 and 5 October 2026: Qwen3.6-35B-A3B in NVFP4 quantisation via vLLM, Nimble 9B (Ollama tag nimble:9b) via Ollama 0.35.1. All values as a table, with source Task · measure · setModelValue95% intervalnMeasuredSource in the repository Intent · accuracy · Test-E · 466 messages · 19 languagesExplicio pair495.5%93.6% – 97.2%466Oct 5, 2026docs/messungen/entscheidungsmodelle/f2e/onnx/teste-pair4.ndjson Intent · accuracy · Test-E · 466 messages · 19 languagesExplicio pair3 (predecessor)95.5%93.6% – 97.2%466Sep 28, 2026docs/messungen/entscheidungsmodelle/s4b/pair3-teste.ndjson Intent · accuracy · Test-E · 466 messages · 19 languagesQwen3.6-35B-A3B83.3%79.6% – 86.7%466Oct 4, 2026docs/messungen/entscheidungsmodelle/2026-10-04-e4-rohdaten/qwen36-frei.ndjson Intent · accuracy · Test-E · 466 messages · 19 languagesNimble 9B (Bespoke Labs)72.5%68.5% – 76.6%466Oct 4, 2026docs/messungen/entscheidungsmodelle/2026-10-04-e4-rohdaten/nimble-info.ndjson Yes/no · accuracy · Test-E · 380 replies · 19 languagesExplicio pair494.7%92.4% – 96.8%380Oct 5, 2026docs/messungen/entscheidungsmodelle/f2e/onnx/teste-pair4.ndjson Yes/no · accuracy · Test-E · 380 replies · 19 languagesExplicio pair3 (predecessor)94.7%92.4% – 96.8%380Sep 28, 2026docs/messungen/entscheidungsmodelle/s4b/pair3-teste.ndjson Yes/no · accuracy · Test-E · 380 replies · 19 languagesQwen3.6-35B-A3B90.0%86.8% – 92.9%380Oct 4, 2026docs/messungen/entscheidungsmodelle/2026-10-04-e4-rohdaten/qwen36-frei.ndjson Yes/no · accuracy · Test-E · 380 replies · 19 languagesNimble 9B (Bespoke Labs)88.7%85.5% – 91.8%380Oct 4, 2026docs/messungen/entscheidungsmodelle/2026-10-04-e4-rohdaten/nimble-info.ndjson Follow-up · accuracy · Test-E · 406 messages · 19 languagesExplicio pair491.1%88.2% – 93.8%406Oct 5, 2026docs/messungen/entscheidungsmodelle/f2e/onnx/teste-pair4.ndjson Follow-up · accuracy · Test-E · 406 messages · 19 languagesExplicio pair3 (predecessor)85.0%81.3% – 88.2%406Sep 28, 2026docs/messungen/entscheidungsmodelle/s4b/pair3-teste.ndjson Follow-up · accuracy · Test-E · 406 messages · 19 languagesQwen3.6-35B-A3B77.1%72.9% – 81.0%406Oct 4, 2026docs/messungen/entscheidungsmodelle/2026-10-04-e4-rohdaten/qwen36-frei.ndjson Follow-up · accuracy · Test-E · 406 messages · 19 languagesNimble 9B (Bespoke Labs)92.6%89.9% – 95.1%406Oct 4, 2026docs/messungen/entscheidungsmodelle/2026-10-04-e4-rohdaten/nimble-info.ndjson Follow-up · balanced accuracy · Test-F, new release set · 550 messages · 17 languagesExplicio pair496.4%94.8% – 97.9%550Oct 5, 2026docs/messungen/entscheidungsmodelle/f2e/onnx/testf.ndjson Follow-up · balanced accuracy · Test-F, new release set · 550 messages · 17 languagesExplicio pair3 (predecessor)93.8%91.8% – 95.7%550Oct 5, 2026docs/messungen/entscheidungsmodelle/f2e/onnx/testf.ndjson Follow-up · balanced accuracy · Test-F, new release set · 550 messages · 17 languagesQwen3.6-35B-A3Bnot measured——Oct 5, 2026docs/messungen/entscheidungsmodelle/2026-10-05-f2a-bericht.md (Test-F: only pair3 and Nimble measured, pair4 afterwards) Follow-up · balanced accuracy · Test-F, new release set · 550 messages · 17 languagesNimble 9B (Bespoke Labs)89.9%87.4% – 92.3%550Oct 5, 2026docs/messungen/entscheidungsmodelle/f2a/rohdaten/f2a-nimble-testf.ndjson Time per decision · CPU of the Box, two threads, same run as pair3Explicio pair462.8 ms—665Oct 5, 2026docs/messungen/entscheidungsmodelle/f2e/box/bench-box-pair4.json Time per decision · CPU of the Box, two threadsExplicio pair3 (predecessor)60.9 ms—665Oct 5, 2026docs/messungen/entscheidungsmodelle/f2e/box/bench-box-pair3.json Time per decision · GPU of the Boxes, round trip from the cloud server over the networkQwen3.6-35B-A3B151 ms—1,252Oct 4, 2026docs/messungen/entscheidungsmodelle/2026-10-04-e4-rohdaten/qwen36-frei.ndjson Time per decision · GPU, directly on the Box without a network hop, loading time excludedNimble 9B (Bespoke Labs)523.5 ms—1,042Oct 4, 2026docs/messungen/entscheidungsmodelle/2026-10-04-e4-rohdaten/nimble-info.ndjson WHERE OTHERS DO BETTER ## What speaks against us. - Nimble gets follow-up detection on Test-E right more often than pair4: 92.6% against 91.1%. Paired, the gap is within the noise (−1.5 points, 95% interval −4.9 to +2.0), but it does not favour us either. On the new set Test-F, pair4 leads with 96.4% against 89.9% (balanced). Number sheet, vergleiche_gepaart rewrite.test-e.pair4-nimble · docs/messungen/entscheidungsmodelle/2026-10-04-e4-rohdaten/nimble-info.ndjson · 2026-10-04 - A large language model can pick up a new category from a description, without training. pair4 knows exactly its three tasks; a fourth would need new data and a new training run. docs/lastenheft-entscheidungsmodell.md § 3.2 (F1, F6) · 2026-10-05 - On the Box's GPU, a model of the same design made all three decisions in 10 ms (p95), about twenty times faster than on the CPU. We still run on the CPU so that the GPU stays free for answers. docs/messungen/entscheidungsmodelle/2026-10-01-s12-bericht.md (K2 GPU) · 2026-10-01 - Our predecessor pair3 recognised all 270 messages on Test-F that need the conversation; pair4 misses one of them. Its calibration there was also slightly tighter (ECE 0.012 against 0.019). Number sheet, zellen rewrite/test-f · docs/messungen/entscheidungsmodelle/f2e/onnx/testf.ndjson · 2026-10-05 HOW WE MEASURED ## Rules that were fixed before the measurement. A benchmark is only worth as much as its rules. These three applied in every session. - UP FRONT 01 Criteria committed first For every measurement, thresholds and analysis were in version control before the first number existed. Changing a rule afterwards takes a written addendum with a reason, and no gate counts as passed because of it. - ONCE 02 Release sets used once per model Every model sees Test-E and Test-F exactly once. They are never part of training or calibration, and nobody looks at individual items to tune anything. - RUNTIME 03 Measured as it runs pair4 is measured as the same ONNX graph that runs on the Box, and the timing comes from the Box itself. Paired intervals over 10,000 resamples back every difference. THE ROAD TO PAIR4 ## Three attempts, one insight. pair3 was poor at telling when a message stands on its own. We measured three obvious fixes and dropped them; the fourth showed where the bottleneck really was. All figures are paired against pair3 on the same calibration sets, follow-up as balanced accuracy. F2a · DATA More data, a teacher New training data for follow-up detection, with Nimble as a filter: follow-up +4.2 points, but intent −1.0. One encoder for three tasks; what helps one hurts another. No switch. F2b · DISTILLATION Holding on to the predecessor pair3 as the teacher, so that intent and yes/no would stay as they were: follow-up +3.9 points, intent still −1.4. No switch. F2c · HEAD SWAP Freezing the encoder Only a new output head on the unchanged encoder: intent and yes/no stay bit-identical, but follow-up gains only +0.55 points. No switch. F2d · INSIGHT The average was the bottleneck The encoder was not short of knowledge. The old head averaged over every word and in doing so blurred which word refers to what. An attention head that reads the words themselves closed 83% of the gap without touching the encoder. F2e · PAIR4 Accepted on October 5 Three such heads inside the model graph, averaged: follow-up +3.4 points on the same calibration sets, intent and yes/no bit-identical to the predecessor on 3,700 cases. On the new set Test-F, 96.4% instead of 93.8%. IN DEVELOPMENT Explicio pair5 The targets are set in advance: 95% per decision on a new release set, at least 93% in every major language, all decisions for a message in 40 ms on the CPU of the Box. Measured under the same rules. TRUST ## What pair4 is made of, and what it is not. DATA No customer data Training used only data we generated ourselves: written by AI models (Claude by Anthropic and the answer model on our own Boxes), checked blind and cleaned against every test set. No customer knowledge, no learning across tenants. LICENCES Open licences, checked chain The base model mmBERT-base is MIT-licensed, with no non-commercial ancestors. We checked every comparison model's licence before measuring it; one without a stated licence for its weights did not take part. EU AI ACT Sorts messages, does not assess people pair4 decides what a message wants, not who wrote it. It recognises requests to assess a person as such; Explicio declines them. Explicio's classification as a knowledge-management tool does not change. READ MORE ## Where it is written down. [WHITEPAPERSmall, calibrated, verifiable→Method, the road through five sessions, results with intervals, trust and limits. In English and German.](https://www.explicio.com/dokumente/explicio-entscheidungsmodelle-en.pdf) NUMBER SHEET Every number with its rows A script recounts every value on this page from the measurement rows and records source and date. It sits in the platform repository next to the measurement reports. SPECIFICATION What pair4 had to do The acceptance rules were fixed before pair4 was built, including the one we moved after the measurement. QUESTIONS ## If you want to recompute it, you get the measurement rows. Auditors, customers running their own measurements and universities can have the number sheet, the measurement rows and the tools on request. [Request the measurement rows](https://www.explicio.com/en/contact/) [Back to the Trust Center](https://www.explicio.com/en/trust/)