Connect’s Benchmarking report finds orchestrated SLMs outperform frontier LLMs across highly regulated enterprise domains
Connect, a leading technology-agnostic systems integrator specialising in AI-led contact centre solutions, has today published a technical benchmarking report evaluating the deployment viability of compact Small Language Models (SLMs) across enterprise workloads. Testing covered representative enterprise datasets across three core sectors, namely finance, healthcare and insurance.
To address whether compact, local models can meet stringent operational requirements in highly regulated sectors, Connect conducted the benchmark study to evaluate real-world performance using the company’s proprietary 4-billion-parameter model stack, Connect AI Elly.
Researchers evaluated 19 prominent models across 52 professional domains over sequential editing workflows, and found that even frontier AI models corrupt or lose an average of 25% of document content over extended, multi-turn interactions – after just 20 sequential interactions, the most advanced models corrupted a quarter of the document’s content.
The benchmark study evaluated three distinct deployment patterns across realistic enterprise tasks:
- Baseline SLM: A single 4B-parameter model running without domain tuning or automated task routing, representing a standard, off-the-shelf compact deployment.
- Domain-Tuned SLM: The same 4B model fine-tuned specifically on domain-specific terminology, task formats, and expected response patterns.
- Orchestrated SLM Ensemble: A multi-agent routing architecture that decomposes complex workflows across specialised micro-models and SLMs, separating duties across a domain router, entity extractor, summariser, retriever/re-ranker, verifier model, and safety classifier.
While frontier Large Language Models (LLMs) excel at open-ended reasoning and highly fluid dialogue, Alfredo Gemma, AI Solution Director at Connect, explains that asking a single generalist model to perform intent classification, numerical extraction, contextual summarisation, and safety checks simultaneously introduces operational risks.
Commenting on the report, he said: “As organisations move beyond experimentation and start deploying AI into operational workflows, there is growing demand for models that are smaller, faster, more cost-effective, and easier to run in controlled environments. The benchmark report reveals that orchestrating multiple domain-specific SLM components improves task accuracy, factual grounding, and summarisation quality by ~13 percentage points overall compared to single-model architectures, outperforming many foundation LLMs.”
The report found that domain tuning delivered an immediate performance boost, increasing overall accuracy from 72.67% to 81.20%, with particularly strong gains in insurance sentiment and medical QA.
The Orchestrated SLM Ensemble performed best overall, achieving an average score of 85.42% – a 12.75 percentage-point improvement over the baseline. Orchestration also drove significant gains in evidence recall and financial and clinical summarisation.
In cross-domain testing, the Orchestrated SLM Stack outperformed several leading foundation models, including ChatGPT, Gemini, Amazon Nova, Mixtral-8x22B and Qwen3-14B, while also exceeding Claude in key healthcare and insurance tasks.
“For contact centre leaders and enterprise architects, the choice isn’t simply between large and small models; it’s about finding the right model for the right task,” explains Gemma. “Connect’s benchmark report shows that orchestrating specialised models can deliver the domain-specific performance enterprises need, while providing greater control over cost, data and risk.
“By combining micro models for focused tasks such as entity extraction with domain-tuned SLMs for more complex workflows, organisations can reduce unnecessary cloud processing and reliance on a single vendor. Running this approach on local, sovereign infrastructure also brings greater cost predictability, keeps sensitive data within the organisation’s control and enables a configurable verification process to improve transparency and reduce the risk of hallucinations.
“Ultimately, bigger doesn’t automatically mean better. A carefully orchestrated, hybrid approach can give enterprises the performance they need while improving data sovereignty, explainability and return on their AI investment,” Gemma concluded.
To download the Benchmark report in full, please visit: https://www.weconnect.tech/orchestrated-model-beats-size-benchmark-report/