Put together, this is evidence that the vendors building these tools understand the coverage-versus-accuracy gap well enough to design around it, even while their marketing pages talk about " 100 %" without qualification.
STUDY FOUND SOMETHING SHARPER THAN " MISSES CONTEXT "
Vendor documentation and practitioner threads are useful, but a 2026 paper accepted into the Association for Computational Linguistics( ACL)' s Findings track goes further.
Researchers evaluated 18 large language models( LLMs) scoring 3,000 real contact center transcripts, then tested what happened when they held the substance of a call constant and changed something that shouldn ' t matter to the score. Namely a note implying the agent ' s identity, a customer ' s account tier, or a snippet of that agent ' s prior coaching history.
The result: pass / fail judgments flipped between 5.4 % and 13 % of the time across models, purely from those irrelevant changes.
The single biggest trigger was priming the model with coaching-notes history, which pushed the flip rate as high as 16.4 %.
Communications style differences, direct versus indirect phrasing, formal versus casual tone, moved scores less( in the 7 % range), which the researchers took as a sign that the models were comparatively better at separating what someone meant from how they phrased it.
The same paper reports raw accuracy against human-annotated ground truth across those 18 models, ranging from 65.94 % for one open-source model up to 93.84 % for the strongest one tested, with no consistent split between objective and subjective criteria.
32 CONTACT CENTER PIPELINE
Worth disclosing directly: the paper ' s authors are researchers at Observe. AI, one of the vendors named earlier in this piece. That doesn ' t make the finding less real, since it went through the same peer review as any other ACL submission, but it isn ' t independent industry research either.
What this adds to the practitioner accounts above is a mechanism, not just anecdotes.
A score can shift because of who reviewed an agent ' s file last week, not because of anything said on the call actually being scored. That ' s a more specific problem than " AI misses empathy ": it ' s AI treating something irrelevant to the interaction, like who last reviewed the agent ' s coaching file or what tier the customer ' s account sits on, as if it were evidence of how the call actually went.
WHAT A PROGRAM THAT AC- COUNTS FOR THIS LOOKS LIKE
The practical version of this isn ' t complicated, but it does require treating your scorecard as something to audit rather than something to hand over to automation wholesale.
1. Start by sorting your existing scorecard criteria into two buckets, not by topic but by what it takes to score them correctly.
• Bucket 1, pattern-matchable criteria( disclosures, verification, mandated language).
• Bucket 2, context-dependent criteria. Everything else, anything where the right answer depends on understanding the specific situation rather than checking for a phrase, and which needs a human-in-the-loop before it gets treated as a final grade.
2. Hand over Bucket 1 to AI with reasonable confidence, and expect it to be fast and consistent at scale in a way no human team reviewing 1 %-2 % of calls ever could be.
That includes the two categories from the original framing, empathy and judgment calls on workarounds, but it also includes edge cases inside criteria that look objective on paper. Like a compliance step that had a legitimate
AI QUALITY SCORING
BEFORE TRUSTING AI OUTPUT... RUN IT IN PARALLEL AGAINST HUMAN SCORES FOR A STRETCH...
exception, where a script deviation was the right call given what the customer said.
Before trusting AI output for trend reporting or coaching decisions, run it in parallel against human scores for a stretch, long enough to see where the two disagree and why.
There ' s no single industry-standard length for this, and be skeptical of anyone who quotes you a precise week count as an established benchmark; it isn ' t one. What matters is the discipline of comparing before committing, not the specific calendar length.
The comparison should check more than raw agreement, too. Given what the ACL research found about scores shifting based on irrelevant context, it ' s worth deliberately testing whether the same call, stripped of anything about who the agent is or what happened in their last coaching session, gets scored the same way twice.
Indeed, one QA team from that same Reddit thread described the opposite failure mode; after switching to AI-driven scoring, their pass rate dropped so far that management redefined the passing threshold from 94 % down to 60 %.
This occurred not because performance had actually declined, but because nobody had validated the new scoring before making it the number that mattered.
That ' s the actual risk. Not that AI QA tools are bad at their job. It ' s that " 100 % coverage " quietly becomes " 100 % of scores treated as final," without anyone deciding that on purpose.
Owen Zhang is the editor of CommsAdvisor. com, where he tests and compares contact center, help desk, and customer support software for buyers evaluating vendors like Zendesk, Freshdesk, and Aircall. He previously worked as a software engineer, including a stint at Huawei, before moving into independent B2B software reviews.