Correlated Perturbations and Effective Sample Size in Robustness Benchmarks

Abstract

In modern robustness benchmarking of artificial intelligence foundation models, evaluating stability under linguistic perturbations is central to certifying clinical deployment safety. Typically, researchers generate multiple semantically equivalent paraphrases for each evaluation instance and compute aggregate flip rates across the resulting dataset. However, standard statistical reporting routines routinely treat perturbed evaluation pairs as independent observations, ignoring the profound intraclass correlation that arises when multiple perturbations are evaluated on the exact same underlying base instance. In this paper, we establish a rigorous statistical and mathematical framework for analyzing correlated perturbations in clinical and natural language robustness benchmarks. We prove that treating clustered perturbations as independent trials produces severe variance underestimation, deflating standard errors by up to 75% and generating falsely narrow confidence intervals that exaggerate benchmark precision. We formalize the exact effective sample size under exchangeable and autoregressive perturbation correlation models, derive clustered variance estimators, and prove that question-level flip rates and pair-level discordance rates estimate fundamentally distinct statistical quantities. Through an extensive Monte Carlo simulation study spanning 100,000 synthetic trials and an empirical audit of multi-center radiographic benchmarks across four foundational vision-language architectures, we demonstrate that a nominal benchmark of 1,000 paired questions often yields an effective sample size of fewer than 180 independent units. We provide exact design recommendations, sample size inflation factors, and hypothesis testing protocols to ensure rigorous, statistically sound robustness audits in high-stakes artificial intelligence deployments.