Commentary on “Nonparametric identification is not enough, but randomized controlled trials are”: Statistical considerations for generating reliable evidence across a spectrum of studies that increasingly involve real-world elements

Judea Pearl, quoted in Pearl and Mackenzie (2008), stated that “once we have understood why [randomized controlled trials] RCTs work, there is no need to put them on a pedestal and treat them as the gold standard of causal analysis, which all other methods should emulate.” In Aronow et al. (2024), this claim is refuted, drawing on results of Robins and Ritov (1997). The argument is made that statistical estimation and inference tend to be fundamentally more difficult in observational studies than in randomized controlled trials, even when all confounders are observed and measured without error. We congratulate the authors for raising this highly timely, interesting discussion and welcome this opportunity to join this important debate. In this commentary, we focus on what it takes to generate reliable evidence across a spectrum of studies that increasingly involve real-world elements and less control over design. A related question is whether, along this spectrum of studies, the reliability of evidence generated by a statistical analysis decreases. We claim that this is not the case, but that the challenge for the appropriate statistical method increases, requiring sophisticated and careful execution.