Countering the Curse of Dimensionality: Exploring Data-generating Mechanisms Through Participant Observation and Mechanistic Modeling; Epidemiology

Public-health researchers face ethical and practical barriers that often preclude attaining a sufficient degree of experimental control to formally infer causality. Ethical concerns rule out many experiments on human health, and diverse socio-ecologic mechanisms shaping disease outcomes challenge experimental design (Figure part A). Researchers are instead relying on large observational datasets to attain causal understanding, but this requires balancing populations across all potentially relevant variables for an outcome of interest. As the number and interdependence of such variables increase, the size of the required dataset rapidly exceeds plausible levels (“Curse of Dimensionality”; fig. 1 Part B).2J