How to uncover real effects in observational studies without falling into false alarms?

Imagine you want to know if an intervention – say a new school program – affects several outcomes like grades, motivation, and well‑being. In a true experiment you could randomly assign the program to some classes and compare results. In real life, however, we often work with observational data: groups aren’t randomly created, but they are "matched" to make them similar on many characteristics.

The problem? Testing many outcomes at once raises the risk of false discoveries – findings that look significant but are just chance. On top of that, an unmeasured confounder could be silently driving the observed differences.

The new approach offers two key tools. First, it builds "sensitivity sets" for the proportion of true hypotheses within any subset of outcomes the researcher picks. In plain words, it tells you: "Even if there’s hidden bias, at least 70 % of the results you selected are truly caused by the treatment." Second, these sets are valid simultaneously for all possible subsets, letting you explore freely without pre‑selecting which outcomes to test.

To compute the sets, the authors use a sequence of integer programs (a kind of mathematical optimization) together with screening steps that cut down the number of combinations to evaluate. Simulations and a real‑world case – the long‑term impact of childhood abuse – show the method is practical and more robust than traditional analyses. In short, you can now hunt for promising groups of outcomes while keeping high confidence that they reflect real effects, even when hidden factors are at play.

The method rests on a robust inference framework that merges multiple‑hypothesis testing theory with Rosenbaum's sensitivity analysis. For any subset S of hypotheses, the false discovery proportion (FDP) is defined as \(\text{FDP}(S)=\frac{V(S)}{|S|}\), where V(S) counts the true nulls within S. The goal is to construct a confidence interval for the proportion of true hypotheses (1‑FDP) that holds simultaneously for all possible S.

To obtain such intervals, the authors formulate an integer programming problem that maximizes V(S) subject to constraints derived from the sensitivity model. The sensitivity model introduces a parameter \(\Gamma\) that quantifies the maximum odds ratio of treatment assignment between similar but unobserved units. Varying \(\Gamma\) assesses the robustness of discoveries against unmeasured confounders.

The procedure has three main steps: (1) a preliminary screening to discard clearly non‑significant hypotheses, thus shrinking the problem size; (2) solving a sequence of integer programs, each corresponding to an increasing \(\Gamma\) value; (3) aggregating the results to build the sensitivity sets, i.e., confidence intervals for the proportion of true hypotheses in each subset. The algorithm exploits monotonicity: if a subset S is accepted for a given \(\Gamma\), all its subsets are accepted for smaller \(\Gamma\) values.

Simulations reveal that, compared with traditional FDR‑controlling methods, the new approach achieves higher coverage of the true discovery proportion, especially for moderate \(\Gamma\) (1.2‑1.5). In the childhood‑abuse case study, the findings indicate that at least 65 % of the selected variables remain significant even for \(\Gamma=1.3\), suggesting strong long‑term effects not explainable by observed confounders. This makes the method especially valuable in settings where causal inference is challenging and robustness to hidden bias is essential.