How to Pick the Most Important Variables with the φ‑Test?

Imagine you have an AI model that predicts house prices, but you don’t know which features (rooms, distance to downtown, etc.) truly matter. The φ‑test is a new method that merges SHAP explanations—already popular for assigning importance to each feature—with selective inference, a statistical tool that rigorously controls significance after variable selection.

Starting from a trained model and an evaluation dataset, the φ‑test first screens out low‑impact features using SHAP values. Then it builds a linear surrogate model only with the remaining variables, applying a selection rule that yields valid p‑values and confidence intervals even after the selection step.

The output is a global table that, for each retained feature, shows: (i) a SHAP‑based global score, (ii) the surrogate coefficient, and (iii) statistical inference metrics. Experiments on real tabular regression tasks with tree‑based and neural networks show that φ‑test preserves most of the original model’s predictive power while using just a few features, and the selected sets stay stable across resamples and model types. In short, φ‑test acts as a practical global explanation layer, linking SHAP‑driven importance with classical statistical inference.

The φ‑test rests on two theoretical pillars: Shapley attributions and selective inference. The former, introduced by Lloyd Shapley in 1953, provides a fair contribution value for each feature by averaging marginal effects over all possible feature insertion orders. In practice, SHAP estimates how a prediction would change if a feature were added or removed, accounting for interactions with other variables.

Selective inference, on the other hand, tackles the classic "p‑value inflation" problem that arises when testing hypotheses after data-driven selection. Traditional statistical tests assume hypotheses are fixed a priori; if variables are selected first, resulting p‑values become overly optimistic. Selective inference corrects this bias by computing the conditional distribution of the estimator given the selection process.

In the φ‑test, SHAP values guide an initial screening phase: thresholds (e.g., the 95th percentile) discard features with negligible contributions. Remaining features are fed into a linear surrogate model, typically a LASSO or ridge regression, with a selection rule (e.g., the knockoff filter or Lee et al.'s post‑selection inference). This rule is designed to have a closed‑form selective inference, allowing exact p‑values and confidence intervals.

The output is a global table where each row contains: (1) the mean SHAP value (global importance), (2) the linear model coefficient β (indicating direction and magnitude of effect), and (3) the post‑selection p‑value. Confidence intervals are built by conditioning on the selection event, guaranteeing nominal coverage.

Experiments on datasets such as "California Housing" or "Bike Sharing" show that φ‑test can reduce the number of features from dozens to a handful while preserving similar R² to the original model. Moreover, feature‑selection stability is assessed via bootstrapping: the frequency with which a feature appears in the top‑k is high (>80%). This suggests φ‑test is not only interpretable but also robust to data variations and backbone choices.

Finally, φ‑test can be plugged into existing machine‑learning pipelines: after training a black‑box model, φ‑test provides a global explanation layer and, if desired, a slimmer model built on the selected features, lowering computational costs and enhancing transparency.