Political Analysis Advance Access published online on February 12, 2008
Political Analysis, doi:10.1093/pan/mpm032
A Comment on Diagnostic Tools for Counterfactual Inference
Department of Political Science, Yale University, PO Box 208301, New Haven, CT 06520
London School of Economics, Department of Economics, Houghton Street, London WC2A 2AE, UK
e-mail: nicholas.sambanis{at}yale.edu (corresponding author)
e-mail: a.michaelides{at}lse.ac.uk
We evaluate two diagnostic tools used to determine if counterfactual analysis requires extrapolation. Counterfactuals based on extrapolation are model dependent and might not support empirically valid inferences. The diagnostics help researchers identify those counterfactual "what if" questions that are empirically plausible. We show, through simple Monte Carlo experiments, that these diagnostics will often detect extrapolation, suggesting that there is a risk of biased counterfactual inference when there is no such risk of extrapolation bias in the data. This is because the diagnostics are affected by what we call the n/k problem: as the number of data points relative to the number of explanatory variables decreases, the diagnostics are more likely to detect the risk of extrapolation bias even when such risk does not exist. We conclude that the diagnostics provide too severe a test for many data sets used in political science.
Authors' note: We thank Komei Fukuda, Don Green, Alan Gerber, and Jasjeet Sekhon for their generous help, Mike Kane for assistance with R programming, and five anonymous referees for constructive comments.