Political Analysis Advance Access published online on January 6, 2009
Political Analysis, doi:10.1093/pan/mpn012
| ||||||||||||||||||||||||||||||||||||||||||||||||||||
Treatment Spillover Effects across Survey Experiments
Department of Political Science and Institute for Legal, Legislative, and Policy Studies in the Center for State Policy and Leadership, University of Illinois at Springfield, Mail Stop PAC 350, Springfield, IL 62703
e-mail: jtran8{at}uis.edu (corresponding author)
Department of Political Science, Michigan State University, 303 South Kedzie Hall, East Lansing, MI 48824
e-mail: leedan{at}msu.edu
Department of Political Science, Duke University, 326 Perkins Library, Box 90204, Durham, NC 27708-0204
e-mail: aldrich{at}duke.edu
Embedding experiments within surveys has reinvigorated survey research. Several survey experiments are generally embedded within a survey, and analysts treat each of these experiments as self-contained. We investigate whether experiments are self-contained or if earlier treatments affect later experiments, which we call "experimental spillover." We consider two types of bias that might be introduced by spillover: mean and inference biases. Using a simple procedure, we test for experimental spillover in two data sets: the 1991 Race and Politics Survey and a survey containing several experiments pertaining to foreign policy attitudes. We find some evidence of spillover and suggest solutions to avoid bias.
Authors’ note: Replication materials and an Appendix are available on the Political Analysis Web site. We would like to thank Richard Herrmann for sharing data and helping us understand the data's structure. We also would like to thank Paul Sniderman and Eric Lawrence for their advice, as well as presentation participants at Public Institutions and Public Choice at Duke University, University of Illinois at Urbana-Champaign, and Essex University for helpful comments on earlier drafts of this paper. Conflict of interest statement. None declared.