Hüseyin Atakan KESKİN, Mehmet Levent YILMAZ

REVIEW OF EXPERIMENTAL DESIGNS AND METHODS IN ECONOMICS OF EDUCATION RESEARCH

The field of economics of education has been receiving constant attention with the advent of growth theories stating how education can produce sustainable long-run economic growth and increase people’s skills (Hanushek & Woessmann, 2015; Mincer, 1974). Despite its universal relevance, educational research has not been placed on the same scale with fields such as medicine due to seldom usage of robust quantitative research and the dearth of causal inferences (Creemers, Kyriakidēs, & Sammons, 2010). Given this, leading countries in educational research such as the United States (Scimago Journal & Country Rank, 2018) have been initiating “repeated calls for education policy to rely on a foundation of scientifically based research” (Angrist, 2003, para. 1) to nudge the field of education towards using rigorous and innovative methodological methods and experiments (Murnane & Willett, 2011). Using an experimental method is most suitable when the research aims to test the impact of intervention within the respected field of research (Beach & Pedersen, 2016). Henceforward, this paper addresses the opportunities and challenges of using experimental methods in educational interventions, particularly randomized control trials (RCT) and quasi-experiments that test the impact of financial incentives to increase student outcomes. The first section is an overview of experimental designs, followed by sections delineating on RCTs and quasi-experiments, and discussing empirical studies that employ such methods. It should be noted that this paper argues in favor of neither quantitative nor qualitative research methods as both methods can produce quality research if implemented rigorously (Lodico, Spaulding, & Voegtle, 2010).

Keywords:

Economics of Education, Research Methods, Experimental Research Design, Educational Outcomes, Randomized Controlled Trial Quasi Experiments, Financial Incentives,

PDF

___

ANGRIST, J. (2003, Summer). Randomized Trials and Quasi-Experiments in Education Research. Retrieved March 11, 2019, from https://www.nber.org/reporter/summer03/angrist.html
ANGRIST, J. D. (2004). American Education Research Changes Tack. Oxford Review of Economic Policy, 20(2), 198-212. doi:10.1093/oxrep/grh011
ANGRIST, J., BETTINGER, E., BLOOM, E., KING, E., & KREMER, M. (2001). Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment. THE AMERICAN ECONOMIC REVIEW, 1535-1558. doi:10.3386/w8343
ANGRIST, J., & Lavy, V. (2002). The Effect of High School Matriculation Awards: Evidence from Randomized Trials. American Economic Review. doi:10.3386/w9389
ATTIA, A. (2005). Bias in RCTs: Confounders, selection bias and allocation concealment. Middle East Fertility Society Journal, 10(3), 258-261. Retrieved March 14, 2019.
BEACH, D., & PEDERSEN, R. B. (2016). Causal case study methods foundations and guidelines for comparing, matching and tracing. Ann Arbor: University of Michigan Press.
BESLEY, T. (2009). Assessing the quality of educational research in higher education: International perspectives. Rotterdam, Netherlands: Sense.
BRADSHAW, C. P., ZMUDA, J. H., KELLAM, S. G., & Ialongo, N. S. (2009). Longitudinal impact of two universal preventive interventions in first grade on educational outcomes in high school. Journal of Educational Psychology, 101(4), 926-937. doi:10.1037/a0016586
COHEN, L., MANION, L., & MORRISON, K. (2018). Research methods in education. New York: Routledge.
CONNOLLY, P., BIGGART, A., MILLER, S., OHARE, L., & THURSTON, A. (2017). Using randomised controlled trials in education. Thousand Oaks, CA: SAGE.
CONNOLLY, P., KEENAN, C., & URBANSKA, K. (2018). The trials of evidence-based practice in education: A systematic review of randomised controlled trials in education research 1980–2016. Educational Research, 60(3), 276-291. doi:10.1080/00131881.2018.1493353
COOK, T. D., & CAMPBELL, D. T. (1979). Quasi-experimentation: Design and analysis for field settings. Chicago: Rand McNally.
COWEN, N., VIRK, B., MASCARENHAS-KEYES, S., & CARTWRIGHT, N. (2017). Randomized Controlled Trials: How Can We Know “What Works”? Critical Review, 29(3), 265-292. doi:10.1080/08913811.2017.1395223
CREEMERS, B. P., KYRIAKIDĒS, L., & SAMMONS, P. (2010). Methodological advances in educational effectiveness research. Abingdon, Oxon, England: Routledge.
ECCLES, M., GRIMSHAW, J., CAMPBELL, M., & RAMSAY, C. (2003). Research designs for studies evaluating the effectiveness of change and improvement strategies. Quality and Safety in Health Care, 12(1), 47-52. doi:10.1136/qhc.12.1.47
EVERY-PALMER, S., & HOWICK, J. (2014). How evidence-based medicine is failing due to biased trials and selective publication. Journal of Evaluation in Clinical Practice, 20(6), 908-914. doi:10.1111/jep.12147
FRIEDMAN, M. (1968). Capitalism and freedom. Chicago, Ill: University of Chicago Press.
GORRALL, B., CURTIS, J., LITTLE, T., & PANKO, P. (2015). Alternatives to Randomized Control Trials: A Review of Three Quasi-experimental Designs for Causal Inference. Actualidades En Psicología, 29(119), 19-27. doi:10.15517/ap.v29i119.18810
GRAY, D. E. (2018). Doing research in the real world. London: SAGE Publications.
HANUSHEK, E. A., & WOESSMANN, L. (2015). The knowledge capital of nations: Education and the economics of growth. Cambridge: The MIT Press.
HOWELL, W. G., & PETERSON, P. E. (2006). The education gap: Vouchers and urban schools. Washington (D.C.): Brookings Institution Press.
HOWELL, W. G., & PETERSON, P. E. (2018, SEPTEMBER 10). The Education Gap. Retrieved March 14, 2019, From https://www.brookings.edu/book/the-education-gap/
HUTCHISON, D., & STYLES, B. (2010). A Guide To Running Randomised Controlled Trials For Educational Researchers. Slough: NFER.
KABISCH, M., RUCKES, C., SEIBERT-GRAFE, M., & BLETTNER, M. (2011). Randomized controlled trials: part 17 of a series on evaluation of scientific publications. Deutsches Arzteblatt international, 108(39), 663-8.
KAHAN, B. C., REHAL, S., & CRO, S. (2015). Risk of selection bias in randomised trials. Trials, 16(1). doi:10.1186/s13063-015-0920-x
KENDALL, J. M. (2003). Designing a research project: Randomised controlled trials and their principles. Emergency Medicine Journal, 20(2), 164-168. doi:10.1136/emj.20.2.164
LODICO, M. G., SPAULDING, D. T., & VOEGTLE, K. H. (2010). Methods in Educational Research: From Theory to Practice. San Francisco: Jossey-Bass.
MARTIN, G. H. (2013, November 10). Randomized Control Trials and Confounding. Retrieved March 14, 2019, from https://www.youtube.com/watch?v=7ybuE39BpQ8
MINCER, J. (1974). Schooling, Experience And Earnings. New York: National Bureau Of Economic Research.
MUIJS, D. (2011). Doing quantitative research in education with SPSS. Los Angeles: Sage Publications.
MURNANE, R. J., & WILLETT, J. B. (2011). Methods matter: Improving causal inference in educational and social science research. Oxford: Oxford University Press.
PALLANT, J. (2016). SPSS survival manual: A step by step guide to data analysis using SPSS. Maidenhead: Open University Press.
PETERSON, P. E., & HOWELL, W. G. (2004). Efficiency, Bias, and Classification Schemes: A Response to Alan B. Krueger and Pei Zhu. American Behavioral Scientist, 47(5), 699-717. doi:10.1177/0002764203260158
PRICE, P. C., JHANGIANI, R. S., & CHIANG, A. (2015, October 13). Quasi-Experimental Research. Retrieved March 15, 2019, from https://opentextbc.ca/researchmethods/chapter/quasi-experimental-research/
PUNCH, K., & OANCEA, A. (2014). Introduction to research methods in education. London: SAGE Publications.
SCIMAGO JOURNAL & COUNTRY RANK IN EDUCATION. (n.d.). Retrieved March 11, 2019, from https://www.scimagojr.com/countryrank.php?category=3304
SHADISH, W. R., COOK, T. D., & CAMPBELL, D. T. (2002). Experimental and quasi-experimental designs for generalized causal inference. Belmont, CA: Wadsworth Cengage Learning.
SHAUGHNESSY, J. J., ZECHMEISTER, E. B., & ZECHMEISTER, J. S. (2015). Research methods in psychology. Dubuque: McGraw-Hill Education.
THOMPSON, C. (2006). Research Study Designs: Experimental and Quasi-Experimental. Air Medical Journal, 25(6), 242-246. doi:10.1016/j.amj.2006.09/001
VAUS, D. A. (2001). Chapter 4, Experimental Design. In Research Design in Social Research (pp. 53-69). Los Angeles, CA: Sage.
WEST, ET AL. (2008). Alternatives to the randomized controlled trial. American journal of public health, 98(8), 1359-66.
WHITE, H., & S. SABARWAL (2014). Quasi-experimental Design and Methods, Methodological Briefs: Impact Evaluation 8, UNICEF Office of Research, Florence.
WOESSMANN, L., & PETERSON, P. E. (2014). Schools and the Equal Opportunity Problem. Cambridge: MIT Press.