Kirkpatrick Model: Its Limitations as Used in Higher Education Evaluation

One of the widely known evaluation models adapted to education is the Kirkpatrick model. However, this model has limitations when used by evaluators especially in the complex environment of higher education. Addressing the scarcity of a collective effort on discussing these limitations, this review paper aims to present a descriptive analysis of the limitations of the Kirkpatrick evaluation model in the higher education ﬁeld. Three themes of limitations were found out: propensity towards the use of the lower levels of the model; rigidity wich leaves out other essential aspects of the evaluand; and paucity of evidence on the causal chains among the levels. It is suggested that, when employing the Kirkpatrick model in higher education, evaluators should address these limitations by considering more appropriate methods, integrating contextual inputs in the evaluation framework, and establishing causal relationships among the levels. These suggestions to address the limitations of the model are discussed at the end of the study.

Anahtar Kelimeler:

Kirkpatrick model, Program evaluation, Higher education, Limitation

Kirkpatrick Model: Its Limitations as Used in Higher Education Evaluation

Keywords:

Kirkpatrick model, Program evaluation, Higher education, Limitation,

PDF

___

Abdulghani, H., A Al Drees, A.M., Khamis, N., & Irshad, M. (2014). Research methodology workshops evaluation using the Kirkpatrick’s model: Translating theory into practice. Medical Teacher, 36(1), s24-s29. https://doi.org/10.3109/0142159x.2014.886012
Abernathy D.J. (1999). Thinking outside the evaluation box. Training Development, 53(2), 18-23. Retrieved from https://eric.ed.gov/?id=EJ578905
Alliger, G., and E. Janak. Kirkpatrick’s Levels of Training Criteria: Thirty Years Later. Personnel Psychology 42(2), 331 341. https://doi.org/10.1111/j.1744-6570.1989.tb00661.x
Arthur, W., Jr., Bennet, W., Edens, P.S., & Bell, S.T. (2003). Effectiveness of training in organizations: A meta-analysis of design and evaluation features. Journal of Applied Psychology, 88(2), 234-245. http://dx.doi.org/10.1080/0144341032000060110
Arthur, W., Jr., Tubre, T. C., Paul, D. S., & Edens, P. S. (2003). Teaching effectiveness: The relation- ship between reaction and learning criteria. Educational Psychology, 23(3), 275-285. https://doi.org/10.1080/0144341032000060110
Aryadoust, V. (2017). Adapting Levels 1 and 2 of Kirkpatrick’s model of training evaluation to examine the effectiveness of a tertiary-level writing course. Pedagogies: An International Journal, 12(2), 151-179. https://doi.org/10.1080/1554480X.2016.1242426
Bates, R. (2004). A critical analysis of evaluation practice: the Kirkpatrick Model and the principle of beneficence. Evaluation and Program Planning, 27, 341 347. https://doi.org/10.1016/j.evalprogplan.2004.04.011
Baskin, C. (2001, December). Using Kirkpatrick’s four-level-evaluation model to explore the effectiveness of collaborative online group work. In Proceedings of the Annual Conference of the Australasian Society for Computers in Learning in Tertiary Education (pp. 37-44). Melbourne, Australia: Biomedical Multimedia Unit, The University of Melbourne.
Bowen, G. A. (2009) Document analysis as a qualitative research method. Qualitative Research Journal, 9(2), 27-40. https://doi.org/10.3316/qrj0902027
Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77-101. https://doi.org/10.1191/1478088706qp063oa
Chang, N. & Chen, L. (2014). Evaluating the learning effectiveness of an online information literacy class based on the Kirkpatrick framework. Libri, 64(3), 211 223. https://doi.org/10.1515/libri-2014-0016
Covington, J.A. (2012). Efficacy of webinar training for continuing professional education: applications for school personnel in k-12 settings (Ph.D. dissertation, University of North Carolina) Retrieved from https://eric.ed.gov/?id=ED550661
Dewi. L.R., & Kartowagiran, B. (2018). An evaluation of internship program by using Kirkpatrick evaluation model. Research and Evaluation in Education, 4(2), 155-163. https://doi.org/10.21831/reid.v4i2.22495
Embi, Z.C., Neo, T.K., & Neo, M. (2017). Using Kirkpatrick’s evaluation model in a multimedia-based blended learning environment. Journal of Multimedia Information System, 4(3), 115-122, 2383-7632. http://dx.doi.org/10.9717/JMIS.2017.4.3.115
Farjad, S. (2012). The Evaluation effectiveness of training courses in university by Kirkpatrick model. Procedia – Social and Behavioral Sciences, 46, 2837 2841. https://doi.org/10.1016/j.sbspro.2012.05.573
Frey, B. (2018). The SAGE encyclopedia of educational research, measurement, and evaluation. Thousand Oaks, CA: SAGE Publications, Inc. https://doi.org/10.4135/9781506326139
Guptill, A. (2016). Secondary sources in their natural habitats. Writing in College. Retrieved from https://milnepublishing.geneseo.edu/writing-in-college-from-competence-to-excellence/chapter/secondary-sources-in-their- natural-habitats/
Haupt, G. & Blignaut, S. (2007). Uncovering learning outcomes: explicating obscurity in learning of aesthetics in design and technology education. International Journal of Technology and Education, 18(4), 361-374. https://doi.org/10.1007/s10798-007-9029-1
Hilbert, J., Preskill, H., & Russ-Eft, D. (1997). Evaluating training’s effectiveness. In L. Bassi & D. Russ- Eft (Eds.), What works: Assessment, development, and measurement (pp. 109–150). Alexandria, VA: American Society for Training and Development.
Kirkpatrick, D. L. (1959). Techniques for evaluating training programs. Journal of the American Society of Training and Development, 13, 3-9.
Kirkpatrick, D. L. (1976). Evaluation of training. In R. L. Craig (Ed.), Training and development handbook: A guide to human resource development (2nd ed., pp. 301–319). New York: McGraw-Hill.
Kirkpatrick D. L, Kirkpatrick J. D. (2006). Evaluating training programs: The four levels (3rd ed.). San Francisco: Berrett-Koehler Publication.
Lambert, N. (2011). Ban happy sheets! - Understanding and using evaluation. Nurse Education Today, 32(1), 1-4. https://doi.org/10.1016/j.nedt.2011.05.020
Liao, S.C., & Hsu, S.Y. (2019). Evaluating a continuing medical education program: New World Kirkpatrick Model Approach. International Journal of Management, Economics and Social Sciences, 8(4), 266-279. http://dx.doi.org/10.32327/IJMESS/8.4.2019.17
Lillo-Crespo, M., Sierras-Davo, M.C., McRae, R., & Rooney. K. (2017). Developing a framework for evaluating the impact of Healthcare Improvement Science Education across Europe: A qualitative study. Journal of Educational Evaluation in Health Profession, 14, 28. https://doi.org/10.3352/jeehp.2017.14.28
Masood, R., & Usmani, M.A.W. (2015). A study for program evaluation through Kirkpatrick’s model. Khyber Medical University Journal, 2(7), 76 80. Retrieved from https://www.kmuj.kmu.edu.pk/article/view/15377
Miller, B.J. (2018). Utilizing the Kirkpatrick model to evaluate a collegiate high-impact leadership development program (Master’s thesis, Texas A&M University, ‎College Station, Texas‎). Retrieved from https://oaktrust.library.tamu.edu
Moreira I.C., Ramos, I., Ventura, S.R., Rodrigues, P.P. (2018). Learner’s perception, knowledge and behaviour assessment within a breast imaging E-Learning course for radiographers. European Journal of Radiology, 111, 47 55. https://doi.org/10.1016/j.ejrad.2018.12.006
Morrison J. (2003). ABC of learning and teaching in medicine: Evaluation. British Medical Journal, 326(7385), 385-387. https://doi.org/10.1136/bmj.326.7385.385
Nickols, F. W. (2000). Evaluating training: There is no "cookbook" approach. Retrieved May 05, 2020 from: http://home.att.net/ ~nickols/evaluating_ training.htm
Paull, M., Whitsed, C. & Girardi, A. (2016). Applying the Kirkpatrick model: Evaluating an Interaction for Learning Framework curriculum intervention. Issues in Educational Research, 26(3), 490-502. Retrieved from https://www.iier.org.au/iier26/paull.pdf
Quintas, C., Fernandes Silva, I., & Tiexiera, A. (2017). Assessing an e-Learning and b-Learning Model - A study of perceived satisfaction. International Journal of Information and Education Technology, 7(4), 265-268. https://doi.org/10.18178/ijiet.2017.7.4.878
Reio, T.G., Rocco, T.S., Smith, D.H., & Chang, E. (2017). A Critique of Kirkpatrick’s Evaluation Model. New Horizons in Adult Education & Human Resource Development 29(2), 35-53. https://doi.org/10.1002/nha3.20178
Rouse D.N. 2011. Employing Kirkpatrick’s evaluation framework to determine the effectiveness of health information management courses and programs. Perspectives of Health Information Management, 8, 1c 5c. Retrieved from https://www.ncbi.nlm.nih.gov/pubmed/21464860
Sahin, V. (2006). Evaluation of the in-service teacher training program “The Certificate for Teachers of English” at the Middle East Technical University School of Foreign Languages. (Dissertation, Middle East Technical University, Ankara, Turkey) Retrieved from https://etd.lib.metu.edu.tr/upload/12607752/index.pdf
Shelton, S., & Alliger, G. M. (1993). Who’s afraid of level 4 evaluation? A practical approach. Training and Development Journal, 47, 43 46. Retrieved from https://eric.ed.gov/?id=EJ463549
Sileyew, K. J. (2019). Research design and methodology. In E. Abu- Taieh, A. El Mouatasim, & I.H. Al Hadid (eds.), Cyberspace. London: IntechOpen. https://doi.org/10.5772/intechopen.78887
Steele, L. M., Mulhearn, T. J., Medeiros, K. E., Watts, L. L., Connelly, S. & Mumford, M.D. (2016). How do we know what works? A review and critique of current practices in ethics training evaluation. Accountability in Research, 23(6), 319 350. http://dx.doi.org/10.1080/08989621.2016.1186547
Tamkin, P., Yarnall, J. & Kerrin, M. (2002). Kirkpatrick and beyond: A review of models of training evaluation (Report No. 392). London, England: Institute for Employment Studies
Topno, H. (2012). Evaluation of training and development: An analysis of various models. IOSR Journal of Business and Management, 5(2), 16-22. https://doi.org/10.9790/487x-0521622
Yardley, S. & Dornan, T. (2012). Kirkpatrick’s levels and education ‘evidence’. Medical Education, 46(1), 97-106. https://doi.org/10.1111/j.1365-2923.2011.04076.x