Pamukkale critical thinking skill scale: a validity and reliability study

Pamukkale critical thinking skill scale: a validity and reliability study

The aim of this study is to develop a valid and reliable measurement tool that measures critical thinking skills of university students. Pamukkale Critical Thinking Skills Scale was developed as two separate forms; multiple choice and open-ended. The validity and reliability studies of the multiple-choice form were constructed on two different theoretical frameworks as classical test theory and item-response theory. According to classical test theory, exploratory and confirmatory factor analyses were performed, to item-response theory, the Generalized Partial Credit Model (GPCM) for one-dimensional and multi-category scales was tested for the construct validity of the multiple-choice form of the scale. Analysis results supported the unidimensional structure of the scale. The reliability analyzes showed that the internal consistency coefficient of the scale and the item-total correlation values were high enough. The test-retest analysis results supported that the scale shows stability over time regarding the field it measures. The results of the item-response theory-based analysis also showed that the scale met the item-model fit assumptions. In the evaluation of the open-ended form of the scale, a rubric was used. Several studies were conducted on the validity and reliability of the open-ended form, and the results of the analysis provided psychometric support for the validity and reliability. As a result, Pamukkale Critical Thinking Skills Scale, which was developed in two forms, is a valid and reliable measurement tool to measure critical thinking skills of university students. The findings were discussed in the light of the literature and some suggestions were given.

___

  • Abrami, P.C., Bernard, R.M., Borokhovski, E., Wade, A., Surkes, M. ., Tamim, R., & Zhang, D. (2008). Instructional interventions affecting critical thinking skills and dispositions: A stage 1 meta-analysis. Review of Educational Research, 78(4), 1102-1134.
  • Anderson, L.W., & Krathwohl, D.R. (2001). A taxonomy for learning, teaching and assessing: A revision of Bloom’s taxonomy of educational objectives: Complete Edition. Longman.
  • Atay, S., Ekim, E., Gökkaya, S., & Sağım, E. (2009). Sağlık Yüksekokulu öğrencilerinin eleştirel düşünme düzeyleri. [Critical thinking tendencies of Health School students] Sağlık Bilimleri Fakültesi Hemşirelik Dergisi, 39-46.
  • Ayberk, B., & Çelik, M. (2007). Watson-Glaser Eleştirel, Akıl Yürütme Gücü Ölçeği'nin (W-GEAYGÖ) üniversite ikinci, üçüncü ve dördüncü sınıf İngilizce bölümü öğretmen adayları üzerindeki güvenlik çalışması. [Reliability study related to the power of Watson-Glaser critical thinking appraisal scale on university second, third and fourth-grade English department teacher candıdates] Çukurova Üniversitesi Sosyal Bilimler Enstitüsü Dergisi, 16 (1), 101-112
  • Aybek, E.C. (2021). CatIRT tools: A “Shiny” application for item response theory calibration and computerized adaptive testing simulation. Journal of Applied Testing Technology, 22(1), 23-27.
  • Baker, F.B. (2001). The basics of item response theory. College Park, ERIC, Clearinghouse on Assessment and Evaluation.
  • Batur, Z., & Özcan, H.Z. (2020). Eleştirel düşünme üzerine yazılan lisansüstü tezlerinin bibliyometrik analizi. [Bibliometric analysis of graduate theses written on critical thinking] Uluslararası Türkçe Edebiyat Kültür Eğitim Dergisi, 9(2), 834-854.
  • Bailey, R., & Mentz, E. (2015). IT teachers’ experience of teaching–learning strategies to promote critical thinking. Issues in Informing Science and Information Technology, 12(1), 141–152.
  • Bensley, D.A., Crowe, D.S., Bernhardt, P., Buckner, C., & Allman, A.L. (2010). Teaching and assessing critical thinking skills for argument analysis in psychology. Teaching of Psychology, 37(2), 91–96. https://doi.org/10.1080/00986281003626656
  • Bennett, D.A. (2001). How can I deal with missing data in my study? Australian and New Zealand Journal of Public Health, 25(5), 464–469.
  • Bentler, P.M. (1990). Comparative fit indexes in structural models. Psychological Bulletin, 107(2), 238–246. https://doi.org/10.1037/0033-2909.107.2.238
  • Bilican, S.D. (2021). Başarı testlerinin geliştirilmesi ve madde yazımı [Development of achievement tests and item writing] Yıldırım, Ö. ve Kartal, S.K. (Ed.), Eğitimde Ölçme ve Değerlendirme [Measurement and Evaluation in Education] (125-163) 1. Baskı, Lisans Yayıncılık.
  • Browne, N., & Freeman, K. (2000). Distinguishing features of critical thinking classrooms. Teaching in Higher Education. 5(3), 301-309. https://doi.org/10.1080/713699143
  • Boyd, E.M., & Fales, A.W. (1983). Reflective learning: Key to learning from experience. Journal of Humanistic Psychology, 23(2), 99 117. https://doi.org/10.1177/0022167883232011
  • Browne, M.W., & Cudeck, R. (1993). Alternative ways of assessing model fit. K. A. Bollen and J.S. Long (Ed.), Testing structural equation models (pp. 136-162). Sage.
  • Carpendale, J.I., & Lewis, C. (2006). How children develop social understanding, Blackwell.
  • Chalmers, R.P. (2012). Mirt: A multidimensional item response theory package for the R environment. Journal of Statistical Software, 48(6), 1 29. https://doi.org/10.18637/jss.v048.i06
  • Cisneros R .M. (2009). Assessment of critical thinking in pharmacy students. American Journal of Pharmaceutical Education, 73(4), 66. https://doi.org/10.5688/aj730466
  • Cohen, R.J., Swerdlik, M.E., Smith, D.K., & Cohen, R.J. (1992). Psychological testing and assessment: An introduction to tests and measurement. Mayfield Pub. Co.
  • Comfort, L.K. (2007). Crisis management in hindsight: cognition, communication, coordination, and control. Public Administration Review, 67(1), 189–197.
  • Crockett, L. (2019). Future-focused Learning: 10 essential shifts of everyday practice. Solution Tree Press.
  • Dewey, J. (1933). How we think. DC Herman.
  • Doğan, N. (2013). Eleştirel düşünmenin ölçülmesi [Measuring the Critical Thinking]. Cito Eğitim: Kuram ve Uygulama, 22(1), 29-42.
  • Doğanay, A., Akbulut-Taş, M., & Erden, Ş. (2007). Üniversite öğrencilerinin bir güncel tartışmalı konu bağlamında eleştirel düşünme becerilerinin değerlendirilmesi. [Assessing university students’ critical thinking skills in the context of a current controversial issues]. Kuram ve Uygulamada Eğitim Yönetimi, 52(1), 511-546.
  • Dumitru, D., Bîgu, D., Elen, J., Jiang, L., Railiene, A., Penkauskiene, D., Papathanasiou, I.V., Tsaras, K., Fradelos, E.C., Ahern, A.K., McNally, C., O'Sullivan, J., Verburgh, A.P., Jarošová, E., Lorencová, H., Poce, A., Agrusti, F., Re, M.R., Puig, B., Blanco, P., Mosquera, I., Crujeiras-Pérez, B., Dominguez, C., Cruz, G., Silva, H., & Morais, M.D., Nascimento, M.M., & Payan-Carreira, R. (2018). A European review on Critical Thinking educational practices in Higher Education Institutions. http://hdl.handle.net/10197/9865
  • Duru, E., Duatepe-Paksu, A., Balkıs, M., Duru, S., & Bakay, E. (2020). Examination of 21st century competencies and skills of graduates from the perspective of sector representatives and academicians. Journal of Qualitative Research in Education, 8(4), 1059-1079. https://doi.org/10.14689/issn.2148-2624.8c.4s.1m
  • Dwyer, C.P., Hogan, M.J., & Stewart, I. (2014). An integrated critical thinking framework for the 21st century. Thinking Skills and Creativity, 12(1), 43 52. https://doi.org/10.1016/j.tsc.2013.12.004
  • Ebel, R.L. (1972). Essentials of educational measurement. Prentice-Hall.
  • Eğmir, E., & Ocak, G. (2016). Eleştirel düşünme becerisini ölçmeye yönelik bir başarı testi geliştirme. [Developing an achievement test towards evaluating critical thinking skill]. Turkish Studies, 11(19), 337-360.
  • Ennis, R. (1991). Critical thinking: A streamlined conception. Teaching Philosophy, 14(1), 15-24. http://dx.doi.org/10.5840/teachphil19911412
  • Ennis, R. H., & Millman, J. (1985). Cornell Critical Thinking Test (Level X). Critical Thinking Press & Software.
  • Facione, P.A. (1990a). Critical thinking: A statement of expert consensus for purposes of educational assessment and ınstruction. executive summary: “The Delphi Report”. The California Academic Press.
  • Facione, P.A. (1990b). The California Critical Thinking Skills Test-College Level Technical Report 1: Experimental Validation and Content Validity. The California Academic Press.
  • Facione, P.A. (1990c). The California Critical Thinking Skills Test-College Level Technical Report 2: Factors Predictive of Critical Thinking Skills. The California Academic Press.
  • Facione, N.C. (1997) Critical thinking assessment in nursing education programs An aggregate data analysis. The California Academic Press.
  • Facione, P.A. (2015). Critical thinking: What it is and why it counts. http://www.student.uwa.edu.au/__data/assets/pdf_file/0003/1922502/Critical-Thinking-What-it-is-and-why-it-counts.pdf
  • Facione, P.A., & Facione N.C. (1992). The california critical thinking dispositions inventory. The California Academic Press.
  • Fayers, P., & Machin, D. (1995). Factor analysis for assessing validity. Quality of Life Research, 4(5), 424.
  • Flores, K., Matkin, G.S., Burbach, M.E., Quinn, C.E., & Harding, H. (2012). Deficient critical thinking skills among college graduates: ımplications for leadership. Educational Philosophy and Theory, 44(2), 212 230. https://doi.org/10.1111/j.1469 5812.2010.00672.x
  • Güçlü, G., & Evcili, F. (2021). Sağlık hizmetleri meslek yüksekokulu öğrencilerinin eleştirel düşünme yetileri ve boyun eğici davranış eğilimlerinin incelenmesi. [Investigation of critical thinking qualifications and submissive behavior tendency of health services vocational school students]. Turkish Journal of Science and Health. 2(1), 31-39.
  • Haladyna, T.M. (1997). Writing test ıtem to evaluate higher order thinking. Allyn & Bacon.
  • Hambleton, R.K., & Swaminathan, H. (1985). Item response theory. Kluwer-Nijhoff Publishing.
  • Huber, C.R., & Kuncel, N.R. (2016). Does college teach critical thinking? A meta-analysis. Review of Educational Research, 86(2), 431 468. https://doi.org/10.3102/0034654315605917
  • Jacobs, S.S. (1995). Technical characteristics and some correlates of the California Critical Thinking Skills Test, Forms A and B. Research in Higher Education, 36(1), 89-108. https://doi.org/10.1007/BF0220776
  • Irani, T., Rudd, R., Gallo, M., Ricke, J., Friedel, C., & Rhoades, E. (2007) Critical thinking ınstrumentation manual. University of Florida.
  • Lipman, M. (1988) Critical thinking—What can it be? Educational Leadership, 461(1), 38–43.
  • Marais, I. (2009). Response dependence and the measurement of change. Journal of Applied Measurement, 10(1), 17-29.
  • Marin, L., & Halpern, D. (2011). Pedagogy for developing critical thinking in adolescents: Explicit instruction produces greatest gains. Thinking Skills and Creativity, 6(1), 1–13.
  • Mazer, J.P., Hunt, S.K., & Kuznekoff, J.H. (2007). Revising general education: Assessing a critical thinking ınstructional model in the basic communication course. The Journal of General Education 56(3), 173-199. https://doi.org/10.1353/jge.0.0000
  • Mpofu, N., & Maphalala, M.C. (2017). Fostering critical thinking in initial teacher education curriculums: A comprehensive literature review. Gender and Behaviour, 15(2), 9342–9351.
  • Msila, V. (2014). Critical Thinking in open and distance learning programmes: Lessons from the University of South Africa’s NPDE Programme. Journal of Social Sciences, 38(1), 33–42
  • Nalçacı, A., Meral, E., & Şahin, İ.F. (2016). Sosyal bilgiler öğretmen adaylarının eleştirel düşünme ile medya okuryazarlıkları arasındaki ilişki [Correlation between critical thinking and media literacy of social sciences pre-service teachers]. Doğu Coğrafya Dergisi, 21(36), 1-12. https://doi.org/10.17295/dcd.99051
  • Norris, S.P., & Ennis, R.H. (1990). The practitioners' guide to teaching thinking series. Evaluating Critical Thinking. Hawker Bronlow Education.
  • Özmen, K.S. (2008). İngilizce öğretmeni eğitiminde eleştirel düşünce: Bir vaka çalışması. [Critical Thinking in English teacher education: A case study]. Ekev Akademi Dergisi, 12(36), 253-266.
  • Orhan, A., & Çeviker-Ay, Ş. (2022). Developing the critical thinking skill test for high school students: A validity and reliability study. International Journal of Psychology and Educational Studies, 9(1), 132-144. https://dx.doi.org/10.52380/ijpes.2022.9.1.561
  • Parkhurst, H.B. (1999). Confusion, lack of consensus, and the definition of creativity as a construct. Journal of Creative Behavior, 33(1), 1–21.
  • Paul, R. (2005) The state of critical thinking today, New Directions for Community Colleges, 130(1), 27–38.
  • Paul, R., & Elder, L. (2001) Critical thinking: Inert information, activated ignorance, and activated knowledge, Journal of Developmental Education, 25(2), 36–37.
  • Paul, R.W., & Elder, L. (2002). Critical thinking: Tools for taking charge of your professional and personal life. Pearson Education Inc.
  • Paul, R., & Nosich, G. (1991). A proposal for the national assessment of higher-order thinking. Paper commissioned by the U.S. Department of Education Office of Educational Research and Improving National Center for Education Statistics.
  • Pascarella, E.T., & Terenzini, P.T. (1991). How college affects students: Findings and insights from twenty years of research. Jossey-Bass.
  • Pascarella, E.T., & Terenzini, P.T. (2005). How college affects students: A third decade of research. Jossey-Bass.
  • Portney, L.G., & Watkins, M.P. (2000) Foundations of clinical research: Applications to practice. 2nd Edition, Prentice Hall.
  • Puig, B., Blanco-Anaya, P., Bargiela, I.M., & Crujeiras-Pérez, B. (2019). A systematic review on critical thinking intervention studies in higher education across professional fields. Studies in Higher Education, 44(5), 860-869.
  • R Core Team (2021). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria.
  • Rezaee, M., Farahian, M., & Ahmadi, A. (2012). Critical thinking in higher education: Unfulfilled expectations. BRAIN. Broad Research in Artificial Intelligence and Neuroscience, 3(2), 64-73.
  • Ruggiero V.R. (1990) Beyond feelings. A guide to critical thinking, (3rd ed.) Mayfield Publishing.
  • Røe, C., Damsgård, E., Fors, T., & Anke, A. (2014). Psychometric properties of the pain stages of change questionnaire as evaluated by Rasch analysis in patients with chronic musculoskeletal pain. BMC Musculoskelet Disord, 15(1), 95. https://dx.doi.org/10.1186/1471-2474-15-95, PubMed 24646065
  • Sahool, S., & Mohammed, C.A. (2018). Fostering critical thinking and collaborative learning skills among medical students through a research protocol writing activity in the curriculum. Korean J Med Educ, 30(2), 109-118. https://doi.org/10.3946/kjme.2018.86
  • Schafer, J.L. (1999). Multiple imputation: a primer. Stat Methods in Med., 8(1), 3–15. https://doi.org/10.1191/096228099671525676
  • Shipman, V. (1983). New Jersey test of reasoning skills. IAPC, Test Division, Montclair State College.
  • Siegel, H. (1988). Educating for reason: Rationality, critical thinking, and education. Routledge.
  • Snyder, L.G., & Snyder, M.J. (2008). Teaching critical thinking and problem solving skills. The Journal of Research in Business Education, 50 (2), 90–99.
  • Stevens, J. (1992). Applied multivariate statistics for the social sciences. Second Edition, Lawrence Erlbaum Associates.
  • Tabachnick, B.G., & Fidell, L.S. (2013). Using multivariate statistics (6th ed.), Allyn and Bacon.
  • Tolman, E.C. (1932). Purposive behavior in animals and men. Century/Random House UK.
  • Trilling, B., & Fadel, C. (2009). 21st-century skills: Learning for life in our times. John Wiley & Sons.
  • Uzuntiryaki- Kondakçı, E., & Çapa-Aydın, Y. (2013). Predicting critical thinking skills of university students through metacognitive self-regulation skills and chemistry self-efficacy. Educational Sciences: Theory & Practice, 13(1), 666-670.
  • Van Laar, E., van Deursen, A.J.A. M., van Dijk, J.A.G.M., & de Haan, J. (2019). Determinants of 21st-century digital skills: A large-scale survey among working professionals. Computers in Human Behavior, 100, 93–104. https://doi.org/10.1016/j.chb.2019.06.017
  • Voogt, J., & Roblin, N.P. (2012). A comparative analysis of international frameworks for 21st-century competencies: implications for national curriculum policies. Journal of Curriculum Studies, 44(3), 299–321.
  • Watson, G., & Glaser, E.M. (1980). Watson-Glaser critical thinking appraisal. Psychological Corporation
  • Williams, R.L., Oliver, R., Allin, J.L., Winn, B., & Booher, C.S. (2003). Psychological critical thinking as a course predictor and outcome variable. Teaching of Psychology, 30(3), 220–223. https://doi.org/10.1207/S15328023TOP3003_04
  • Yen, W.M. (1993). Scaling performance assessments: Strategies for managing local item dependence. Journal of Educational Measurement, 30(1), 187-213.