The Examination of Measurement Invariance and Differential Item Functioning of PISA 2015 Cognitive Tests In Terms of The Commonly Used Languages

The aim of the present study was to examine the measurement invariance (MI) of the reading, mathematics, and science tests in terms of the commonly used languages. It also aimed to examine the differential item functioning (DIF) of the PISA test, the original items of which are in the languages of English and French, in terms of the language variable. As a result of the MI analyses, it was concluded that the best working model for the reading comprehension, mathematics and science tests in terms of the commonly used languages for all three tests was the strong factorial invariance model. As a result, it was found that MI could not be established for any of these tests. Moreover, the analyses indicated that population heterogeneity (PH) could not be obtained in any of the three tests, and that the latent means for these tests were not equal. Furthermore, DIF was identified in more than half of the items. It can be deduced that this condition may result in an increase in the number of items with DIF specifically in terms of the variety in the number of languages.

Yaygın Kullanılan Diller Açısından PISA 2015 Bilişsel Testleri İçin Ölçme Değişmezliğinin ve Değişen Madde Fonksiyonlarının İncelenmesi

: Bu araştırmanın amacı, PISA 2015 uygulamasına ait okuduğunu anlama becerisi, matematik ve fen okuryazarlığı testlerinin yaygın kullanılan diller açısından ölçme değişmezliğini ve söz konusu testlere ait maddeler için orijinal dili olan İngilizce ve Fransızca olan PISA’nın dil değişkenine göre değişen madde fonksiyonunu incelemektir. Bu doğrultuda PISA uygulamasında orijinal diller dışında yaygın kullanılan beş test dili belirlenmiştir. Ölçme değişmezliği analizleri sonucunda okuduğunu anlama becerisi, matematik ve fen okuryazarlığı testlerinin üçü için de yaygın kullanılan diller açısından en iyi çalışan modelin güçlü faktöriyel değişmezlik modeli olduğu ve dolayısıyla bu testler için ölçme değişmezliğinin sağlanamadığı sonucuna ulaşılmıştır. Diğer taraftan maddelerin yarısından çoğunda DMF tespit edilmiştir. Bu durum, özellikle dil sayısındaki çeşitlilik ile birlikte değişen madde fonksiyonuna sahip madde sayısının da artmasına neden olabileceği biçiminde düşünülebil

PDF

___

Abedi, J. (2004). The no child left behind act and English language learners: Assessment and accountability issues. Educational Researcher, 33(1), 4-14. https://doi.org/10.3102/0013189X033001004

Abedi, J., Hofstetter, C. H., & Lord, C. (2004). Assessment accommodations for English language learners: Implications for policy-based empirical research. Review of Educational Research, 74(1), 1-28. https:// doi.org/10.3102/00346543074001001

Agresti, A. (2010). Analysis of ordinal categorical data (Vol. 656). John Wiley & Sons. https://doi.org/10.1002/ 9780470594001

Alatlı, B. K., & Bökeoğlu, Ö. Ç. (2018). Uluslararası öğrenci değerlendirme programı (PISA -2012) okuryazarlık testlerinin ölçme değişmezliğinin incelenmesi. İlköğretim Online, 17(2), 1096-1115. DOI: 10.17051/ ilkonline.2018.419357

Allalouf, A. (2000, April). Retaining Translated Verbal Reasoning Items by Revising DIF Items. Annual Meeting of the American Educational Research Association, New Orleans, LA, USA. https://doi.org/ 10.1111/j.1745-3984.1999.tb00553.x

Allalouf, A., Hambleton, R. K., & Sireci, S. G. (1999). Identifying the causes of DIF in translated verbal items. Journal of educational measurement, 36(3), 185-198.

Ardic, E. O., & Gelbal, S. (2017). Cross-Group Equivalence of Interest and Motivation Items in PISA 2012 Turkey Sample. Eurasian Journal of Educational Research, 68, 221 -238. https://doi.org/10.14689/ ejer.2017.68.12

Ayvallı, M. (2016). PISA 2012 matematik okuryazarlığı testinin ölçme değişmezliğinin incelenmesi (Yayınlanmamış Yüksek Lisans Tezi). Akdeniz Üniversitesi, Antalya.

Brown, T. A. (2006). Confirmatory factor analysis for applied research. (First Edition). New York: Guilford Publications, Inc.

Butler, F. A., Bailey, A. L., Stevens, R., Huang, B., & Lord, C. (2004, December). Academic English in Fifth-Grade Mathematics, Science, and Social Studies Textbooks. Center for Research on Evaluation Standards and Student Testing CRESST. (ERIC Document Reproduction Service No. ED483409). https://files.eric.ed.gov/fulltext/ED483409.pdf

Byrne, B. M. (2006). Structural equation modeling with EQS and EQS/Windows: Basicconcepts, applications, and programming. (Second Edition). California: Sage Publications, Inc.

Chen, F. F. (2007). Sensitivity of goodness of fit indexes to lack of measurement invariance. Structural Equation Modeling, 14 (3), 464-504. https://doi.org/10.1080/10705510701301834

Ercikan, K., & Koh, K. (2005). Examining the construct comparability of the English and French versions of TIMSS. International Journal of Testing, 5(1), 23-35. https://doi.org/10.1207/s15327574ijt0501_3

Everitt, B. S. & Howell, D. C. (2005). Encyclopedia of statistics in behavioral sciences. Chichester: John Wiley & Sons, Ltd. https://doi.org/10.1002/0470013192

French, B. F., Finch, W. H., & Vazquez, J. A. V. (2016). Differential item functioning on mathematics items using multilevel SIBTEST. Psychological Test and Assessment Modeling, 58(3), 471.

Grisay, A., & Monseur, C. (2007). Measuring the equivalence of item difficulty in the various versions of an international test. Studies in educational evaluation, 33(1), 69-86. https://doi.org/10.1016/j.stueduc.2007 .01.006

He, W., & Wolfe, E. W. (2010). Item equivalence in English and Chinese translation of a cognitive development test for preschoolers. International Journal of Testing, 10(1), 80-94. https://doi.org/10.1080/ 15305050903534738

Huang, X., Wilson, M., & Wang, L. (2016). Exploring plausible causes of differential item functioning in the PISA science assessment: language, curriculum or culture. Educational Psychology, 36(2), 378-390. https://doi.org/10.1080/01443410.2014.946890

Huang, X. (2010). Differential Item Functioning: The Consequence of Language, Curriculum, or Culture?. University of California, Berkeley.

Marotta, L., Tramonte, L., & Willms, J. D. (2015). Equivalence of testing instruments in Canada: Studying item bias in a cross-cultural assessment for preschoolers. Canadian Journal of Education, 38(3), 1.

Nguyen, H. T., & Cortes, M. (2013). Focus on Middle School: Teaching Mathematics to ELLs: Practical Research-Based Methods and Strategies: Detra Price-Dennis, Editor. Childhood Education, 89(6), 392-395. https://doi.org/10.1080/00094056.2013. 854130

Nunnaly, J. C. & Bernstein, I. H. (1994). Psychometric theory. (Third Edition). New York: McGraw-Hill, Inc.

Oliveri, M. E., Ercikan, K., & Zumbo, B. (2013). Analysis of sources of latent class differential item functioning in international assessments. International Journal of Testing, 13(3), 272-293. https://doi.org/10.1080/ 15305058.2012.738266

Price, L. R., & Oshima, T. C. (1998). Differential Item Functioning and Language Translation: A Cross-National Study with a Test Developed for Certification.

Raju, N. S. (1990). Determining the significance of estimated signed and unsigned areas between two item response functions. Applied Psychological Measurement, 14(2), 197-207. https://doi.org/10.1177/ 014662169001400208

Roth, W. M., Ercikan, K., Simon, M., & Fola, R. (2015). The assessment of mathematical literacy of linguistic minority students: Results of a multi-method investigation. The Journal of Mathematical Behavior, 40, 88-105. https://doi.org/10.1016/j.jmathb.2015.01.004

Van de Vijver, F. J., & Poortinga, Y. H. (1997). Towards an integrated analysis of bias in cross-cultural assessment. European journal of psychological assessment, 13(1), 29 37. https://doi.org/10.1027/1015-5759.13.1.29

Xie, Y., & Wilson, M. (2008). Investigating DIF and extensions using an LLTM approach and also an individual differences approach: an international testing context. Psychology Science, 50(3), 403.

Zieky, M. (1993). Practical questions in the use of DIF statistics in test development. In P. W. Holland, and H. Wainer (Eds.), Differential item functioning (pp. 337-347). Hillsdale, NJ: Lawrence Erlbaum.