Pısa öğrenci anketinin kültürler arası eşdeğerliği

Bu araştırmada Uluslararası Öğrenci Değerlendirme Programı (PISA) 2006 kapsamında uygulanan öğrenci anketinin kültürler ve diller arası eşdeğerliği, Avustralya, Yeni Zelanda, Amerika Birleşik Devletleri ve Türkiye örneklemleri üzerinde karşılaştırmalı olarak incelenmiştir. Araştırmanın birinci aşamasında, öğrenci anketinin faktör yapısı Doğrulayıcı Faktör Analizi (DFA) yöntemiyle incelenmiştir. İkinci aşamada ise, Ortalama ve Kovaryans Yapıları (OKY; Sörbom, 1974) metoduna dayanan Çoklu Grup Doğrulayıcı Faktör Analizi (ÇG-DFA) kullanılarak anket maddelerinin farklı kültürler ve diller arasında Değişen Madde Fonksiyonu (DMF) gösterip göstermediği araştırılmıştır. Son aşamada, uzman görüşlerine başvurularak, maddelerin DMF göstermesinin nedenleri belirlenmeye çalışılmıştır. DFA sonuçları ölçme modelinin tüm örneklemlerde aynı faktör yapısına sahip olduğunu göstermiştir. ÇG-DFA bulguları ülkeler arasında değişen fonksiyon gösteren maddelerin olduğunu ortaya koymuştur. Ülkeler arasında dilsel ve kültürel farklılıklar arttıkça DMF gösteren maddelerin sayısının da arttığı gözlenmiştir. Maddelerin DMF göstermesinin asıl nedenlerinin çeviri problemleri ve kültürel farklılıklar olduğu sonucuna varılmıştır.

Cross-cultural equivalence of the pısa student questionnaire

In this study, cross-lingual and cross-cultural equivalence of the Programme for International Student Assessment (PISA) 2006 Student Questionnaire was assessed comparatively across Turkey, the United States of America, Australia, and New Zealand samples. At the first stage of this study, the factor structure of the questionnaire constructs was examined using Confirmatory Factor Analysis (CFA). At the second stage, Multi-Group Confirmatory Factor Analyses (MG-CFA) based on Mean and Covariance Structures (MACS; Sörbom, 1974) were conducted in order to determine whether the questionnaire items exhibit differential item functioning (DIF) across different cultures or languages. At the last stage, causes of DIF were identified by expert judgments. CFA results showed that the measurement model had the same factor structure across samples. MG-CFA results indicated that some items exhibited DIF between countries. It is observed that as the linguistic and cultural differences increased between countries, the number of DIF items increased. The main reasons for DIF were found to be due to poor translation and differences in cultural relevance.

PDF

___

Allalouf, A., Hambleton, R., & Sireci, S. (1999). Identifying the Causes of DIF in Translated Verbal Items. Journal of Educational Measurement, 36, 185-198.
American Educational Research Association, American Psychological Association, & National Council on Measurement in Education. (1999). Standards for Educational and Psychological Testing. Washington, DC: American Educational Research Association.
Brown, T. A. (2006). Confirmatory Factor Analysis for Applied Research. New York: The Guilford Press.
Byrne, B. M. (2008). Testing for Multigroup Equivalence of a Measuring Instrument: A Walk Through the Process. Psicothema, 20 (4), 872-882.
Byrne, B. M., & Stewart, S. M. (2006). The MACS Approach to Testing for Multigroup Invariance of a Second-Order Structure: A Walk Through the Process. Structural Equation Modeling, 13(2), 287-321.
Chan, D. (2000). Detection of Differential Item Functioning on the Kirton Adaptation-Innovation Inventory Using Multiple-Group Mean and Covariance Structure Analyses. Multivariate Behavioral Research, 35, 169-199.
Cheung, G. W., & Rensvold, R. B. (1999). Testing Factorial Invariance Across Groups: A Reconceptualization and Proposed New Method. Journal of Management, 25, 1-7.
De Klerk, G. (2008). Cross Cultural Testing. In M. Bron, C.D. Foxcroft & R. Butter (Eds.), Online Readings in Testing and Assessment, International Test Commission, http://www.intestcom.org/ Publications/ORTA.php
Drasgow, F., & Kanfer, R. (1985). Equivalence of Psychological Measurement in Heterogeneous Populations. Journal of Applied Psychology, 70(4), 662-680.
Ercikan, K. (1998). Translation Effects in International Assessments. International Journal of Educational Research, 29, 543-553.
Ercikan, K. (2002). Disentangling the Sources of Differential Item Functioning in Multilingual Assessments. International Journal of Testing, 2 (3&4), 199-215.
Ercikan, K., Gierl, M. J., McCreith, T., Puhan, G., & Koh, K. (2004). Comparability of Bilingual Versions of Assessments: Sources of Incomparability of English and French Versions of Canada’s National Achievement Tests. Applied Measurement in Education, 17(3), 301-321.
Ercikan, K., & Koh, K. (2005). Examining the Construct Comparability of the English and French Versions of TIMSS. International Journal of Testing, 5(1), 23-35.
Gierl, M. J. (2000). Construct Equivalence on Translated Achievement Tests. Canadian Journal of Education, 25(4), 280-296.
Gierl, M., J., & Khaliq, S., N. (2001). Identifying Sources of Differential Item and Bundle Functioning on Translated Achievement Tests: A Confirmatory Analysis. Journal of Educational Measurement, 38(2), 164-187.
Grisay, A. (2003). Translation Procedures in OECD/PISA 2000 International Assessment. Language Testing, 20(2), 225-240
Grisay, A., de Jong, J. H. A. L., Gebhardt, E., Berenzer, A., & Halleux-Monseur, B. (2007). Translation Equivalence Across PISA Countries. Journal of Applied Measurement, 8 (3), 249-266.
Grisay, A. & Monseur, C. (2007). Measuring the Equivalence of Item Difficulty in the Various Versions of an International Test. Studies in Educational Evaluation, 33, 69-86.
Hair, J. F., Tatham, R. L., Anderson, R. E. & Black, W. (1998). Multivariate Data Analysis (5th ed.). New York: Prentice Hall.
Hambleton, R.K. (1994). Guidelines for Adapting Educational and Psychological Tests: A Progress Report. European Journal of Psychological Assessment, 10, 229-240.
Hambleton, R.K. (2002). Adapting Achievement Tests into Multiple Languages for International Assessments. In A. Porter y A. Gamoran (Ed.), Methodological Advances in Large-scale Crossnational Education Surveys (pp. 58-79) Washington: National Academy of Sciences.
Hambleton, R.K. (2005). Issues, Designs, and Technical Guidelines for Adapting Tests into Multiple Languages and Cultures. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 3-38). Hillsdale, NJ: Lawrence Erlbaum.
Hambleton, R.K., & de Jong, J. H. A. L. (2003). Advances in Translating and Adapting Educational and Psychological Tests. Language Testing, 20, 127-134.
Hambleton, R. K., & Patsula, L. (1998). Adapting Tests for Use in Multiple Languages and Cultures. Social Indicators Research, 45, 153-171.
Hambleton, R. K., & Patsula, L. (1999). Increasing the Validity of Adapted Tests: Myths to be Avoided and Guidelines for Improving Test Adaptation Practices. Journal of Applied Testing Technology, 1(1), 1-30.
Hui, C. H., & Triandis, H. C. (1985). Measurement in Cross-Cultural Psychology: A Review and Comparison of strategies. Journal of Cross-Cultural Psychology, 16(2), 131-152.
International Test Commission (2001). International Test Commission Guidelines for Test Adaptation. London: Author.
Jöreskog, K. G,. & Sörbom, D., (1999). Lisrel 8.30: Structural Equation Modeling with the Simplis Command Language, Chicago, IL: Scientific Software International, Inc.
Jöreskog, K. G., & Sörbom, D. (2001). Lisrel 8: User’s Reference Guide. Chicago: Scientific Software International.
Jöreskog, K. G., & Sörbom, D. (2002). PRELIS 2: User’s Reference Guide. Chicago: Scientific Software International.
Little, T. D. (1997). Mean and Covariance Structures (MACS)Analyses of Cross- cultural Data: Practical and Theoretical Issues. Multivariate Behavioral Research, 32, 53-76.
MEB (2007). PISA 2006 Uluslararası Öğrenci Değerlendirme Programı Ulusal Ön Raporu. MEB, Ankara.
MEB (2010). PISA 2009 Uluslararası Öğrenci Değerlendirme Programı Ulusal Ön Raporu. MEB, Ankara.
Meredith, W. (1993). Measurement Invariance, Factor Analysis and Factorial Invariance. Psychometrica, 58, 525-543.
OECD (2007). PISA 2006: Science Competencies for Tomorrow’s World: Volume 1 and Volume 2. Paris: OECD Publications.
OECD (2009). PISA 2006 Technical Report. Paris: OECD Publications. PISA Web Site: www.pisa.oecd.org
Schmitt, N., & Kuljanin, G. (2008). Measurement Invariance: Review of Practice and Implications. Human Resource Management Review, 18, 210-222.
Schulz, W. (2003). Validating Questionnaire Constructs in International Studies. Two Examples from PISA 2000. Paper Presented at the Annual Meetings of the American Educational Research Association (AERA), Chicago, 21-25 April.
Schulz, W. (2005). Testing Parameter Invariance for Questionnaire Indices Using Confirmatory Factor Analysis and Item Response Theory. Paper Presented at the Annual Meetings of the American Educational Research Association (AERA), San Francisco, 7-11 April.
Schulz, W. (2008). Questionnaire Construct Validation in the International Civic and Citizenship Education Study. Paper presented to the 3rd IEA International Research Conference, Taipei, September.
Sireci, S. G. (2005). Using Bilinguals to Evaluate the Comparability of Different Language Versions of a Test. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 93-115). Hillsdale, NJ: Lawrence Erlbaum.
Sireci, S. G., & Allalouf, A. (2003). Appraising Item Equivalence Across Multiple Languages and Cultures. Language Testing 20(2), 148-166.
Sireci, S. G., & Berberoğlu, G. (2000). Using Bilingual Respondents to Evaluate Translated – Adapted Items. Applied Measurement in Education, 13(3), 229-248.
Sireci, S. G., & Swaminathan, H. (1996). Evaluating Translation Equivalence: So What’s the Big Dif? Paper Presented at the Annual Meeting of the Northeastern Educational Research Association, Ellenville, NY, October.
Sörbom, D. (1974). A General Method for Studying Differences in Factor Means and Factor Structures between Groups. British Journal of Mathematical and Statistical Psychology, 27, 229-239.
Stark, S., Chernyshenko, O. S., & Drasgow, F. (2006). Detecting Differential Item Functioning with Confirmatory Factor Analysis and Item Response Theory: Toward a Unified Strategy. Journal of Applied Psychology, 91(6), 1292-1306.
Tabachnick, B. G. & Fidell, L. S. (2007). Using Multivariate Statistics (5th ed.). Boston MA: Allyn & Bacon.
Turner, R., & Adams, R. J. (2007). The Programme for International Student Assessment: an Overview. Journal of Applied Measurement, 8(3), 237-248.
Van de Vijver, F. J. R., & Poortinga, Y. H. (2005). Conceptual and Methodological Issues in Adapting Tests. In R. K. Hambleton, P. F. Merenda, & C. D. Spielberger (Eds.), Adapting educational and psychological tests for cross-cultural assessment (pp. 39-63). Hillsdale, NJ: Lawrence Erlbaum.
Van de Vijver, F. J. R., & Tanzer, N. K. (2004). Bias and Equivalence in Cross-Cultural Assessment. European Review of Applied Psychology, 54, 119-135.
Walker, M (2007). Ameliorating Culturally Based Extreme Response Tendencies to Attitude Items. Journal of Applied Measurement, 8(3), 267-278.
Wu, Amery D., Li, Zhen & Zumbo, Bruno D. (2007). Decoding the Meaning of Factorial Invariance and Updating the Practice of Multi-group Confirmatory Factor Analysis: A Demonstration with TIMSS Data. Practical Assessment Research & Evaluation, 12(3). Available online: http:// pareonline.net/pdf/v12n3.pdf
Yıldırım, H. H (2006). “The Differential Item Functioning (DIF) Analysis of Mathematics Items in the International Assessments Programs.” Yayınlanmamış Doktora Tezi. Orta Doğu Teknik Üniversitesi, Ankara.
Yıldırım, H. H. & Berberoğlu, G. (2009). Judgmental and Statistical Analyses of the PISA-2003 Mathematics Literacy Items. International Journal of Testing, 9(2), 108-121.
Zumbo, B. D., (2003). Does Item-Level DIF Manifest Itself in Scale-Level Analyses? Implications for Translating Language Tests. Language Testing, 20 (2), 136-147.
Zumbo, B. D. (2005). Manifestation Of Differences In Item-Level Characteristics In Scale-Level Measurement Invariance Tests Of Multi-Group Confirmatory Factor Analyses. Journal of Modern Applied Statistical Methods, 4 (1), 275-282.
Zumbo, B. D., Sireci, S. G., & Hambleton, R. K. (2003). Re-Visiting Exploratory Methods for Construct Comparability: Is there something to be Gained From the Ways of Old? Paper Presented in the Symposium Construct Comparability Research: Methodological Issues and Results, National Council on Measurement in Education, April, Chicago, IL.