Cross-cultural structural parameter invariance on PISA 2006 student questionnaires

The Program for International Student Assessment (PISA), OECD tarafından gerçekleştirilen bazı temel alanlarda öğrencilerin bilgi ve becerilerini değerlendirmeyi amaçlayan bir programıdır. PISA genel anlamda 15 yaş grubunda yer alan öğrencilerin, okuma, matematik ve fen bilimleri alanlarında öğrendikleri bilgi ve becerileri gerçek yaşam durumlarına uygulama ve uyarlayabilme yeteneğine odaklanmaktadır. PISA’nın değerlendirme süreçleri temelde, öğrencilerin öğrendiklerine ilişkin farkındalık düzeylerini ve bu öğrendikleri bilgileri okul veya okul dışı ortamlarda nasıl uygulayabildiklerini saptamayı amaçlamaktadır. PISA her uygulamasında, okuma, matematik ve fen bilimleri alanlarından birine derinlemesine odaklanmakta, ancak diğer iki alanda da değerlendirme yapmaktadır. PISA’nın 2006 yılındaki uygulamasında topladığı veriler öğrencilerin başarı düzeylerinin yanı sıra; öz-bildirimli (self-report) tutumlar, ilgiler, motivasyon, öğrenme davranışları gibi değişkenlerle ilgilidir. Anketlerden elde edilen veriler genelde öğrenci performanslarındaki değişkenliği açıklamada kullanmak üzere toplanmaktadır. Araştırmanın Amacı PISA gibi kültürler arası çalışmalarda, ölçekler farklı ülkelere uygulandığı için tek bir form kullanılması mümkün değildir, o ülkelerin diline çevrilmesi gerekir. Dil farklılıkları ölçek eşitsizlikleri üzerinde güçlü bir etkiye sahip olabilir. Bununla birlikte, farklı ülkelerin farklı kültür ve dil durumlarından dolayı, çeviri testler tüm kültürlerde aynı şekilde işlev görmeyebilir. Bu durum testin eşit olamayabildiği veya farklı kültürler için adil olamadığı şeklinde isimlendirilebilmektedir. Çoklu-grup doğrulayıcı faktör analiz (DFA) modeli testin faktör yapısı, faktör yükleri ve faktör korelasyonları, hata varyansları değişmezliğini veya eşitliğini test etme yoluyla bir ölçme aracının kültürlerarası geçerliğini değerlendirme yöntemi olarak bilinir. Bu araştırma fen bilimleri bağlamıyla ilgili PISA anketinin faktör yapısını ve 10 ülke örneklemi arasında anketin eşitliğini çoklu-grup doğrulayıcı faktör analizi kullanarak incelemeyi amaçlamaktadır.

PISA 2006 öğrenci anketinin yapısal parametrelerin kültürlerarası değişmezliğinin incelenmesi

Problem Statement: In cross-cultural studies such as PISA, it is not possible to use a single form, since the scales are applied to different countries, and it is necessary to translate the form into the language of the country that will use it. Language differences may have a strong impact on measurement inequalities. Nevertheless, translated tests may not function in the same way, because of different culture and language characteristics of different countries. This situation may be described as the test not being equivalent, or fair, for different cultures. Translation is the first step of a long-lasting process in adapting the test to different cultures; the basic objective of adaptation is to preserve the structural equivalence between the versions of two or more languages, and to protect the test content. Purpose of Study: This study aims to examine the factorial invariance of some of the PISA questionnaire in relation to its scientific context, and the equality of the questionnaire across the ten countries, by a multi-group confirmatory factor analysis model. Methods: In this study, samplings from ten countries were used. For the crosscultural invariance of PISA questionnaires, a set of confirmatory factor analysis procedures were used. If the introduction of a set of invariance constraints results in a substantial reduction in goodness of fit, then there is evidence against the appropriateness of those invariance constraints. Confirmatory factor analyses were conducted with LISREL. Findings and Results: As a result, in the model in which there is a constraint indicating that factor loadings should be equal for all countries, there is no evidence of a decrease in fit index level, exceeding the criterion in comparison to the baseline model. This result strongly supports the conclusion that factor loadings do not vary from one country to another. However, in the model in which error variances are also constrained, NNFI and RFI fit indexes show higher declines than .01, when compared to the baseline model. Conclusions and Recommendations: This finding indicates that error variances may vary from one country to another. Furthermore, fit indexes show higher decreases, exceeding the limits in the model in which there is a constraint on equivalency of correlations between factors, when compared to the baseline model.

PDF

___

AERA, APA, & NCME (1999). Standards for educational and psychological testing, Washington, DC: Author.
Akın, A., & Çetin, B. (2007). Achievement goal orientations scale: The study of validity and reliability. Egitim Arastirmalari-Eurasian Journal of Educational Research, 26, 1-12.
Allalouf, A., Hambleton, R. K., & Sireci, S. G. (1999). Identifying the causes of DIF in translated verbal items. Journal of Educational Measurement, 36(3), 185-198.
Bagby. M., Taylor, G. J., Quilty, L. C., & Parker, J. D. A. (2007). Reexamining the factor structure of the 20-item Toronto Alexithymia Scale: Commentary on Gignac, Palmer, and Stough. Journal of Personality Assessment, 89(3), 258-264.
Beckstead, J. W., Yang, C. Y., & Lengacher, C. A. (2008). Assessing cross-cultural validity of scales: A methodological review and illustrative example. International Journal of Nursing Studies, 45, 110-119.
Bentler, P. M. (1990). Comparative fit indices in structural models. Psychological Bulletin, 107, 238-246.
Bollen, K. A. (1989). Structural equations with latent variables, New York: Wiley.
Breakwell, G. M., & Beardsell, S. (1992) Gender, parental, and peer influences upon science attitudes and activities. Public Understanding of Science, 1, 183-197.
Byrne, B. M. (1998). Structural equation modeling with LISREL, PRELIS, and SIMPLIS: Basic concepts, applications and programming, Mahwah, NJ: Lawrence Erlbaum Associates.
Byrne, B. M. (2006). Structural equation modeling with EQS: Basic concepts, applications, and programming, Mahwah, NJ: Lawrence Erlbaum.
Byrne, B., & Campbell, T. L. (1999). Cross-cultural comparisons and the presumption of equivalent measurement and theoretical structure: A look beneath the surface. Journal of Cross-Cultural Psychology, 30(5), 555-574.
Cheung, G. W., & Rensvold, R. B. (2002). Evaluating goodness-of-fit indexes for testing measurement invariance. Structural Equation Modeling, 9, 233-255.
Crawley, F. E., & Black, C. B. (1992). Causal modelling of secondary science students intentions to enroll in physics. Journal of Research in Science Teaching, 29, 585- 599.
Drasgow, F., & Kanfer, R. (1985) Equivalence of psychological measurement in heterogeneous populations. Journal of Applied Psychology, 70, 662-680.
Ercikan, K. (2002). Disentangling sources of differential item functioning in multilanguage assessments. International Journal of Testing, 2(3&4), 199-215.
Gardner, P. L. (1975). Attitudes to science. Studies in Science Education, 2, 1-41.
Haladyna, T., Olsen, R., & Shaughnessy, J. (1982). Relations of student, teacher, and learning environment variables to attitudes to science. Science Education, 66, 671-687.
Hilton, S. C., Schau, C., & Olsen, J. A. (2004). Survey of Attitudes Toward Statistics: Factor structure invariance by gender and by administration time. Structural Equation Modeling, 11(1), 92-109.
Hui, C. H., & Triandis, H. C. (1989). Effects of culture and response format on extreme response style. Journal of Cross-Cultural Psychology, 20(3), 296-309.
Jöreskog, K. G. (1971). Simultaneous factor analysis in several populations. Psychometrica, 36(4), 409-426.
Jöreskog, K. G., & Sörbom, D. (1993). LISREL 8: Structural equation modeling with the SIMPLIS command language, Chicago: Scientific Software International.
Keys, W. (1987). International studies in pupil performance: Aspects of science education in English schools, Windsor: NFER-Nelson.
Klieme, E., & Baumert, J. (2001). Identifying national cultures of mathematics education: Analysis of cognitive demands and differential item functioning in TIMSS. European Journal of Psychology of Education, 15(3), 385 - 402.
Kline, R. B. (2005). Principles and practice of structural equation modeling. New York: Guilford.
Koballa, T. R. (1995). Children's attitudes toward learning science. In S. Glynn and R. Duit (Eds.), Learning science in the schools: Research reforming practice (pp. 59-84). Mahwah, NJ: Erlbaum.
Lemke, M., Sen, A., Pahlke, E., Partelow, L., Miller, D., Williams, T., Kastberg, D., & Jocelyn, L. (2004). International outcomes of learning in mathematics literacy and problem solving: PISA 2003 results from the U.S. perspective. Education Statistics Quarterly, 6 (4), 20-25.
Marsh, H. W. (1994). Confirmatory factor analysis models of factorial invariance: A multifaceted approach. Structural Equation Modeling, 1, 5-34.
Marsh, H. W., & Yeung, A. S. (1996). The distinctiveness of affects in specific school subjects: An application of confirmatory factor analysis with the National Educational Longitudinal Study of 1988. American Educational Research Journal, 33, 665-689.
Marsh, H. W., Balla, J. R., & McDonald, R. P. (1988). Goodness-of-fit indices in confirmatory factor analysis: The effect of sample size. Psychological Bulletin, 102, 391-410.
Marsh, H. W., Hau, K. T., & Grayson, D. (2005). Goodness of fit evaluation in structural equation modeling. In A. Maydeu-Olivares & J. McCardle (Eds.), Contemporary psychometrics. A festschrift to Roderick P. Mc-Donald (pp. 275-340). Mahwah, NJ: Lawrence Erlbaum Associates, Inc.
Marsh, H. W., Hau, K. T., Artelt, C., Baumert, J., & Peschar, J. L. (2006). OECD's brief self-report measure of educational psychology's most useful affective constructs: Cross-cultural, psychometric comparisons across 25 countries. International Journal of Testing, 6(4), 311-360.
Milli Eğitim Bakanlığı Eğitimi Araştırma ve Geliştirme Dairesi Başkanlığı ([EARGED] 2007). PISA 2006 uluslar arası öğrenci başarılarını değerlendirme programı ulusal ön rapor [PISA 2006 the program for international student assessment national brief report]. Retrieved June 30 2008 from http://earged.meb.gov.tr/pisa/dil/tr/pisa2006.html.
Neidorf, T. S., Binkley, M., Gattis, K., & Nohara, D. (2006). Comparing mathematics content in the national assessment of educational progress (NAEP), trends in international mathematics and science study (TIMSS), and program for international student assessment (PISA) 2003 assessments (NCES 2006-029). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved June 30 2008 from ttp://nces.ed.gov/pubsearch.
Oliver, J. S., & Simpson, R. D. (1988). Influences of attitude toward science, achievement motivation, and science self concept on achievement in science: A longitudinal study. Science Education, 72, 143-155.
OECD (2006). Assessing Scientific, reading and mathematical literacy. Paris: OECD Publications.
OECD (2007). Science competencies for tomorrow’s world volume 1: Analysis. Paris: OECD Publications.
Osborne, J., Simon, S., & Collins, S. (2003). Attitudes towards science: A review of the literature and its implications. International Journal of Science Education, 25(9), 1049-1079.
Pae, T. I., & Park, G. P. (2006). Examining the relationship between differential item functioning and differential test functioning. Language Testing, 23, 475-496.
Piburn, M. D. (1993). If I were the teacher . . . qualitative study of attitude toward science. Science Education, 77, 393-406.
Poortinga, Y. H. (1989). Equivalence of cross-cultural data: An overview of basic issues. International Journal of Psychology, 24, 737-756.
Ramsden, J. M. (1998). Mission impossible?: Can anything be done about attitudes to science? International Journal of Science Education, 20(2), 125-137.
Robin, F., Sireci, S. G., & Hambleton, R. K. (2003). Evaluating the equivalance of different language versions of a credentialing exam. International Journal of Testing, 3(1), 1-20.
Schulz, W. (2005a, April). Testing parameter invariance for questionnaire indices using confirmatory factor analysis and item response theory. Paper presented at the Annual Meetings of the American Educational Research Association, San Francisco, USA, 7-11 April.
Schulz, W. (2005b, April). Mathematics self-efficacy and student expectations: Results from PISA 2003. Paper presented at the Annual Meetings of the American Educational Research Association, Montreal, USA, 11-15 April.
Tempelaar, D., van der Loeff, S. S., & Gijselaers, W. (2007). A structural equation model analyzing the relationship of students’ attitudes toward statistics, prior reasoning abilities, and course performance. Statistics Education Research Journal, 6, 78-102.
Valbuena, N. (2003). An empirical comparison of measurement equivalence methods based on confirmatory factor analysis (with mean and covariance structures analysis) and item response theory. Unpublished doctoral dissertation, The Graduate College of Illinois Institute of Technology, Chicago, USA.
Van Aalst, H. F. (1985). A model of interest, motivation, and learning. In M. Lehrke, L. Hoffman and P. L. Gardner (eds), Interests in science and technology education: Conference proceedings (Kiel: IPN), 49-57.
Yıldırım, H. H. (2006). The differential item functioning (dif) analysis of mathematics items in the international assessment programs. Unpublished doctoral dissertation, The Graduate School of Natural and Applied Sciences of Middle East Technical University, Ankara, Turkey.
Zhu, W., & Kang, S. J. (1998). Cross-cultural stability of the optimal categorization of a Self-efficacy Scale: A rasch analysis. Measurement in Physical Education and Exercise Science, 2(4), 225-241.