A Study on Detecting Differential Item Functioning of PISA 2006 Science Literacy Items in Turkish and American Samples

Problem Statement: Item bias occurs when individuals from differentgroups (different genders, cultural backgrounds, etc.) have differentprobabilities of responding correctly to a test item despite having the sameskill levels. It is important that tests or items do not have bias in order toensure the accuracy of decisions taken according to test scores. Thus, itemsshould be tested for bias during the process of test development andadaptation. Items used in testing programs, such as the Program forInternational Student Assessment (PISA) study, whose results informeducational policies throughout the participating countries, should bereviewed for bias. The study examines whether item bias of the 2006 PISAscience literacy test, in Turkish sample versus American sample,Purpose of the Study: The aim of this study is to analyze the measurementequivelance of the PISA science literacy test of 2006 in Turkish andAmerican groups in terms of structural invariance and also to determinewhether the science literacy items include inter-cultural bias.Methods: The study included data for 757 Turkish and 856 American 15-year-old students. Exploratory factor analysis (EFA) and confirmatoryfactor analysis (CFA) tests were performed to determine whether the PISAscience literacy test was equivalent in measurement construct in bothgroups; multi group confirmatory factor analysis (MCFA) was used toidentify differences in the factor structure according to cultures.Differential item functioning (DIF) was detected via the Mantel–Haenszel (MH), Simultaneous Item Bias Test (SIBTEST), and Item Response TheoryLikelihood- Ratio Analysis (IRT-LR) procedures.Findings and Results: : According to the MCFA results PISA 2006 scienceliteracy test, both Turkish and American groups showed equivalentmeasurement constructs. Moreover, the three analyses methods agreed atB and C levels for 15 items in the Turkish sample and 25 items in theAmerican sample in terms of DIF. According to expert opinions, commonsources for item bias were: familiarity with item content and differing skilllevels between cultures.Conclusions and Recommendations: The 38 items that showed DIF by each ofthe three methods were accepted as having DIF. The findings of thepresent study, that there is a possible source of bias in the items, will notchange the average level of student performance in participatingcountries. However, it will be beneficial to review item content before testadministration, in order to reduce the erroneous items with DIF acrossdifferent language and cultural groups in international comparativestudies.

___

  • Abe, J. A. (2004). Shame, guilt, and personality judgment. Journal of Research in Personality, 38, 85–104.
  • Akba?, M., & Erden-?mamo?lu, S. (2010). Cinsiyet ve ba?lanma stillerinin utanç, suçluluk ve yalnzlk duygularn yordama gücünün ara?trlmas. [The prediction of gender and attachment styles on shame, guilt, and loneliness]. Educational Sciences: Theory & Practice, 10 (2), 651-682.
  • Barrett, K. C. (1995). A functionalist approach to shame and guilt. In J. P. Tangney & K. W. Fischer (Eds.), Self-conscious emotions: The psychology of shame, guilt, embarrassment, and pride (pp. 25-63). New York, NY: Guilford Press.
  • Baumeister, R. F., Stillwell, A. M., & Heatherton, T. F. (1995). Interpersonal aspects of guilt: Evidence from narrative studies. In J. P. Tangney, & K. W. Fischer (Eds.), Self-conscious emotion: The psychology of shame, guilt, embarrassment, and pride (pp. 255–273). New York, NY: Guilford Press.
  • Bedford, O., & Hwang, K. (2003). Guilt and shame in Chinese culture: A crosscultural framework from the perspective of morality and identity. Journal for the Theory of Social Behavior, 33, 127-144.
  • Benetti-McQuoid, J., & Bursik, K. (2005). Individual differences in experiences of and responses to guilt and shame: Examining the lenses of gender and gender role. Sex Roles, 53 (1-2), 133-142.
  • Bybee, J., & Quiles, Z. N. (1998). Guilt and mental health. In J. Bybee (Ed.), Guilt and children (pp. 269–291). New York, NY: Academic Press.
  • Costa, P. T. Jr., & McCrae. R. R. (1992). Revised NEO Personality Inventory (NEO-PI-R) and NEO Five Factor Inventory (NEO-FFI) professional manual. Odessa. FL: Psychological Assessment Resources.
  • Costa, P. T., & Widiger, T. A. (1994). Personality disorders and the five-factor model of personality. Washington, DC: American Psychological Association.
  • Cross, S. E., & Madson, L. (1997). Models of the self: Self-construal theory and gender. Psychological Bulletin, 122, 5-37.
  • Ek?i, H. (2004). Ki?ilik ve ba?açkma: Ba?açkma tarzlarnn durumsal ve e?ilimsel boyutlar üzerine çok yönlü bir ara?trma. [Personality and coping: A multidimensional research on situational and dispositional coping]. Kuram ve Uygulamada E?itim Bilimleri/ Educational Sciences: Theory & Practice, 4 (1), 94- 98.
  • Einstein, D., & Lanning, K. (1998). Shame, guilt, ego development, and the five-factor model of personality. Journal of Personality, 66, 555–582.
  • Fossum, M. A., & Mason, M. J. (1986). Facing shame: Families in recovery. New York, NY: Norton.
  • Gilbert, P., Pehl, J., & Allan, S. (1994). The phenomenology of shame and guilt: An empirical investigation. British Journal of Medical Psychology, 67, 23-36.
  • Göktan, B., & Akba?, M. (2010). An investigation on Turkish military school students: Are there associations among big five personality factors, perceived family environment and hopelessness?, Procedia - Social and Behavioral Sciences, 2 (2), 5458-5462.
  • Graziano, W. G., & Eisenberg, N. (1997). Agreeableness: A dimension of personality. In R. Hogan, J. A. Johnson, & S. R. Briggs (Eds.), Handbook of personality psychology (pp. 795-824). San Diego, CA: Academic Press.
  • Graziano, W. G., Jensen-Campbell, L. A., & Hair, E. C. (1996). Perceiving interpersonal conflict and reacting to it: The case for agreeableness. Journal of Personality and Social Psychology, 70, 820-835.
  • Graziano, W. G., & Ward, D. (1992). Probing the Big Five in adolescence: Personality and adjustment during a developmental transition. Journal of Personality, 60, 425–429.
  • Gülgöz, S. (2002). Five-factor model and NEO-PPI in Turkey. In R.R. McCrae, & Allik, J. (Eds.). The five factor model of personality across cultures (pp. 175-195). New York, NY: Kluwer Academia /Plenum Publishers.
  • Harder, D. W. (1995). Shame and guilt assessment, and relationships of shame- and guilt-proneness to psychopathology. In J. P. Tangney, & K. W. Fischer (Eds.), The self-conscious emotions (pp. 368–392). New York, NY: Guilford Press.
  • Harder, D. W., & Greenwald, D. F. (1999). Further validation of the shame and guilt scales of the Harder Personal Feelings Questionnaires-2. Psychological Reports, 85, 271–281.
  • Hoffmann, M. L. (1998). Varieties of empathy-based guilt. In J. Bybee (Ed.), Guilt and children (pp. 91-112). Boston, MA: Academic Press.
  • Hofstede, G. (1991). Cultures and organizations: Software of the mind. London: McGraw- Hill.
  • Izard, C. E. (1978). Guilt, conscience and morality. Human emotions (pp. 421-452). New York, NY: Plenum Press.
  • Johnson, J. A., & Ostendorf. F. (1993). Clarification of the Five-factor-model with the abridged big Five dimensional circumflex. Journal of Personality and Social Psychology. 65, 563–576.
  • Jones, W. H., Kugler, K., & Adams, P. (1995). You always hurt the one you love: Guilt and transgressions against relationship partners. In J. P. Tangney, & K. W.
  • Fischer (Eds.), Self-conscious emotions: The psychology of shame, guilt, embarrassment, and pride (pp. 274–300). New York, NY: Guilford Press.
  • John, O. P., Caspi, A., Robins, R. W., Moffitt, T. E., & Stouthamer-Loeber, M. (1994). The ‘‘little five’’: Exploring the nomological network of the five-factor model of personality in adolescent boys. Child Development, 65, 160–178.
  • Jung, H. (2002). Shame as a mediator between parenting and psychological adjustment for Causian and Korean students: A structural equation modeling approach (Unpublished dissertation), Biola University, USA.
  • Ka?tçba?, Ç. (2010). Benlik, aile ve insan geli?imi: Kültürel psikoloji. [Family, self, and human: Development across cultures]. ?stanbul: Koç University Publishing.
  • Ka?tçba?, Ç., & Ataca, B. (2005). Value of children and family change: A three decade portrait from Turkey. Applied Psychology: International Review, 54, 317- 337.
  • Kurt, E. (2001). Effects of religiosity, personality traits and social responsibility attitudes on nonspontaneous helping behavior in a natural disaster (Unpublished master thesis). Bosporus University, Istanbul, Turkey.
  • Lewis, H. B. (1971). Shame and guilt in neurosis. New York: International Universities Press.
  • Lewis, H. B. (1987). Introduction: Shame – The “sleeper” in psychopathology. In H. B. Lewis (Ed.), The role of shame in symptom formation (pp. 1-28). Hillsdale, NJ: Erlbaum.
  • Lewis, M., Alessandri, S. M., & Sullivan, M. W. (1992). Differences in shame and pride as a function of children’s gender and task difficulty. Child Development, 63, 630-638.
  • Lindsay-Hartz, J., De Riviera, J., & Mascolo, M. F. (1995). Differentiating guilt and shame and their effects on motivation. In J. P. Tangney & K. W. Fischer (Eds.), Self conscious emotions: The psychology of shame, guilt, embarrassment, and pride (pp.174-300). New York, NY: Guilford Press.
  • McCrae, R. R. (1996). Social consequences of experiential openness. Psychological Bulletin, 120, 323-337.
  • McCrae, R. R. (2002). NEO-PPI data from 36 cultures: Further intercultural comparisons. In R.R. McCrae & J. Allik (Eds.), The five factor model of personality across cultures (pp. 105-125). New York, NY: Kluwer Academic/Plenum Publishers.
  • McCrae, R. R., & Costa, P. T. (1997). Personality trait structure as a human universal. American Psychologist, 52, 509–516.
  • Nathanson, D. L. (1992). Shame and pride: affect, sex, and the birth of the self (1st ed.). New York, NY: Norton.
  • Niedenthal, P. M., Tangney, J. P., & Gavanski, I. (1994). “If only I weren’t” versus “If only I hadn’t”: Distinguishing shame and guilt in counterfactual thinking. Journal of Personality and Social Psychology, 67, 585–595.
  • O’Connor, L. E., Berry, J. W., & Weiss, J. (1999). Interpersonal guilt, shame, and psychological problems. Journal of Social and Clinical Psychology, 18, 181–203.
  • Oyserman, D., Coon, H.M., & Kemmelmeier, M. (2002). Rethinking individualism and collectivism: Evaluation of theoretical assumptions and meta-analyses. Psychological Bulletin, 128 (1), 3-72.
  • Penley, J.A., & Tomaka, J. (2002). Associations among the Big Five, emotional responses, and coping with acute stress. Personality and Individual Differences, 32, 1215-1228.
  • Roberts, B. W., & DelVecchio, W. F. (2000). The rank-order consistency of personality traits from childhood to old age: A quantitative review of longitudinal studies. Psychological Bulletin, 126, 3–25.
  • Robins, R. W., John, O. P., & Caspi, A. (1998). The typological approach to studying personality development. In R. B. Cairns, L. Bergman, & J. Kagan (Eds.), Method and models for studying the individual (pp. 135–160). Thousand Oaks, CA: Sage.
  • Robins, R. W., Hendin, H. M., & Trzesniewski, K. H. (2001). Measuring global selfesteem: Construct validation of a single item measure and the Rosenberg Self- Esteem scale. Personality and Social Psychology Bulletin, 27, 151–161.
  • Sava?r. I., & ?ahin. N. H. (1997). Bili?sel-davran?ç terapilerde de?erlendirme: Sk kullanlan ölçekler. [The evaluation of cognitive-behavioral therapy: The scales used frequently]. Ankara: TPD Yaynlar.
  • Sayar, K. (2003). Kültürel bak? açsndan benlik ve ki?ilik. [Self and personality from cultural view]. Yeni Symposium, 41(2), 78-85.
  • Smith, B. D., Hanges, J., & Dickson, M.W. (2001). Personnel selection and Five-Factor Model: Reexamining the effects of applicant's frame of reference. Journal of Applied Psychology, 86 (2), 304-315.
  • Smith, C. A., & Lazarus, R. S. (1993). Appraisal components, core relational themes, and the emotions. Cognition and Emotion, 7, 233-269.
  • Schore, A. N. (1998). Early shame experiences and infant brain development. In P. Gilbert & B. Andrews (eds.), Shame: Interpersonal behavior, psychopathology, and culture (pp. 57-77). New York, NY: Oxford University Press.
  • Tangney, J. P. (1991). Moral affect: The good, the bad, and the ugly. Journal of Personality and Social Psychology, 61, 598– 607.
  • Tangney, J. P. (1995). Shame and guilt in interpersonal relationships. In J. P. Tangney, & K. W. Fischer (Eds.), Self-conscious emotions: Shame, guilt, embarrassment, and pride (pp. 114–139). New York, NY: Guilford Press.
  • Tangney, J. P., Burgraff, S. A., & Wagner, P. E. (1995). Shame-proneness, guiltproneness, and psychological symptoms. In J. P. Tangney, & K. W. Fischer (Eds.), The self-conscious emotions (pp.343–367). New York, NY: Guilford Press.
  • Tangney, J. P., & Dearing, R. L. (2002). Shame and guilt. New York, NY: Guilford Press.
  • Tangney, J. P., Miller, R. S., Flicker, L., & Barlow, D. H. (1996). Are shame, guilt, and embarrassment distinct emotions? Journal of Personality and Social Psychology, 6, 1256-1269.
  • Tangney, J. P., Wagner, P. E., Fletcher, C., & Gramzow, R. (1992). Shamed into anger? The relation of shame and guilt to anger and self-reported aggression. Journal of Personality and Social Psychology, 62, 669–675.
  • Taylor, G. (1985). Pride, shame and guilt. Oxford: Oxford University Press. Triandis, H. C. (1994). Recherches récentes sur l’individualisme et le collectivisme [Recent research on individualism and collectivism]. Les Cahiers Internationaux de Psychologie Sociale, 23, 14-27.
  • Tong, E. M., Bishop, G. D., Enkelmann, H. C., Why, Y. P., Diong, S. M., Ang, J., & Khader, M. (2006). The role of the Big Five in appraisals. Personality and Individual Differences, 41, 513–523.
  • Woien, S. L., Ernst, H. A. H., Patock-Peckham, C., & Nagoshi, C. T. (2003). Validation of the TOSCA to measure shame and guilt. Personality and Individual Differences, 35, 313-326.
  • Zhong, J., Li, B., & Qian, M. (2002). Esteem in the personality, shame and mental health model: Its direct and moderating effects (in Chinese). Chinese Journal of Clinical Psychology, 10, 241–245.