Gaining a Better Understanding of General Mattering Scale: An Application of Classical Test Theory and Item Response Theory

The current study shows applications of both classical test theory (CTT) and item response theory (IRT) to psychology data. The study discusses item level analyses of General Mattering Scale produced by two theories as well as strengths and weaknesses of both measurement approaches. The survey consisted of a total of five Likert-type items. Each student chose the best answers from the given categories (e.g., not at all, a little, somewhat, very much). We specifically run generalized partial credit IRT model. Overall, we discussed that while CTT provides comparatively superficial information but, IRT allows gaining deeper insight into the test items and response categories. We concluded that the meaning of item properties differ in CTT and IRT, and this difference may lead to different interpretations. We aimed to encourage psychologists and counsellors to give more consideration to the IRT applications when assessing the psychometric features of the items.

Anahtar Kelimeler:

Classical test theory, Item response theory, Generalized partial model, Item analysis, General mattering scale

Gaining a Better Understanding of General Mattering Scale: An Application of Classical Test Theory and Item Response Theory

Keywords:

Classical test theory, item response theory, generalized partial model, item analysis, general mattering scale,

PDF

___

Arbuckle, J. L. (2014). Amos 23.0 user's guide. Chicago, IL: IBM SPSS.
Baker F. (1992). Item response theory. New York, NY: Marcel Dekker, INC.
Brodin, U., Fors, U., & Laksov, K. B. (2010). The application of item response theory on a teaching strategy profile questionnaire. BMC medical education, 10, 14-14- doi:10.1186/1472-6920-10-14
Cha, M. (2016). The mediation effect of mattering and self-esteem in the relationship between socially prescribed perfectionism and depression: Based on the social disconnection model. Personality and Individual Differences, 88, 148-159. doi:10.1016/j.paid.2015.09.008
Cho, S., Drasgow, F., & Cao, M. (2015). An investigation of emotional intelligence measures using item response theory. Psychological Assessment, 27, 1241–1252. doi:10.1037/pas0000132
Connolly, K. M., & Myers, J. E. (2003). Wellness and mattering: The role of holistic factors in job satisfaction. Journal of Employment Counseling, 40, 152-160. doi:10.1002/j.2161-1920.2003.tb00866.x
Crocker, L. M., & Algina, J. (1986). Introduction to classical and modern test theory. New York, NY: Holt, Rinehart and Winston.de Ayala, R. J. (2009). The theory and practice of item response theory. New York, NY: Guilford.
Demir, M., Özen, A. & Doğan, A. (2012). Friendship, perceived mattering and happiness: A study of American and Turkish college students. The Journal of Social Psychology, 152, 659-664. doi: 10.1080/00224545.2011.650237
Dodeen, H., & Al-Darmaki, F. (2016). The application of item response theory in developing and validating a shortened version of the Emirate Marital Satisfaction Scale. Psychological Assessment, 28, 1625-1633. doi:10.1037/pas0000296.supp
Embretson S.E, & Reise, S.P. (2000). Item response theory for psychologists. Mahwah, IN: Lawrence Erlbaum Associates, Inc.
Flett, G. L., Galfi-Pechenkov, I., Molnar, D. S., Hewitt, P. L., & Goldstein, A. L. (2012). Perfectionism, mattering, and depression: A mediational analysis. Personality and Individual Differences, 52, 828-832. doi:10.1016/j.paid.2011.12.041
Gomez, R., & Vance, A. (2015). Item response theory properties of the internalizing disorders in adolescents. Journal of Childhood & Developmental Disorders, 1, 1-10. doi:10.4172/2472-1786.100006
Haktanir, A., Lenz, A. S., Can, N., & Watson, J. C. (2016). Development and evaluation of Turkish language versions of three positive psychology assessments. International Journal for the Advancement of Counselling, 38, 286-297. doi:10.1007/s10447-016-9272-9
Hays, R. D., Morales, L. S., & Reise, S. P. (2000). Item response theory and health outcomes measurement in the 21st century. Medical Care, 38, II28-II42. doi:10.1097/00005650-200009002-00007
Huggins-Manley, A. C., & Algina, J. (2015). The partial credit model and generalized partial credit model as constrained nominal response models, with applications in M Plus. Structural Equation Modeling: A Multidisciplinary Journal, 22, 308-318. doi:10.1080/10705511.2014.937374
IBM Corp. (2013). IBM SPSS Statistics for Windows, Version 22.0. Armonk, NY: IBM Corp.
Karaman, M. A., Balkin, R. S., & Juhnke, G. A. (2018). Turkish adaptation of the Juhnke–Balkin Life Balance Inventory. Measurement and Evaluation in Counseling and Development, 51, 141–150. doi:10.1080/07481756.2017.1308226
Lee, Y. S., Krishnan, A., & Park, Y. S. (2012). Psychometric properties of the Children’s Depression Inventory: An item response theory analysis across age in a nonclinical, longitudinal, adolescent sample. Measurement and Evaluation in Counseling and Development, 45, 84-100. doi: 10.1177/0748175611428329
Locke, B. D., McAleavey, A. A., Zhao, Y., Lei, P. W., Hayes, J. A., Castonguay, L. G., ... & Lin, Y. C. (2012). Development and initial validation of the Counseling Center Assessment of Psychological Symptoms–34. Measurement and Evaluation in Counseling and Development, 45, 151-169. doi: 10.1177/0748175611432642
Lord, F.M. (1980). Applications of Item Response Theory to Practical Testing Problems. Hillsdale, NJ: Lawrence Erlbaum.
Marcus, F. M. (1991). Mattering: Its measurement and theoretical significance for social psychology. Paper presented at the annual meeting of the Eastern Sociological Association, Cincinnati.
Masters, G. (1982). A Rasch model for partial credit scoring. Psychometrika, 47, 149–174.
Muraki, E. (1992). A generalized partial credit model: Application of an EM algorithm. Applied Psychological Measurement, 16, 159-176.
Rayle, A. D. (2006). Mattering to others: Implications for the counseling relationship. Journal of Counseling & Development, 84, 483-487. doi:10.1002/j.1556-6678.2006.tb00432.x
R Core Team (2016). R: A language and environment for statistical computing [computer software manual]. Vienna, Austria. Retrieved from http://www.R-project.org/
Riemer, M., & Kearns, M. A. (2010). Description and psychometric evaluation of the Youth Counseling Impact Scale. Psychological Assessment, 22, 259-268. doi:10.1037/a0018507.
Rizopoulos, D. (2017). Package “ltm”, Latent Trait Models under IRT. Retrieved from https://cran.r-project.org/web/packages/ltm/ltm.pdf
Rosenberg, M., & McCullough, B. C. (1981). Mattering: Inferred significance and mental health among adolescents. Research in Community & Mental Health, 2, 163-182.
Samejima, F. (1969). Estimation of latent ability using a response pattern of graded scores. Psychometrika, 34, 1–97. doi:10.1007/bf03372160 Tasca, G. A., Cabrera, C., Kristjansson, E., MacNair-Semands, R., Joyce, A. S., & Ogrodniczuk, J. S. (2016). The therapeutic factor inventory-8: Using item response theory to create a brief scale for continuous process monitoring for group psychotherapy. Psychotherapy Research, 26, 131-145. doi:10.1080/10503307.2014.963729
Tovar, E., Simon, M. A., & Lee, H. B. (2009). Development and validation of the college mattering inventory with diverse urban college students. Measurement and Evaluation in Counseling and Development, 42, 154-178. doi:10.1177/0748175609344091
Tovar, E. (2013). A conceptual model on the impact of mattering, sense of belonging, engagement/involvement, and socio-academic integrative experiences on community college students’ intent to persist (Doctoral Dissertation). Retrieved from CGU Theses & Dissertations (Paper 81). doi: 10.5642/cguetd/81
Turner, B. M., Betz, N. E., Edwards, M. C., & Borgen, F. H. (2010). Psychometric examination of an inventory of self-efficacy for the Holland vocational themes using item response theory. Measurement and Evaluation in Counseling and Development, 43, 188-198. doi:10.1177/0748175610384810
Zanon, C., Hutz, C. S., Yoo, H. H., & Hambleton, R. K. (2016). An application of item response theory to psychological test development. Psicologia: Reflexão e Crítica, 29, 1-10. doi:10.1186/s41155-016-0040-x