The Effects of Log Data on Students’ Performance

This study aimed to assess the relationships between response times (RTs), the number of actions taken to solve a given item, and student performance. In addition, the interaction between the students’ information and communications technology (ICT) competency, reading literacy, and log data (time and number of actions) were examined in order to gain additional insights regarding the relations between student performance and log data. The sample consisted of 2 348 students who participated in the triennial international large-scale assessment of the Programme for International Student Assessment (PISA). For the current study, 18 items in the one cluster of the 91st booklet were chosen. To achieve the aim of the study, explanatory item response modeling (EIRM) framework based on generalized linear mixed modeling (GLMM) was used. The results of this study showed that students who spent more time on items and those that took more actions on items were more likely to answer the items correctly. However, this effect did not have variability across items and students. Moreover, the interaction only with reading and the number of actions was found to have a positive effect on the students’ overall performance.

PDF

___

Azzolini, D., Bazoli, N., Lievore, I., Schizzerotto, A., & Vergolini, L. (2019). Beyond achievement. A comparative look into 15-year-olds’ school engagement, effort and perseverance in the European Union. Luxembourg: Publications Office of the European Union.
Bates, D., Maechler, M., & Bolker, B. (2012). lme4: Linear mixed-effects models using S4 classes, 2011[R package version 0.999375-42]. Retrieved from http://CRAN.R-project.org/package=lme4.
Briggs, D. C. (2008). Using explanatory item response models to analyze group differences in science achievement. Applied Measurement in Education, 21(2), 89-118. doi: 10.1080/08957340801926086
De Ayala, R. J. (2009). The theory and practice of Item Response Theory. New York, NY: The Guilford Press.
De Boeck, P., & Wilson, M. (Eds.). (2004). Explanatory item response models: A generalized linear and nonlinear approach. New York, NY: Springer. doi: 10.1007/978-1-4757-3990-9
De Boeck, P., Bakker, M., Zwitser, R., Nivard, M., Hofman, A., Tuer- linckx, F., & Partchev, I. (2011). The estimation of item response models with the lmer function from the lme4 package in R. Journal of Statistical Software, 39(12), 1-28. doi: 10.18637/jss.v039.i12
Desjardins, C. D., & Bulut, O. (2018). Handbook of educational measurement and psychometrics using R. New York, NY: Chapman and Hall/CRC.
Dodonova Y. A., & Dodonov Y. S. (2013). Faster on easy items, more accurate on difficult ones: Cognitive ability and performance on a task of varying difficulty. Intelligence, 41(1), 1-10. doi: 10.1016/j.intell.2012.10.003
Goldhammer, F., & Klein-Entink, R. (2011). Speed of reasoning and its relation to reasoning ability. Intelligence, 39(2-3), 108-119. doi: 10.1016/j.intell.2011.02.001
Goldhammer, F., Naumann, J., & Greiff, S. (2015). More is not always better: The relation between item response and item response time in Raven’s matrices. Journal of Intelligence, 3(1), 21-40. doi: 10.3390/jintelligence3010021
Goldhammer, F., Naumann, J., Stelter, A., Tóth, K., Rölke, H., & Klieme, E. (2014). The time on task effect in reading and problem solving is moderated by task difficulty and skill: Insights from a computer-based large-scale assessment. Journal of Educational Psychology, 106(3), 608-626. doi: 10.1037/a0034716
Greiff, S., Niepel, C., Scherer, R., & Martin, R. (2016). Understanding students’ performance in a computer-based assessment of complex problem solving: An analysis of behavioral data from computer-generated log files. Computers in Human Behavior, 61(2016), 36-46. doi: 10.1016/j.chb.2016.02.095
Greiff, S., Wüstenberg, S., & Avvisati, F. (2015). Computer-generated log-file analyses as a window into students’ minds? A showcase study based on the PISA 2012 assessment of problem solving. Computers & Education, 91(2015), 92-105. doi: 10.1016/j.compedu.2015.10.018
Greiff, S., Wüstenberg, S., Csapo, B., Demetriou, A., Hautamaki, J., Graesser, A. C., Martin, R. (2014). Domain-general problem-solving skills and education in the 21st century. Educational Research Review, 13(2014), 74-83. doi: 10.1016/j.edurev.2014.10.002
He, Q., von Davier, M., & Han, Z. (2018). Exploring process data in problem-solving items in computer-based large-scale assessments. In H. Jiao, W. R. Lissitz, & A. Van Wie (Eds.), Data analytics and psychometrics: Informing assessment practices (pp. 53-76). Charlotte, NC: Information Age Publishing.
Herborn, K., Stadler, M., Mustafić, M., & Greiff, S. (2018). The assessment of collaborative problem solving in PISA 2015: Can computer agents replace humans? Computers in Human Behavior. Online first. doi: 10.1016/j.chb.2018.07.035
Klein-Entink, R. H., Fox, J. P., & van der Linden, W. J. (2009). A multivariate multilevel approach to the modeling of accuracy and speed of test takers. Psychometrika, 74(1), 21-48. doi: 10.1007/s11336-008-9075-y
Kyllonen, P., & Zu, J. (2016). Use of response time for measuring cognitive ability. Journal of Intelligence, 4(4), 1-29. doi: 10.3390/jintelligence4040014
Lasry, N., Watkins, J., Mazur, E., & Ibrahim, A. (2013). Response times to conceptual questions. American Journal of Physics, 81(9), 703-706. doi: 10.1119/1.4812583
Lee, Y. H., & Haberman, S. J. (2016) Investigating test-taking behaviors using timing and process data. International Journal of Testing, 16(3), 240-267, doi: 10.1080/15305058.2015.1085385
Lee, Y. H., & Jia, Y. (2014). Using response time to investigate students’ test-taking behaviors in a NAEP computer-based study. Large-scale Assessments in Education, 2(8), 1-24. doi: 10.1186/s40536-014-0008-1
Organisation for Economic Cooperation and Development. (2017a). PISA 2015 assessment and analytical framework: Science, reading, mathematic, financial literacy and collaborative problem solving. Paris: OECD Publishing. doi: 10.1787/9789264281820-en
Organisation for Economic Cooperation and Development. (2017b). PISA 2015 results (Volume V): Collaborative problem solving. Paris: OECD Publishing.
Organisation for Economic Cooperation and Development. (2017c). PISA 2015 technical report. Paris: OECD Publishing.
Petscher, Y., Mitchell, A. M., & Foorman, B. R. (2015). Improving the reliability of student scores from speeded assessments: An illustration of conditional item response theory using a computer-administered measure of vocabulary. Reading and Writing, 28(1), 31-56. doi: 10.1007/s11145-014-9518-z
R Development Core Team (2018). R: A language and environment for statistical computing. Vienna, Austria: R Foundation for Statistical Computing.
Rasch, G. (1960). Probabilistic models for some intelligence and attainment tests. Copenhagen, Denmark: Danish Institute for Educational Research.
Rijmen, F., Tuerlinckx, F., De Boeck, P., & Kuppens, P. (2003). A nonlinear mixed model framework for item response theory. Psychological Methods, 8(2), 185-205. doi: 10.1037/1082-989X.8.2.185
Rios, J. A., Guo, H., Mao, L., & Liu, O. L. (2017). Evaluating the impact of careless responding on aggregated-scores: To filter unmotivated examinees or not? International Journal of Testing, 17(1),74-104. doi: 10.1080/15305058.2016.1231193
Robitzsch, A. (2019). Package “sirt”. Retrieved from https://cran.r-project.org/web/packages/sirt/sirt.pdf
Schnipke, D. L., & Scrams, D. J. (2002). Exploring issues of examinee behavior: Insights gained from response-time analyses. In N. C. Mills., M. T. Potenza, J. J. Fremer & C. W. Ward (Eds.), Computer-based testing: Building the foundation for future assessments (pp. 237-266). New York, NY: Psychology Press.
Su, S., & Davison, M. L. (2019) Improving the predictive validity of reading comprehension using response times of correct item responses. Applied Measurement in Education, 32(2), 166-182. doi: 10.1080/08957347.2019.1577247
van der Linden, W. J. (2009). Conceptual issues in response‐time modeling. Journal of Educational Measurement, 46(3), 247-272. doi: 10.1111/j.1745-3984.2009.00080.x
Verbić, S., & Tomić, B. (2009). Test item response time and the response likelihood. Retrieved from http://arxiv.org/ftp/arxiv/papers/0901/0901.4356.pdf
Wickham, H. (2012). reshape2: Flexibly reshape data: a reboot of the reshape package. Retrieved from http://cran.ms.unimelb.edu.au/web/packages/reshape2/
Wilson, M., De Boeck, P., & Carstensen, C. H. (2008). Explanatory item response models: A brief introduction. In J Hartig, E Klieme, & D. Leutner (Eds.), Assessment of competencies in educational contexts (pp. 91-120). Cambridge, MA: Hogrefe.
Wise, S. L. (2006). An Investigation of the differential effort received by items on a low-stakes computer-based test. Applied Measurement in Education, 19(2), 95-114. doi: 10.1207/s15324818ame1902_2
Wise, S. L., & DeMars, C. E. (2005). Low examinee effort in low stakes assessment: Problems and potential solutions. Educational Assessment, 10(1), 1-17. doi: 10.1207/s15326977ea1001_1
Wise, S. L., Kingsbury, G. G., Thomason, J., & Kong, X. (2004, April). An investigation of motivation filtering in a statewide achievement testing program. Paper presented at the annual meeting of the National Council on Measurement in Education, San Diego, CA.
Wu, A. D., Chen, M. Y., & Stone, J. E. (2018). Investigating how test-takers change their strategies to handle difficulty in taking a reading comprehension test: Implications for score validation. International Journal of Testing, 18(3), 253-275. doi: 10.1080/15305058.2017.1396464