Automated Quality Assurance of Educational Testing
This paper presents a study on known approaches for quality assurance of educational test and test items. On its basis a comprehensive approach to the quality assurance of online educational testing is proposed to address the needs of all stakeholders (authors of online tests, teachers, students, experts, quality managers, etc.). According to the proposed approach is developed an original software application Test Quality Evaluation (TQE) for the automation of the stakeholders’ activities for quality assurance of educational tests throughout the whole lifecycle. The application retrieves and provides analysis of data from online tests conducted and specially designed surveys for quality evaluation of educational tests by students and experts. It allows tracking and evaluating the quality of educational tests in real time and provides the related quantitative data in different levels of generalization – in the level of a separate educational test, of educational tests of an entire course, or educational tests of a subject area. The software application has been put under real-time testing for quality evaluation of educational tests, included in e-learning courses from different subject areas that prove its applicability.
___
- Amouei, A., Barari, R., Naghipour D., Mortazavi Y., & Hosseini S. Reza (2014). Evaluation
of Multiple Choice Questions Quality Trend as Structure and Taxonomy. FUTURE
of MEDICAL EDUCATION JOURNAL, 4(3), 26-30.
APA (2014). The Standards for Educational and Psychological Testing, Retrieved July 2,
2017, from http://www.apa.org/science/programs/testing/standards
aspx#overview
Blackboard Help (2017). Running Item Analysis on a Test, Retrieved July 2, 2017, from
https://en-us.help.blackboard.com
CDC (2017). Checklist to Evaluate the Quality of Questions, Retrieved April 12, 2017, from
http://www.cdc.gov/healthyyouth/ evaluation/ index.htm
CFATIQC (2017). Common Formative Assessment Test Item Quality Checklist, Retrieved
July 2, 2017, from https://docs.google.com/document/preview?hgd=1&id=
1vmkf0UAU21u8bGRRwtFb__4kw6NThEDM8WD9NyMgSFU&pli=1
CITL. (2017). Improving Your Test Questions. Retrieved July 2, 2017, from
http://cte.illinois.edu/testing/exam/test_ques3.html
91
Cronbach, L. J. (1971). Test validation. In R. L. Thorndike (ed.). Educational
Measurement, 2nd ed., 443–507. Washington, D.C.: American Council on
Education.
Dill, D. D. (2010). “Quality Assurance in Higher Education: Practices and Issues.” In P. P.
Peterson, E. Baker, and B. McGaw, (eds.). International Encyclopedia of
Education. Third Edition. 377-383.
Doneva, R., & Gaftandzhieva, S. (2015). Automated e-learning quality evaluation.
Proceedings of the International Conference on e-Learning. Berlin, Germany.
ISSN 2376-6698, 156-162.
EFPA (2013). Revised EFPA Review Model for the Description and Evaluation of
Psychological and Educational Tests, Test Review Model Version 4.2.6, Retrieved
July 2, 2017, from
http://www.efpa.eu/download/650d0d4ecd407a51139ca44ee704fda4
EUSHARE (2015). European Standards and Guidelines for Quality Assurance in the EHEA.
Belgium: EUSHARE.
Gaftandzhieva S. (2016). Automated Evaluation of Students’ Satisfaction, International
Journal of Information Technologies and Security (IJITS), 8(1), 31-40.
Gaftandzhieva S. (2017). A Model and System for Dynamic Quality Evaluation in Higher
Education. (Doctoral dissertation). Available from University of Plovdiv.
Gierl, M.J., & Hollis, L. (2013). Evaluating the quality of medical multiple-choice items
created with automated processes, Medical Education 2013, 47(7), 726–733, doi:
10.1111/medu.12202.
Hambleton, RK, Swaminathan, H, Rogers, HJ (1991). Fundamentals of items response
theory. Newbury Park (California): Sage Publications. 174 p.
Hamilton, L. S., Stecher, Br. M.,& Klein St.P. (2002). Making Sense of Test-Based
Accountability in Education. RAND.
ISO 9000:2015 (2015). Quality management systems — Fundamentals and vocabulary,
Retrieved July 2, 2017, from https://www.iso.org/obp/ui/#iso:std:iso:9000:ed4:v1:en
JasperSoft (2017). Business Intelligence Solutions. Retrieved July 2, 2017, from
http://www.jaspersoft.com/business-intelligence-solutions
Legault N. (2017). Post-Course Evaluations for е-Learning: 60+Questions to Include,
Retrieved July 2, 2017, from https://community.articulate.com/articles/postcourse-evaluations-for-e-learning-60-questions-to-include
Machado-da-Silva, F, Meirelles, F , Filenga, D , Filho, M . (2015). Student Satisfaction
Process In Virtual Learning System: Considerations Based In Information And
Service Quality From Brazil’s Experience. Turkish Online Journal of Distance
Education, 15 (3), 122-142. DOI: 10.17718/tojde.52605.
Mark, D. R. (1985). The Difficulty of Test Items That Measure More Than One Ability,
Applied Psychological Measurement, 9(4), 401 – 412.
Messick, S. V. (1989). Educational Measurement. 3rd ed., 13–103. New York: Macmillan.
Moodle Documentation (2017). Quiz statistics report, Retrieved July 2, 2017, from
https://docs.moodle.org/29/en/Quiz_statistics_report.
Mutiara, D., Zuhairi, A., & Kurniati S. (2007). Designing, Developing, Producing And
Assuring The Quality of Multi-Media Learning Materials For Distance Learners:
Lessons Learnt From Indonesia's Universitas Terbuka. Turkish Online Journal of
Distance Education-TOJDE, ISSN 1302–6488, 8(2), 95-112.
92
Professional Testing (2017). How do you Determine if a Test has Validity, Reliability,
Fairness, and Legal Defensibility?. Retrieved July 2, 2017, from
http://www.proftesting.com/test_topics/test_quality.php
Pyrczak, F. (1973). Validity of the Discrimination Index as a Measure of Item Quality.
Journal of Educational Measurement, 10(3), 227-231.
Rasch (2017). Rasch Software, Retrieved July 2, 2017, from
https://www.rasch.org/rmt/rmt114d.htm
Saad, S., Carter, G.W., Rothenberg, M., & Israelson, E. (1999). Testing and Assessment:
an Employer’s Guide to Good Practises, Employment and Training Administration
(DOL). Washington, DC. Office of Policy and Research.
Thompson, B., & Levitov, J.E. (1985). Using microcomputers to score and evaluate test
items. Collegiate Microcomputer, 3, 163-168.
Totkov, G., Gaftandzhieva, S., & Doneva, R. (2016). Dynamic Quality Evaluation in Higher
Education (with application in e-Learning). Proceedings of the First Varna
Conference on E-learning and Knowledge Management: Bridging the Gap
between Secondary and Higher Education, 8-23.
Totkov, G., Raikova, М., & Kostadinova, H. (2014). The test in e-learning. Plovdiv:
“Rakursi” LTD.