Holistic Rubric Validity and Reliability in Essay Assessment Using Rasch Model in Blended Learning Program

Holistic Rubric Validity and Reliability in Essay Assessment Using Rasch Model in Blended Learning Program

• By Yenni Arif Rahman, Nurhayati S., Fiza Asri Fauziah Habibah, and Fadilah Fadilah;
• Seminar Internasional
• 180
• 685-698

The covid-19 pandemic has forced teachers to adapt with learning method suitable with distance learning, and blended learning program is one of the best options around. Then the necessary assessment of output of this method should be available, valid, and reliable. This study aims to determine the validity and reliability of a holistic scoring rubric developed by Jacob et al. to assess students’ writing in the Blended Learning program by using Rasch model during Covid-19 pandemic. The participants were 55 EFL learners who had taken essay writing course in TOEFL iBT class whereas the writing samples were taken in collaboration with a language center wherein the TOEFL iBT class was held. The research method employed the Rasch model as a quantitative analysis approach by using three Minister software outputs used for data analysis: the “statistical summary output” to obtain figures and data in general, output item one-dimensionality to obtain reliability information, and Fit-Order Items to obtain item validity. The results of the reliability were reflected in Cronbach’s alpha value (a) 0.91, item reliability 0.81, and person reliability 0.89 which show “excellent” reliability performance. The item fit order measured by MNSQ, ZFTD, and PT. Measure correlation with four assessment items: content, structure, diction, and mechanic fulfilled almost all ranges except for writing mechanics showing negative results on the MNSQ. It could be concluded that holistic rubric is the “Valid” measuring tool for assessing students’ abilities in various test settings that require a rubric for students’ essay assessment in blended learning program.

• Katakunci Essay Assessment; Holistic Rubric; Rasch Model; Validity and Reliability
• URL https://www.atlantis-press.com/proceedings/oceri-23/125991509
• Penerbit Atlantis Press, Paris

REFERENSI

1. H. Jacobs., Holly. L., Stephen, A., Zingkgraf., Deanne. R., Wormuth, V., Faye, H., Jane, B.,
Testing ESL Composition: A Practical Approach. Rowley: Newbury House Publishers, Inc,
1981.
2. Y. Y. and L. W. P. Richard P. Bagozzi, “Assessing Construct Validity in Organizational
Research,” Sage Publ. Inc., vol. 36, no. 3, pp. 421–458, 1991, doi: https://doi.org/10.2307/
2393203.
3. B. Huot, “Reliability, Validity, and Holistic Scoring: What We Know and What We Need
to Know,” Coll. Compos. Commun., vol. 41, no. 2, pp. 201–213, 1990, doi: https://doi.org/
https://doi.org/10.2307/358160.
4. S. Dikli, “Assessment at a distance: Traditional vs. Alternative Assessments,” Turkish Online
J. Educ. Technol., vol. 2, no. 3, pp. 13–20, 2003.
5. D. V. E. Lai, E. Wolfe, “Halo Effects and Analytic Scoring : A Summary of Two Empirical
Studies Research,” Lang. Test. Asia. Springer Open, vol. 10, no. 1, 2012, doi: https://doi.org/
10.1186/s40468-020-0098-3.
6. E. I. D. & A. D. B.A, “Analyzing rater severity in a freshman composition course using many
facet Rasch measurement.,” Lang. Test. Asia. Springer Open, vol. 10, no. 1, 2020, doi: https://
doi.org/10.1186/s40468-020-0098-3.
7. P. Hafner, J., & Hafner, “Quantitative analysis of the rubric as an assessment tool: An empirical
study of student peergroup rating,” Int. J. Sci. Educ., vol. 25, no. 12, 2003, doi: 1509–1528.
https://doi.org/10.1080/0950069022000038268.
8. G. Jonsson, Anders; Svingby, “The use of scoring rubrics: Reliability, validity and educational
consequences,” Educ. Res. Rev. Sci. Direct., vol. 2, no. 2, pp. 130–144, 2007, doi: https://doi.
org/10.1016/j.edurev.2007.05.002.
9. J. Arter, J., & McTighe, Scoring rubrics in the classroom: Using performance criteria for
assessing and improving student performance. California: Corwin Press, 2001.
10. N. T. Carr, “A comparison of the effects of analytic and holistic rating scale types in the
context of composition tests,” Issues Appl. Linguist., vol. 11, no. 2, pp. 207–241, 2000.
11. S. M. Kemp, J. E., Morrison, G. R., & Ross, Designing effective instruction (2nd ed.). Upper
Saddle River: Prentice Hall, 1998.
12. P. S. (2006) Cooper, D.R., & Schindler, Business Research Methods. USA: Mcgraw Hill,
2006.
13. H. Mohajan, “Two Criteria for Good Measurements in Research: Validity and Reliability,”
Ann. Spiru Haret Univ., vol. 17, no. 3, pp. 58–82, 2017.
14. T. H. Nguyen, H. R. Han, M. T. Kim, and K. S. Chan, “An Introduction to Item Response
Theory for Patient-Reported Outcome Measurement,” Patient, vol. 7, no. 1, p. 23, 2014, doi:
https://doi.org/10.1007/S40271-013-0041-0.
15. D. Rachman, T. & Napitupulu, “Rasch Model for Validation a User Acceptance Instrument
for Evaluating E-learning System,” CommIT (Communication &Information Technol. J., vol.
11, no. 1, pp. 9–16, 2017.
16. W. Sumintono, B. & Widhiarso, Applications of the Rasch Model to Social Science Research.
Trim Komunikata Publishing House, 2013.
17. S. Wibisono, “Rasch Model Application for Validation of Religious Fundamentalism Measurement Instruments for Muslim Respondents,” J. Pengukuran Psikol. Dan Pendidik.
Indones., vol. 5, no. 1, 2018, doi: https://doi.org/10.15408/jp3i.v5i1.9239.
18. D. Andrich, “A rating formulation for ordered response categories,” Psychometrika, vol. 43,
no. 1, pp. 561–573, 1978, doi: https://doi.org/https://doi.org/10.1007/BF02293814.
19. B. Misbach, I. H., & Sumintono, “Development and Validation of the Instrument ‘Student Perceptions of Teacher Moral Character’ in Indonesia with the Rasch Model,” . PROCEEDING
Semin. Nas. Psikometri, pp. 148–162, 2014.
20. M. S. Boone, W. J., Staver, J. R., Yale, M. S., Boone, W. J., Staver, J. R., & Yale, “Item
Measures. Rasch Analysis in the Human Sciences,” pp. 93–110, 2014, doi: https://doi.org/
10.1007/978-94-007-6857-4_5.
21. C. Bond, T., & Fox, Applying the Rasch Model. Routledge, 2015.
22. P. Djaali, & Muljono, Measurement in the field of education. Jakarta: PT.Grasindo, 2008.
23. P. . Lazarfeld, “Latent Structure Analysis,” S. Koch (Ed), Psychol. A Study a Sci., vol. 3, no.
1, pp. 476–543, 1959.