A Validation of the Assessment Practices Inventory Modified (APIM) Scale using Rasch Measurement Analysis
DOI:
https://doi.org/10.53449/ije.v2i2.85Keywords:
validation, assessment practices inventory modified, Rasch measurement analysisAbstract
There are many instruments that have been designed to measure assessment practices skills, but very few have been validated for their soundness and consistency in measuring lecturers’ assessment practices skills. This study was undertaken to examine the psychometric properties of the Assessment Practices Inventory Modified (APIM) scale, and its soundness in measuring assessment practices skills among university lecturers. A quantitative survey research design was adopted for this study. The 50-item APIM scale on a five-point Likert scale was administered to a sample of 321 lecturers randomly selected from six universities in Uganda. The data collected was analysed using WINSTEPS Rasch Measurement Modelling Program for both Classical Test Theory (CTT) and Item Response Theory (IRT) to test the psychometric properties of the APIM scale. From the results of both the CTT (Cronbach’s alpha and the point bi-serial coefficients) and IRT (category probability curve, item and persons’ reliabilities, item characteristic curve, item difficulty, fit statistics, and principal component analysis) in this study, the APIM scale was found to have adequate psychometric properties in measuring assessment practices skills among university lecturers. The APIM scale was also found to be invariant to gender of the university lecturers. In conclusion, the APIM scale has been found to be sound and consistent in measuring university lecturers’ assessment practices skills. This study has pronounced a sound and consistent instrument in measuring assessment practices skills among university lecturers in Uganda, and has provided universities in Uganda with a valid and reliable instrument which will measure assessment practices skills of their lecturers. The results of this study have highlighted that the APIM scale can universally measure assessment practices skills among university lecturers.
References
Ainol-Madziah, Z., & Noor Lide, A. K. (2006). Classical and Rasch analyses of dichotomously scored reading comprehension test items. Malaysian Journal of ELT Research, 2, 1-20.
Ainsworth, L. (2003a). Power standards: Identifying the standards that matter most. Englewood, CO: Advanced Learning.
Ainsworth, L. (2003b). Un-wrapping the standards: A simple process to make standards manageable. Englewood, CO: Advanced Learning.
Ainsworth, L., & Viegut, D. (2006). Common formative assessments: How to connect standards based instruction and assessment. California: Corwin.
Airasian, P. W., & Russell, M. K. (2008). Classroom assessment: Concepts and applications (6th ed.). New York: McGraw Hill.
Alkharusi, H. (2012). Educational assessment attitudes, competences, knowledge, and practices: An exploratory study of Muscat teachers in the Sultanate of Oman. Journal of Education and Learning, 1(2), 217-232.
Andrich, D. (1988). Rasch models for measurements. Newbury Park, CA: Sage Publications.
Andrich, D. (2005). Rasch models for ordered response categories. In B. S. Everitt, & D. C. Howell (Eds.), Encyclopaedia of statistics in behavioural science (pp. 1698-1707). New York: John Wiley & Sons.
Association of College and Research Libraries [ACRL] (2017). Academic Library impact on student learning and success: Findings from Assessment in Action Team Projects. Prepared by Karen Brown with contributions by Kara J. Malenfant. Chicago: Association of College and Research Libraries.
Biggs, J. (2003). Aligning teaching and assessment to course objectives. Paper presented at ICHEd Conference: Teaching and Learning in Higher Education: New Trends and Innovations, April 2003. University of Aveiro, Portugal. Retrieved from http://www.event.ua.pt/iched/main/invcom/p182.pdf
Bond, T. G., & Fox, C. M. (2001). Applying the Rasch model: Fundamental measurement in the human sciences. New Jersey: Lawrence Erlbaum.
Bond, T. G., & Fox, C. M. (2007). Applying the Rasch model: Fundamental measurement in the human sciences (2nd ed.). New Jersey: Lawrence Erlbaum.
Braney, B. T. (2010). An examination of fourth grade teachers’ assessment literacy and its relationship to students’ reading achievement. (PhD Dissertation). University of Kansas, Kansas.
Brown, G. T. L. (2003), December). Teachers’ instructional conceptions: Assessments relationship to learning, teaching, curriculum, and teacher efficacy. Paper presented at the joint conference of the Australian and New Zealand Associations for Research in Education (AARE/NZARE). University of Auckland, Auckland.
Burry-Stock, J. A., & Frazier, C. H. (2008). Revision of the Assessment Practice Inventory Revised (APIR): A combined exploratory factor analysis and polytomous IRT approach. Paper presented at the American Educational Research Association, March 2008. New York.
Coates H. (2015) Assessment of learning outcomes. In: A. Curaj, L. Matei, R. Pricopie, J. Salmi, & P. Scott (Eds). The European higher education area. Springer, Cham. https://doi.org/10.1007/978-3-319-20877-0_26
Connoley, R. (2004). Criterion referenced assessment. Working paper. Geelong-Victoria: Deakin University.
Fatmawati, A. (2011). Using portfolio to assess student learning of problem solving. Working paper, Geelong-Victoria: Deakin University.
Fisher, W. P. J. (2007). Rating scale instrument quality criteria. Rasch Measurement Transactions, 21, 1095.
Garson, G. D. (1998). Cronbach’s reliability analysis. North Carolina: North Carolina State University.
Gibbs, G. (2006). Why assessment is changing. In C. Bryan & K. Clegg (Ed.), Innovative assessment in higher education (pp. 11-22). London: Routledge.
Gleim, J. A., & Gleim, R. R. (2003). Calculating, interpreting, and reporting Cronbach’s alpha reliability coefficient for Likert type scales. In Midwest research practice conference in adult, continuing, and community education. Ohio: Ohio University.
Greatorex, J., Johnson, M., & Coleman, V. (2017). A review of instruments for assessing complex vocational competence. Research Matters: A Cambridge Assessment Publication. Retrieved from:
http://www.cambridgeassessment.org.uk/research-matters/
Hagquist, C. (2001). Evaluating composite health measures using Rasch modelling: An illustrative example. Social and Preventive Medicine, 46, 369-378.
Hagquist, C. (2008). Psychometric properties of the psycho-somatic problems scale: A Rasch analysis on adolescent data. Social Indicators Research, 86, 511-523.
Hagquist, C., & Andrich, D. (2004a). Measuring subjective health among adolescents in Sweden: A Rasch analysis of the HBSC instrument. Social Indicators Research, 68, 201-220.
Hagquist, C., & Andrich, D. (2004b). Is the sense of coherence instrument applicable on adolescents? A latent trait analysis using Rasch modelling. Personality and Individual Differences, 36, 955-968.
Hambleton, R. K., Swaminathan, H., & Rogers, H. J. (1991). Fundamentals of item response theory. Newbury Park: Sage Publications.
Hassel, S., & Ridout, N. (2018). An investigation of first-year students’ and lecturers’ expectations of university education. Frontiers in Psychology, 8, 2218. doi:10.3389/fpsyg.
Hindi, N., & Miller, D. (2000). A survey of assessment practices in accounting departments of colleges and universities. Journal of Education for Business, 75(5), 286.
Jackson, T. R., Draugalis, J. R., Slack, M. K., & Zachry, W. M. (2002). Validation of authentic performance assessment: A process suited for Rasch modelling. American Journal of Pharmaceutical Education, 66, 233-243.
Lai, J. S., & Eton, D. T. (2002). Clinically meaningful gaps. Rasch Measurement Transactions, 15(4), 850.
Lai, P. S., Sim, S. M., Chua, S. S., Tan, C. H., Ng, C. J., Achike, F. I., & Teng, C. L. (2015). Development and validation of an instrument to assess the prescribing readiness of medical students in Malaysia. BMC medical education, 15, 153. doi:10.1186/s12909-015-0433-z
Linacre, J. M. (1999a). Investigating rating scale category utility. Journal of Outcome Measurement, 3(2), 103-122.
Linacre, J. M. (1999b). Category disordering vs. Step (threshold) disordering. Rasch Measurement Transactions, 13(1), 675.
Linacre, J. M. (2006). WINSTEPS: Rasch measurement computer program (version 3.60). Chicago: Winsteps.com.
Linacre, J. M. (2009). WINSTEPS computer software (version 3.68). Beaverton, Oregon: Winsteps.com.
Linacre, J. M. (2011). User’s guide to WINSTEPS-MINISTEPS. Rasch model computer programs manual 3.73.0.
Maller, S. J. (2001). Differential item functioning in the WISC-III: Item parameters for boys and girls in the national standardisation sample. Education and Psychological Measurements, 61(5), 793-817.
Martell, K., & Calderon, T. (2005). Assessment of student learning in business schools: What it is, where we are, and where we need to go next. In K. Martell & T. Calderon (Eds.), Assessment of student learning in business schools: Best practices each step of the way (pp. 1-22). Tallahassee, Florida: Association for Institutional Research.
Marzano, R. J. (2000). Transforming classroom grading. Alexandria, VA: Association for Supervision and Curriculum Development.
McClarty, K. L., & Gaertner, M. N. (2015). Measuring mastery: Best practices for assessment in competency-based education. Center for College & Career Success, Pearson. Retrieved from:
https://www.luminafoundation.org/files/resources/measuring-astery.pdf
McMillan, J. H. (2001). Secondary teachers’ classroom assessment and grading practices. Educational Measurement: Issues and Practice, 20(1), 20-32.
Messick, S. (1989). Validity. R. In Linn (Ed.), Educational measurement (3rd ed.). Washington, D.C: American Council on Education.
Ministry of Education [MOE]. (2010). Assessment and evaluation in Saskatchewan. Saskatchewan: MOE, Assessment and Evaluation.
National Council for Curriculum and Assessment [NCCA]. (2005). Supporting assessment in schools-1: Assessment in Primary Schools. Retrieved from http://www.ncca.ie/uploadedfiles/primary/ass_pr_schs_05.pdf
O’Connor, K. (2009). Reforming grading practices in secondary schools. Principal's Research Review, 4(1), 1-7.
O’Donovan, B., Price, M., & Rust, C. (2001). The student experience of criterion referenced assessment through the use of a common criteria assessment grid. Innovations in Learning and Teaching International, 38(1), 74-85.
O’Grady, G. (2006). Guide for good practice in assessment. Ngee Ann Polytechnic: Singapore.
Orzolek, D. C. (2006). The paradox of assessment: Assessment as paradox. Research and Issues in Music Education, 4(1), 1-5.
Pallant, J. F., & Tennant, A. (2007). An Introduction to Rasch measurement model: An example using the hospital anxiety and depression scale (HADS). Journal of Clinical Psychology, 46, 1-18.
Peterson, M. W., Einarson, M. K., Augustine, C. H., & Vaughan, D. S. (1999a). Institutional support for student assessment: Methodology and results of a national survey. Stanford, CA: Stanford University, NCPI.
Peterson, M. W., Einarson, M. K., Augustine, C. H., & Vaughan, D. S. (1999b). Designing student assessment to strengthen institutional performance in comprehensive institutions. National Canter for Postsecondary Improvement. Stanford, CA: Stanford University, NCPI.
Pomeranz, J. L., Byers, K. L., Moorhouse, M. D., Velozo, C.A., & Spitznagel, R. J. (2008). Rasch analysis as a technique to examine the psychometric properties of a career ability placement survey subtest. Rehabilitation and Counselling Bulletin, 51(4), 251-259.
Popper, E. (2005). Learning goals: The foundation of curriculum development and assessment. In K. Martell, & V. Calderon (Eds.), Assessment of student learning in business schools: Best practices each step of the way (pp. 1-23). Tallahassee, Florida: Association for Institutional Research.
Price, M., & Rust, C. (1999). The experience of introducing a common criteria assessment grid across an academic department. Quality in Higher Education, 5(2), 133-44.
Rasch, G. (1960). Probabilistic models for some intelligent and attainment tests. Copenhagen: Institute of Educational Research.
Rasch, G. (1966). An item analysis which takes individual differences into account. British Journal of Mathematical and Statistical Psychology, 19(1), 49-57.
Reeves, D. B. (2004). The Case against the Zero. Phi Delta Kappan, 86, 324-326.
Rousselot, N., Tombrey, T., Zongo, D., Mouillet, E., Joseph, J. P., Gay, B., & Salmi, L. R. (2018). Development and pilot testing of a tool to assess evidence-based practice skills among French general practitioners. BMC Medical Education, 18, 254 https://doi.org/10.1186/s12909-018-1368-y
Russell, J., & Markle, R. (2017). Continuing a culture of evidence: Assessment for improvement. Educational Testing Service Research Report Series, 1, 1-10. https://doi.org/10.1002/ets2.12136
Satterly, D. (1989). Assessment in schools (2nd ed.). Oxford: Blackwell.
Shaw, F., Wright, B., & Linacre, J. M. (1992). Disordered steps? Rasch Measurement Transactions, 6(2), 225.
Singh, C. K. S., Lebar, O., Kepol, N., Abdul-Rahman, R., & Mukhtar, K. A. M. (2017). An Observation of classroom assessment practices among lecturers in selected Malaysian higher learning institutions. Malaysian Journal of Learning and Instruction, 14(1) 23-61.
Smith, E. V. Jr., Wakeky, M. B., de Kruif, R. E. L., & Swartz, C. W. (2003). Optimizing rating scales for self-efficacy and other. Research, Educational and Psychological Measurement, 63(3), 369-391.
Tavakol, M., & Dennick, R. (2011). Making sense of Cronbach’s alpha. International Journal of Medical Education, 2, 53-55.
Watkins, D. (1998). Assessing university students in Hong Kong: How and why? In D. Watkins, C. Tang, J. Biggs, & R. Kuisma (Eds.), Assessment of university students in Hong Kong: How and why, assessment portfolio, students’ grading (pp. 5-27). Hong Kong: University of Hong Kong.
Wright, B. D., & Masters, G. N. (1982). Rating scale analysis. Chicago: MESA Press.
Wright, B. D., & Mok, M. (2000). Rasch models overview. Journal of Applied Measurement, 1, 83-106.
Wright, B. D., & Stone, M. H. (2004). Making measures. Chicago: Phaneron Press.
Wright, B. D., Linacre, J. M., Gustafson, J. E., & Martin, L. P. (1994). Reasonable mean square fit values. Rasch Measurement Transactions, 8(3), 370.
Zhang, Z., & Burry-Stock, J. A. (1994). Assessment practices inventory. Tuscaloosa, AL: The University of Alabama.
