AALHE 2017 Annual Conference

This session describes the process of conducting an inter-rater reliability study on a performance-based assessment instrument developed by multiple institutions. Presenters will explain how we developed and refined both the rubric, as well as an online, asynchronous 90-minute training for evaluators. While the instrument has had validity and reliability studies conducted, this presentation focuses on the results of inter-rater reliability analyses. The session is intended for university administrators and faculty members engaged in developing valid and reliable instruments for performance assessment tasks. Participants will learn the challenges and pitfalls in the training development process and we describe the common methods to examine inter-rater reliability. Participants will discuss issues and opportunities we are face as an assessment community when developing performance-based tasks and their accompanying evaluation instruments.