When comparing scores for Autobiographical Submissions completed onsite and offsite, the scores for the offsite submissions were notably higher (offsite mean= 4.4, onsite mean= 4.1). When scoring them using either vertical or horizontal scoring, the interrater reliability was much lower when using vertical scoring (0.03) compared to the horizontal scoring (0.69). When comparing the correlation with the Multiple Mini Interview, the horizontal scoring was a higher correlation (r= 0.44 offsite and 0.65 onsite) than the vertical scoring (r= 0.12 offsite and 0.28 onsite).
Off-site ABS ratings were higher than on-site ratings, and the two sets of ratings were uncorrelated with one another. On-site ABS ratings increased with increased time allowed for completion but the reliability of the measure was unaffected by this variable. To improve ABS validity, modification for the current Web-based submission format warrants consideration.