Situational judgement tests have good levels of predictive validity, reliability, and incremental validity, compared with personality and IQ tests, for testing a range of professional attributes.
The Original CASPer paper (and when it was known as CMSENS). Notably type written responses had higher correlation with MMI and the final model for CASPer had good predictive validity against MMI.
When comparing scores for Autobiographical Submissions completed onsite and offsite, the scores for the offsite submissions were notably higher (offsite mean= 4.4, onsite mean= 4.1). When scoring them using either vertical or horizontal scoring, the interrater reliability was much lower when using vertical scoring (0.03) compared to the horizontal scoring (0.69). When comparing the correlation with the Multiple Mini Interview, the horizontal scoring was a higher correlation (r= 0.44 offsite and 0.65 onsite) than the vertical scoring (r= 0.12 offsite and 0.28 onsite).
Off-site ABS ratings were higher than on-site ratings, and the two sets of ratings were uncorrelated with one another. On-site ABS ratings increased with increased time allowed for completion but the reliability of the measure was unaffected by this variable. To improve ABS validity, modification for the current Web-based submission format warrants consideration.
The results of this study confirmed the importance of cognitive predictors. The situational judgement test showed incremental validity over cognitively oriented measures for curricula that included interpersonal courses, but not for other curricula.
The situational judgement test had a lower correlation with the cognitive ability than did the written version. It also had a higher predictive and incremental validity for prediction interpersonally oriented criteria than did the written version. Applicants also reacted relatively favorably to the test although there was no significant difference in face validity between the formats. Suggested that SJT format changes be made with caution and that validation evidence is needed when changed are proposed.