Multiple Mini-Interviews (MMIs)

1. A comparison of behavioral and multiple mini-interview formats in physician assistant program admissions
Comparing the behavioral interview format and the multiple mini-interview, it is seen that they have similar model fits. However the difference between them is that the multiple mini-interview had more variation in noncognitive traits as well as identified better matching of person ability and station difficulty. The behavioral format did not effectively measure differences in patient characteristics. It is therefore determined that the multiple mini-interview is a more reliable tool for the admissions process in detecting covert professionalism among applicants for physician assistant programs.

2. A comparison of multiple mini-interviews and structured interviews in a UK setting
This study looked at how people in MBBS 4 and MBBS 5 courses performed the MMI. When comparing the two with how they did on the MMI versus the traditional interview, MBBS 4 did just as well on both interviews whereas MBBS 5 did better on the MMI.  MBBS 4 and MBBS 5 did the MMI equally well.

3. A cost efficiency comparison between the multiple mini-interview and traditional admissions interviews
It was determined that the MMI is more reliable compared to the traditional admissions interview as well as having a better predictive power. Even though the MMI requires greater preparatory work and a larger number of rooms, these costs are off-set by the fact that the MMIs use less man-hours.

4. A generalizability analysis of a veterinary school Multiple Mini Interview: effect of number of interviewers, type of interviews, and number of stations
When examining the reliability of the MMIs for veterinary admissions, it was found that it had sufficient reliability. It was also found that using 1 interviewer with 10 stations would meet the same reliability while reducing the resource requirements.

5. A generalizability study of the medical judgement vignettes interview to assess students’ noncognitive attributes for medical school
There is a higher generalizability coefficient when increasing the number of stations to 6 with a single judge at each compare to a 3 station to judge approach. Increasing the number of judges from two to three does not improve the generalizability coefficient. Increasing the number of stations however, has a considerable influence of the overall dependability of the measurement.

6. An admissions OSCE: the multiple mini-interview
This is the classic MMI paper that started it all. It is also a good template around which to structure the reporting of admissions studies, especially if they are concerned with reliability.

7. An evaluation of the multiple mini-interview as a selection tool for dental students
When screening the use of the MMI for dental students, many applicants felt that they had performed well and were prepared for it. The MMIs experience was given a positive review.

8. An exploration of the relationship between emotional intelligence (EI) and the Multiple Mini-Interview (MMI)
Subjects participated in the MMI interview process as well as used the Bar-On EQi emotional intelligence instrument. It was found that there was no relationship between the EI and the MMI. The findings also did not support using the EQi as a prescreening tool for the MMI.

9. Are current strategies for pre-registration student nurse and student midwife selection ‘fit for purpose’ from a UK perspective? Introducing the multiple mini-interview
The MMI was introduced as an alternative method for selecting nurses and midwives. It was found that the MMIs had moderate to good reliability as well as validity and was evaluated positively.

10. Assessment of applicants to the veterinary curriculum using a multiple mini-interview method
Three noncognitive attributes were assessed at each of the five MMI stations by two interviewers. Though it did not correlate with grade point average it did prove to be a reliable and valid method for assessing noncognitive characteristics.

11. Assessment of non-cognitive traits through the admissions multiple mini-interview
When looking at if the MMI was a reliable and valid assessment of non-cognitive traits, it was shown that there were low correlations between the factor analysis and stations. This shows that there were different attributes assessed at each station. When comparing the accepted candidates and the candidates on the waitlist  there were significant differences in scores. The MMIs offer a fair and defensible assessment of applicants with reliable and valid assessments of different non-cognitive attributes.

12. Associations between the big five personality factors and multiple mini-interviews
When data from the MMI scores of applicants to an Australian medical school were examined, it was shown that there were significant uncorrected correlations with extraversion, conscientiousness and agreeableness. They were also shown to be correlations with a situational judgement test of interpersonal understanding. There was not however, a correlation between MMI scores and tests of logical reasoning ability, non-verbal reasoning, or past academic performance.

13. Association between a medical school admission process using the multiple mini-interview and national licensing examination scores
Students that were accepted by an admissions process that used MMI assessment, scored higher on Canadian national licensing examinations than those who were rejected.

14. Comparison of aboriginal and nonaboriginal applicants for admissions on the Multiple Mini-Interview using aboriginal and nonaboriginal interviewers
When having both aboriginal and nonaboriginal applicants as well as interviewers,  it was observed that there was no significance in the interaction between the interviewer and interviewee type.

15. Development and pilot testing of a multiple mini-interview for admission to a pharmacy degree program
Using student, practitioner and faculty interviewers, the MMIs were seen to be a feasible method with reliable results for an admission method that tests different attributes than the Pharmacy College Admission Test or the Prepharmacy Average.

16. Discerning quality: using the multiple mini-interview in student selection for the Australian National University Medical School
When using the MMI at the Australian National University Medical School, candidates reported high levels of satisfaction. It is shown that admission interviews can be efficient and streamlined while still being informative.

17. Does applicant personality influence multiple mini-interview performance and medical school acceptance offers?
When using staff, students, and simulated patients as examiners, students rated candidates more severely than staff and had more unexpected ratings. This can be used to show that a n adjustment for examiner severity/leniency and station difficulty needs to occur to ensure fairness when ranking candidates

18. Does practice make perfect? The effect of coaching and retesting on selection tests used for admission to an Australian medical school
This study shows that coaching does not aid and can even hinder performance on the MMI. Practice however does help scores.

19. Enhancing the reliability of the multiple mini-interview for selecting prospective health care leaders
An anonymous survey was taken to assess interviewers’ and applicants’ opinion on the MMI both in 2009 and in 2010. The survey showed positive feedback from both the interviewers and the applicant with no gender or disadvantaged status bias. When surveyed in 2009 the reliability was lower than it had been in previous studies but once an easy station was replaced with a  more challenging station and a new scoring rubric, the reliability went up in 2010.

20. Evaluation of an interview process for admission into a school of pharmacy
Using the current standardized grading rubric of North Dakota State University, they were looking to see how effective their interview process was. Taking into consideration the scores of the total interview, individual attribute domains, and consistency and reliability of the interviewers, they determined that their current interview process was reliable and consistent but improvements could be made in the future.

21. Evaluation of a multiple mini-interview protocol used as a selection tool for entry to an undergraduate nursing programme
When implementing the use of the MMI, results showed that many of the candidates and interviewers had a better experience with the MMI process over the traditional interview. Unsolicited feedback was that both the candiates and the interviewers would have preferred to have discussed nursing issues to a greater extent.

22. Evaluations of Situational Judgment Test to Assess Non-Academic Attributes in Selection
Situational judgement tests have good levels of predictive validity, reliability, and incremental validity, compared with personality and IQ tests, for testing a range of professional attributes.

23. Experience of establishment of multiple mini structure interview as part of student admission policy at Faculty of Medicine
When the Faculty of Medicine, King Abdulaziz University conducted an MMI, the results showed that men scored significantly higher in stations concerned with personnel character and professionalism. Women scored significantly higher in all other stations. The overall performance of women was greater than men.

24. Experiences of the multiple mini-interview: a qualitative analysis
Surveys were analyzed and some of the main topics discussed were participants’ perspectives on one-to-one interviews, multiple assessment opportunities, standardized scenario-based interviews, mini-interviews, as well as attributes that are and should be assessed by the MMIs.

25. Extending the interview to all medical school candidates – Computer-Based Multiple Sample Evaluation of Noncognitive Skills (CMSENS)
The Original CASPer paper (and when it was known as CMSENS). Notably type written responses had higher correlation with MMI and the final model for CASPer had good predictive validity against MMI.

26. Factors affecting the utility of the multiple mini-interview in selecting candidates for graduate-entry medical school
Interviewer subjectivity is the largest source of error. A low correlation between questions proves there is an importance for context specificity. For precise comparison, the MMI needs to be adequately long.

27. Improving student selection using multiple mini-interviews with multifaceted Rasch modeling
When using staff, students, and simulated patients as examiners, students rated candidates more severely than staff and had more unexpected ratings. This can be used to show that a n adjustment for examiner severity/leniency and station difficulty needs to occur to ensure fairness when ranking candidatesing

28. Internet-based multiple mini-interviews for candidate selection for graduate entry programmes
Using the mean scores of in person MMIs and iMMIs, it was found that there was no significant difference between the MMI and iMMI scores.

29. Interviewer bias in medical student selection
When looking at interviewer ratings and interviee scores, it was found that in 2006, interviewers with a high level of agreeableness gave higher ratings. In 2007, interviewers with a high level of neuroticism gave lower ratings. In 2006 but not 2007, female interviewers gave higher ratings to both male and females. In both 2006 and 2006, there was not correlation between similar sex of interviewer and interviewee, and ratings.

30. Introduction of the multiple mini interview into the admissions process at the University of Calgary: acceptability and feasibility
More than 90% of both interviewers and applicants were in favour of using the MMIs. With the MMIs they were able to interview with fewer interviewers and less time per interviewer was required.

31. Medical school admissions: enhancing the reliability and validity of an autobiographical screening tool
When comparing scores for Autobiographical Submissions completed onsite and offsite, the scores for the offsite submissions were notably higher (offsite mean= 4.4, onsite mean= 4.1). When scoring them using either vertical or horizontal scoring, the interrater reliability was much lower when using vertical scoring (0.03) compared to the horizontal scoring (0.69). When comparing the correlation with the Multiple Mini Interview, the horizontal scoring was a higher correlation (r= 0.44 offsite and 0.65 onsite) than the vertical scoring (r= 0.12 offsite and 0.28 onsite).

32. Medical school preadmission interview: are structured interviews more reliable than unstructured interviews?
The unstructured interview format is seen to be more reliable when examining interrater reliability and test-retest types of reliability. However when combining both unstructured and structured interview formats, they are more reliable combine than being used alone.

33. Multiple mini-interviews: opinions of candidates and interviewers
Questionnaires were filled out by both interviewers and candidates about the use of MMIs for selecting senior house officers to a UK regional pediatric training programme. Both interviewers and candidates were positive about the fairness of the MMI and agreed that the MMI’s questions are appropriate, easy to understand, reliable, and fair.

34. Multiple mini-interviews predict clerkship and licensing examination performance
MMI was the best predictor of OSCE performance, clerkship encounter cards, and clerkship performance ratings.

35. Multiple mini-interview scores of medical school applicants with and without rural attributes
Urban high school graduates scored significantly higher than rural high school graduates. When looking at rural attribute domains, those with rural community service alone had the highest scores and those with rural connections alone had the lowest scores.

36. Multiple mini-interview in the U.K. context: 3 years of experience at Dundee
The MMI used in Dundee was assessed using a questionnaire which confirmed the process was acceptable by all. the Multi-faceted Rasch modelling system was used to control for assessor leniency, stringency, and the impact of using fair scores. It showed good psychometric properties and was also shown to be feasible to use.

37. Multiple mini-interviews versus traditional interviews: stakeholder acceptability comparison
The stakeholders are both the evaluators and the applicants. Questionnaire results showed significant differences between multiple mini-interviews and traditional interviews. The multiple mini-interview was shown to have a more positive result in regards to fairness, imposition of stress, and effectiveness. Applicants also thought that the multiple mini-interview was more enjoyable, allowed them to be competitive, and was often a favourite part of the interview experience. Written responses also gave a positive outlook on the fairness, opportunity to show one’s strength, and constancy of simulations that the MMI provided.

38. Non-association between Neo-5 personality tests and multiple mini-interview
Through voluntary completion of the Neo-5 personality test before admissions tests by medical school applicants, it was found that there was no statistically significant correlation between personality factors and the MMI. There was also no statistically significant correlation between personality factors and cognitive factors such as GPA and the MCAT.

39. Overview: what’s worked and what hasn’t as a guide towards predictive admissions tool development
When looking at the predictive validity of different tools used for admissions, it was found that there were some that were reliable and others were not. The ones that were reliable include academic scores (GPA), aptitude tests (MCAT), and non-cognitive testing (MMI). Those that were not found to be as reliable include personal statements and interviews, emotional intelligence, personality testing, letters of reference, and situational judgement testing.

40. Predictive validity of the Dundee multiple mini-interview
When comparing the MMI with the Universities and Colleges Admissions Service (UCAS), and the UK Clinical Aptitude Test (UKCAT), it is shown that the MMI was the most consistent predictor of success in early years at medical school. The UCAS and UKCAT showed minimal or no predictive ability.

41. Predictive validity of the multiple mini-interview for selecting medical trainees
With more time in training, the correlation between ‘non-cognitive’ and ‘cognitive’ assessment instruments increases.

42. Psychometric properties of the multiple mini-interview used for medical admissions: findings from generalizability and Rasch analyses
MMI is a reliable test. The findings, however, show that “the non-cognitive qualities are not being measured as unique constructs across stations” (Sebok, 2013)

43. Rater and occasion impacts on the reliability of pre-admission assessments
This study compares whether having additional raters or having more interview occasions increases the reliability of the MMIs. After comparing the results, it was found that having more interview occasions increased the reliability of the MMIs more than having additional raters.

44. Resident selection for a physical medicine and rehabilitation program: feasibility and reliability of the multiple mini-interview
This study was to determine how reliable the MMI would be to use for a physical medicine and rehabilitation residency training program. They wanted the process to be more focused on assessing non-cognitive physician competencies. The MMI was found to have good interrater reliability. It was also found to be time and cost efficient as well as feasible.

45. Should candidate scores be adjusted for interviewers stringency or leniency in the multiple mini-interview?
Using the Multi-Facet Rasch model, a candidates score was adjusted for the stringency or leniency of interviewers as well as the difficulty of the questions the candidates are asked.

46. Should efforts in favor of medical student diversity be focused during admissions or farther upstream
When looking at the correlation between the MMIs and diversity, it was found that the MMI scores were uncorrelated to gender, size of community of origin, and income level. It was found to be positively correlated with age but negatively correlated with aboriginal status. Because the MMI did not completely neutralize the diversity-limiting factors, it was concluded that student diversity should be focused more upstream.

47. The ability of the multiple mini-interview to predict preclerkship performance in medical school
MMI, relative to traditional tools designed to assess non-cognitive qualities of applicants, appears to be more beneficial at predicting preclerkship performance.

48. The acceptability of the multiple mini interview for resident selection
Both interviewers and applicants were satisfied with the 8 minutes allotted at each station. Applicants felt they were free from culture and and gender bias. Interviewers thought that the MMI was a fair assessment of potential for family medicine.

49. The multiple mini-interview: how long is long enough?
Comparing the mean scores and the cumulative scores of MMI stations done at 8 minutes and at 5 minutes, it was found that reducing the station length time had minimal effect on the test reliability and applicant ranking.

50. The multiple mini-interview for selection of international medical graduates into family medicine residency education
This study looked at the reliability and validity of using the MMI to gauge professionalism of international medical graduates into a family medicine residency in Alberta. Reliability was assessed by using decision studies and generalisability. Validity was assessed by looking at the differences in MMI socres associated with session, track and socio-demographic characteristics of candidates. It was also assessed by calculating the correlation between the scores of MMIs and compulsory examinations. It was concluded that the MMIs were a reliable and valid way to evaluate international medical graduates’ professionalism.

51. The Objective Structured Interview for Medical Student Selection
The structured admission interview that is used at the University of Newcastle can predict in some measure ultimate failure or success in medical studies.

52. The relationship between the interviewers’ characteristics and ratings assigned during a multiple mini-interview
Using members of the community and faculty members to interview MMIs, it was found that increasing the diversity of the interviewers could increase the diversity of the group of candidates that are accepted.

53. The reliability and acceptability of the Multiple Mini-Interview as a selection instrument for postgraduate admissions
Candidates and interviewers completed a survey after completing the MMI to evaluate the acceptability of its use. It was determined that its reliability was acceptable. Most candidates stated that they could represent themselves accurately and most interviewers thought that they could determine a candidate’s abilities reasonably.

54. The selection process in the Faculty of Medicine: the usefulness of a psychological and aptitude assessment
It is recommended to use a selection model that includes an assessment of psychological features and aptitudes of candidates, as well as cognitive skills.

55. Utility of a writing station in the multiple mini-interview
With the addition of a writing station into the Multiple Mini Interview process, it increased the number of applicants seen in a rotation without increasing the number of interviewers needed.

56. Validating a multiple mini-interview questions bank assessing entry-level reasoning skills in candidates for graduate-entry medicine and dentistry programmes
The results of this validation showed that none of the items on the checklist had significant differential item functioning. It also showed that all 39 questions had a good fit to the item response theory model.