What is differential attainment?

Differential attainment (DA) is a gap in attainment between two groups of people with differing characteristics in one or more areas (see below). The gap cannot be adequately explained by the difference in characteristics. Attainment can be affected in one or more spheres, such as examinations, competitive selection processes, and earnings.

Examples of characteristics associated with differential attainment

• Gender and sex (including gender reassignment).
• Ethnicity.
• Country of primary medical qualification.
• First language.
• Neurodivergence.
• Caring responsibilities (including: pregnancy, sexuality, disability, age).

Why are we tackling it (now)?

In February 2023 the RCoA published an external review into the College’s examinations processes. Among the findings was persistent DA, and the review recommended research into DA in the College, along with more involvement of anaesthetists in training in the assessment process. As a result of this Drs Bishop and Wilson were employed as examination fellows with a remit to conduct this research.

The DA masterclass

Along with College examiners, we developed a DA masterclass (DAMC) for the Primary FRCA SOE following a similar project conducted by the Royal College of Psychiatrists. The DAMC is delivered by RCoA examiners, and the focus is heavily on the structure and culture of the exam – aspects such as communication and psychology. It is a one-day, virtually delivered course consisting of short lectures, a warm-up ‘quick-fire’ round of SOE practice, and then the remainder of the day being SOE practice and feedback. We feel this format helps to tackle aspects of preparing for and sitting the exam that have been suggested as causing DA, namely, communication, understanding cultural norms, peer support, and SOE practice avoidance.

So far, the DAMC has been piloted in the North West and East Midlands Schools, with an upcoming pilot planned in South London at the time of writing. The course is open to any candidates sitting the exam, with no obligation to ‘prove’ or declare their reason for wishing to attend. The attendees are invited to fill out a typical post-teaching-course feedback form and invited for a more in-depth interview with one of the Examination Fellows. This allows us to try to extract some of the nuances that are affecting individuals sitting the exam. So far, the responses from candidates towards the course have been extremely positive, and the interviews have been very eye-opening. We are also correlating attendance at the DAMC with exam outcomes to ascertain any quantitative difference made, although currently numbers are far too small to show a significant difference or lack thereof.

Difficulties

A number of aspects of organising the DAMC and associated research have been challenging. There has been discussion about whether the DAMC would unfairly disadvantage candidates who were not at risk of DA by providing a resource to others that was not available to them. We tackled this in two ways: firstly, the DAMC has been designed to try to tackle the underlying causes of DA so that someone not at risk of DA should not benefit from attending, and secondly by opening up the course to anyone who wished to attend (so it is not an exclusive resource). Also, by not stipulating attendance criteria, we removed the need for candidates to identify if or how they might meet the criteria of ‘at risk of DA’, so if they had a characteristic that they did not wish to disclose, they would be able to keep it confidential from faculty.

Another issue we found was in the design of both the course itself, and the post-course interview. We are trying to tackle and investigate issues that neither of us claim to have any experience of. That is, we are trying to solve problems and ask questions about the lived experience of people with characteristics that we do not share. We have sought input from a variety of people and are reiterating the course with each round of feedback that we get to continue to improve it. We have also recognised that it is impossible to make a ‘one course fits all’ for the different aspects of DA – for example, the reasons that IMGs underperform are different from those for female trainees, and this is a potential shortcoming of the DAMC.

Finally, we question the wisdom of running a course that seeks to teach people who do not ‘fit’ the exam how to adapt to navigate it more successfully, rather than the alternative of adapting the exam to better fit a range of people. Ultimately the RCoA is aiming to do both, and another aspect of our role with the College is having input into the reformatting of the exam. The DAMC is a much more immediate application of this than the reformatting, so it certainly has a role.

Next steps

The DAMC will continue to pilot this year, with preliminary research findings to be submitted for publication. Our intent is to recruit more staff so that we can run the DAMC across multiple regions for each sitting of the exam. The second area for expansion is across the other examination diets (Final FRCA, FPM, FICM). Finally, the DAMC will have to update the material when the new exam format launches, although fortunately for us this is a job for future fellows!