RCoA Exam Fellows’ update

Dr Hannah Bruce and Dr Mark Chen update us on the key role anaesthetists in training play in leading and effecting significant changes to all RCoA examinations.

It’s been more than two years since RCoA published its independent and internal reviews of the FRCA examinations. Back then, we made a commitment to develop our examinations and update our processes, and that’s exactly what we’ve been doing since. One of the positive steps we’ve taken to enhance the quality and effectiveness of examinations is the establishment of three RCoA Examination Fellowships. Hannah and Dean Nash are the current fellows focused on the Final, and Mark has been working on the Primary. In this article, Hannah and Mark update us on the work they’ve been involved with over the last few months.

Primary and Final exam blueprinting

Blueprinting is the process of mapping the exam structure to the curriculum content to ensure the exam tests a fair representation of what is being taught in the curriculum. As all of the College’s examinations are undergoing change, one of our big projects has been to create blueprints for some of the new format exams.

Mark has been focusing on the blueprint for the revised Primary MCQ, which covers a greater breadth of the syllabus, including more questions on data interpretation and the clinical sciences previously examined in the SOE/OSCE. Dean and Hannah are mirroring this process for the Final and updating the syllabus in line with the 2021 Curriculum. The new blueprints will become a core resource for both candidates revising for the exam and examiners responsible for question creation and assessment development.

Differential attainment

The College has been working on reducing the attainment gap for a number of years, and our predecessors have outlined some of the progress in this area in a previous Bulletin article: Tackling differential attainment in the FRCA. In the article they discussed setting up the Differential Attainment Masterclass, a pilot due to complete in April 2025. We have continued contributing to this, and once the final course has taken place in April, we will publish both a qualitative and quantitative analysis of the course.

Exam validity research

The Independent Exam Review recommended research on concurrent, predictive and construct validity. Our predecessor has started work on concurrent validity, specifically looking at whether Primary FRCA exam pass rates correlate with a consultant assessment score that would predict progression from Stage 1 to Stage 2 of training.

Trainee voice

Alongside our individual projects we also contribute to the ‘trainee voice’ within the Examinations department, and as part of this we attend a number of committees (alongside the Anaesthetists in Training Committee representatives). Here, we have been asked to offer trainee perspectives on a number of topics, including discussions about candidate exam feedback, future exam location, exam cost and exam appeals. Seeing the enormous backroom architecture of running these exams has been hugely insightful, and the chairs of these meetings are proactive in canvasing our opinions on the candidate experience when considering the impact of proposed developments.

In Primary, Mark also joined the pool of SBA question writers (alongside our new affiliate examiners). This included attending and contributing to question-finalising and paper-setting meetings. The MCQ Core Group have been enthusiastic about incorporating the perspective of someone recently on the other side of the test process. Several other projects have resulted from this which include improving processes around SBA quality control and transparency in our paper review processes.

Last September, Dean and Hannah attended the New Examiner Training course and, while we thought our time being viva’d was over, it wasn’t! We acted as examples of good and poor candidates and, even though we are post-exam, it still made us sweat! Here we got an essence of the training process for new examiners, in particular how standards are maintained and the emphasis on candidate experience.

Through all our work, what stands out to all of us is the sheer effort that goes into running and developing professional exams behind the scenes. Over the next few years there will be significant changes to all RCoA examinations including both Primary and Final, and so it’s a very exciting time to be involved. It is clear that anaesthetists in training have a key role to play in leading and effecting these changes – providing a perspective critical to ensuring the fairness and relevance of the examinations to those who are sitting them.

Further reading

This Bulletin article on Fostering positive transformations in medical examinations outlined some of the ongoing work in developing the RCoA examinations.