Examination Reviews
The College undertakes internal and independent reviews of its examinations to ensure they remain fair, robust and aligned with best practice in postgraduate medical assessment. These reviews provide assurance and identify opportunities to strengthen and develop our assessment processes.
Internal review
We are committed to improving the calibre of our examinations, ensuring that they are delivered in line with best practice in assessment for each medical specialty. As part of that commitment, we undertake regular internal reviews of the FRCA examination, the most recent of which we published on 11 January 2023.
We are grateful to all members of the Examination Review Group, and also to our lead examiners, for their time and expertise in contributing to the review.
In conducting the review, we recognise that the FRCA is a high stakes assessment that has a significant impact on candidates’ personal and professional lives. Through the inclusion of an anaesthetist in training representative on the Examination Review Group, we have sought to ensure the experiences of candidates were central to the review process.
The review makes a series of recommendations, which we will implement over the coming months and years. In implementing these recommendations, we will:
- Invest additional resources in our examinations function to implement the recommendations of the review and strengthen our capacity to deliver exams. We have established an Examinations Development and Assurance Group to lead the development, quality and alignment of exam processes across the FRCA and Faculty exams.
- Give candidates and doctors in training a greater role in our assessment processes, including more representation on College committees engaged in assessment. These representatives will also play a central role in supporting communication with candidates.
- Review the purpose of our assessments in accordance with the changes made in the curriculum and the changing nature of clinical practice in our specialties. We will consult with members and stakeholders to define this.
- Implement a unified approach to the use of, and training in, standard setting across all our assessments.
- Undertake research to inform our assessment design process. This will include three validity studies (concurrent, predictive and construct) to compare performance in the workplace as estimated by trainers and supervisors with performance in RCoA assessments. We will also undertake further research to investigate the impact of gender, ethnicity and educational background on exam performance.
Alongside our internal review process, we recognise the value of independent assurance and insights into the development and delivery of our assessment processes. To that end, we have also commissioned an external review of the FRCA, FFICM and FFPMRCA exams. The independent review was conducted by Professor John McLachlan and commenced in January 2022.
The scope of the independent review is to consider all aspects of the College’s exams, with particular focus on issues arising from the delivery of the September 2021 FRCA Final Written Exam and the October 2021 FFICM Objective Structured Clinical Examination (OSCE).
While the two reviews differ in their scope and timeframe, the recommendations from both have informed our proposed programme of development for our examinations. Having held publication of the internal review to allow us to draw comparison with the independent review, we are publishing it now to avoid delay. We will publish Professor McLachlan’s report in February 2023, following consultation with our faculties and examiners.
External review
The Royal College of Anaesthetists and the Faculties of Intensive Care and Pain Medicine are committed to delivering fair, robust and high-quality examinations that are aligned with GMC standards and best contemporary practice and that uphold and improve standards of patient care.
Two reviews of examinations delivered by the College were completed during the period January 2020 to July 2022. The first was an internal review with the scope to look at all aspects of the FRCA examination against current best practice in postgraduate medical assessment.
The second was an independent review commissioned by the College to provide independent assurance and insights into the development and delivery of our assessment processes, alongside our regular internal reviews of the FRCA. The review considered all aspects of the FRCA, FFICM and FFPMRCA exams, including an error in the delivery of the September 2021 FRCA Final Written Exam and an unexpectedly low pass rate for the October 2021 FFICM Objective Structured Clinical Examination (OSCE).
The independent review was conducted by Professor John McLachlan and we published his report in full on 16 February 2023. We welcome the findings of this review and encourage members to read our full response.
We will address the recommendations, many of which align closely with those of our recent internal review of the FRCA, working closely with the Faculties of Pain and Intensive Care Medicine. The reviews will inform our developmental roadmap, and we will implement changes with care over the coming months and years, taking into account the needs of candidates and consulting with stakeholders.
While it is likely that changes will be made to the FRCA exam format in future, the current FRCA examinations remain valid, reliable, fit for purpose and are approved by the GMC. We will ensure exam candidates are given at least 12 months’ notice of any significant changes to the FRCA exams in order to facilitate good preparation and support. Therefore, anaesthetists in training and prospective candidates should not change the way they prepare for the current sets of exams until formally notified of any changes.
We have published a summary of our proposed timetable for implementation of the changes resulting from the review, and we highlight the main themes relevant to the FRCA examinations here. We will keep our members and the GMC informed of our progress through regular updates on our website and member communications.
Our programme of improvement
In considering the recommendations from both the independent review and our internal review, we will:
- Invest additional resources in our examinations function to implement changes arising from the review and strengthen our capacity to deliver exams. We have established an Examinations Development and Assurance Group (EDAG) to lead the development, quality and alignment of exam processes across the FRCA and Faculty exams, where appropriate. This group will have an independent chair and comprise examiners from all exams delivered by the College, members of the exams team, lay and trainee representatives and a specialist educational assessment expert.
- Give candidates and anaesthetists in training a greater role in our assessment processes, including more representation on College committees engaged in assessment. These representatives will also play a central role in supporting communication with candidates.
- Review the purpose of our assessments in accordance with the changes made in the curriculum and the changing nature of clinical practice in our specialties. We will consult with members and stakeholders to define this.
- Implement a unified approach to the use of, and training in, standard-setting across the FRCA exams and all assessments delivered by the College.
- Undertake new research to inform our assessment design process. We are committed to investigating the impact of gender, ethnicity and educational background on exam performance. We also aim to undertake validity research to compare exam performance with performance in the workplace as estimated by trainers and supervisors with performance in RCoA assessments.