7.1       Evidence for the Annual Review of Competence Progression [ARCP]

Award of the CCT depends on having completed a GMC approved programme of training and having demonstrated key knowledge and capabilities in the course of assessments. Trainee progress through the curriculum is reviewed at the ARCP process and this determines the learner’s further progress. 

It is the responsibility of the trainee both to understand what evidence will demonstrate appropriate progress and to accumulate and tabulate this evidence. Inability to collect and organise the evidence is itself taken to be a significant failing which is likely to be reflected in other aspects of professional life. To this end, it is emphasised that it is the trainee’s responsibility to ensure that the assessments are completed within in each unit of training.

The ARCP is organised and operated by Postgraduate Deans. Its general principles are laid down by the GMC and are described in the ‘Gold Guide’. The RCoA is responsible for advising on the specific evidence that is required in its specialty training programme.

The Trainee will work with their educational supervisor to develop evidence of satisfactory progression through their agreed learning. A summary of this evidence will then be presented by the educational supervisor to the ARCP in the Educational Supervisors Structured Report (ESSR).

7.2     RCoA Fellowship examinations

7.2.1       Tests of knowledge for the award of a CCT

The tests of knowledge are important milestones for progression in the training programme which coincides with the progression from core to intermediate level; and from intermediate to higher level. The examination is cited as one of the methods of assessment for each competency in the professionalism of medical practice and core and intermediate level units of training. The blueprint of each unit of training and component of the RCoA Primary examination is in Annex B. The blueprint of each unit of training and component of the RCoA Final examination is in Annex C. Trainees should be aware that the questions for each component of the Primary examination may be drawn from any of the competencies (skills and knowledge) from the core units of training, basic sciences and professionalism of medical practice (Annex A). Questions for the Final examination may be drawn from those areas cited for the Primary examination and the competencies (skills and knowledge) from the intermediate units of training, advanced sciences and Professionalism of medical practice (Annex A). 

The RCoA Fellowship examination is a GMC approved assessment for the award of a CCT.

7.2.2       RCoA Fellowship

The RCoA Fellowship is awarded to those individuals who have successfully completed the RCoA Final examination. The awarding of the Fellowship is independent of the training programme and is a College decision. While the Fellowship examinations are embedded in the CCT in Anaesthetics curriculum for the award of a CCT, the Fellowship examination is open to other individuals who meet the eligibility requirements defined in the RCoA Examination Regulations²⁴. The Examination regulations also define those qualifications, which the College accept as exempting qualifications from the Primary examination²⁵.

7.2.3       Faculty of Pain Medicine [FPM] Fellowship

The FPM fellowship is awarded to those individuals who have successfully completed the FPM Fellowship examination and meet the requirements set by the FPM. Successful completion of the examination is not a requirement for the award of a CCT in Anaesthetics and is optional for pain medicine trainees. 

All pain medicine competencies annotated with an E [core, intermediate, higher and advanced] may be examined in the FPM fellowship examination and can count towards the assessment of individual clinical competencies. The exam does not replace the FPM defined number of workplace based assessments. [See section 8.3.3]

More information on the FPM examination can be obtained by contacting the FPM.

7.1     Workplace-Based Assessments [WPBA]

7.3.1       Choosing Appropriate Assessment Tools

The curriculum was reviewed and the cognitive learning outcomes that lend themselves to conventional testing by written and oral examination were marked for formal examination. Those cognitive, psychomotor and behavioural learning outcomes that remained have been allocated to appropriate instruments for WPBA. As an outcome-based curriculum identifies very large numbers of items, a strategy of sampling assessments has been selected in order to make the assessment task manageable and to minimise the disruption of normal work and the possibility of increased risk to patients.

An assessment tool has been identified for every competency in the curriculum. Where possible more than one methodology is identified so that it is possible to triangulate performance. It is intended that a sample of these assessments will be undertaken by each learner. It is also possible that some aspects of or a module outcome could be assessed using other methods, for example, satisfactory completion of an Advanced Life Support course as evidence of completion of a Core Clinical Learning Outcome from the resuscitation module. 

The choice of which outcomes to assess is defined by each School of Anaesthesia and the formative assessments themselves are left to the learner and their educational and clinical supervisors. This will depend on the opportunities that the clinical work presents and the learner’s needs. 

Schools of Anaesthesia are to ensure that the curriculum is adequately sampled through either the use of WPBA and/or other formalised accredited training courses²⁶ to provide the necessary evidence, which along with the professional judgement of trainers, demonstrates whether a trainee has met the standard required by the Completion of Unit of Training form or the attainment levels defined for intensive care medicine.

7.3.2       The Available Assessment Methodologies

Units of Training can be signed off as complete when supervisors are satisfied that the learning outcomes have been achieved. Supervisors should draw upon a range of evidence including the logbook of cases completed, workplace-based assessments and consultant feedback to inform their decision. The logbook review should consider the mix of cases, level of supervision and balance of elective and emergency cases, if relevant, for the unit. Any other evidence provided by the trainee, such as course attendance certificates can be reviewed at this time. All hospitals must identify appropriate designated trainers to sign off each unit of training. Each trainer should be familiar with the Core Clinical Learning Outcomes for the unit of training and be able to provide guidance for trainees who have not yet achieved the learning outcomes. It is possible for a trainee to have all WPBAs signed off but not successfully complete the unit because of, for example, professional attitudes or inappropriate non-technical skills i.e. characteristics which will be captured by consultant feedback.

A pragmatic approach to the choice of assessment methods has been adopted. These are the A-CEX, DOPS and CBD. These methodologies have a practical utility attested to by experience in their use and at least some objective evidence that correctly applied they have validity and reliability. The ALMAT and ICM-ACAT tools have been added to allow the observation/assessment of a whole anaesthetic case or list/clinic/ward work.

7.3.3       How many workplace-based tests

The purpose of the anaesthetic WPBAs is to provide evidence towards achievement of the Core Clinical Learning Outcomes for each Unit of Training.  It is not to tick off each individual competence. The number of observations of work required is a minimum of one for each assessment type as identified in the respective level of training blueprint but the final number of each will ultimately depend on the individual trainee’s performance and advice from the trainer responsible for overseeing the specific Unit of Training in the hospital concerned. 

As noted in section 5.6 [Clinical teaching and supervision] the placement of a trainee with a consultant is always a clinical teaching opportunity and this includes assessment. Any such teaching opportunity should be accompanied by feedback from trainer to trainee [this includes senior trainee to junior trainee, when the opportunity arises] and during such feedback it would often be possible to complete an assessment such as a DOPs or A-CEX.

Taking account of the above there is concern about recommending the number of assessments required; A-CEX, ALMATs, CBDs and DOPs should all be used to inform individual completion of units of training; these must sample widely and in sufficient numbers. Despite concern about recommending a minimum, trainees should successfully complete a minimum of one of each assessment type identified in the WPBA blueprint for each unit of training in annexes B, C, D and E or as advised by the trainer responsible for overseeing the specific Unit of Training in the hospital concerned. The exception is advanced pain medicine; the minimum numbers of assessments are 4 A-CEX, 4 CBD, 6 DOPS, 1 MSF and 2 case studies. One assessment may be linked as evidence to more than one unit of training [except for advanced pain medicine], for example, an A-CEX for regional anaesthesia may also be linked as evidence to sedation and orthopaedics. When deciding to use a WPBA for more than one unit, the assessor must ensure that the assessment appropriately covers aspects of the syllabi for the units of training intended to be assessed. 

The number of assessments for intensive care medicine has been set by the Faculty of Intensive Care Medicine. The assessment requirements are detailed in section 2 of Annex F.

When a trainee’s performance gives cause for concern, more assessments will be needed. It is the responsibility of the trainee to provide at their annual review what they consider to be evidence of performance and progress. They will need evidence for each unit of training or section of the curriculum they have undertaken. It is the educational supervisor’s responsibility to help the trainee to understand what that evidence will be appropriate in their specific circumstances.

Once again it must be stressed that there is no single, valid, reliable test of competence and the ARCP will review all the evidence, triangulating performance measured by different instruments, before drawing conclusions about a trainee’s progress.

The MSF unlike the other workplace based assessments provides feedback on professional attitude and behaviours from a wide range of individuals who have worked with the trainee in the current training year. Other WPBA are a snap shot in time covering a clinical episode where the MSF is used to measure a trainee’s performance across a broader period of time. 

Trainees are required to have at least one MSF completed for each training year. The MSF completed during the ICM and pain medicine rotations satisfies this requirement. However, if concerns have been raised either verbally by staff or as comments on the other WPBAs, then it is appropriate to conduct further MSFs as required. There must be at least 15 assessors and a minimum of 8 responses for each MSF. If the minimum number is not received, the MSF must be redone. Before the MSF is sent out to recipients via the e-Portfolio, the trainee must provide their educational supervisor with a list of names they propose to complete the MSF. The educational supervisor approves the list to ensure that the sample provides an adequate cross section of medical and non-medical staff. The MSF response window will be open for one month.

Consultant feedback, and feedback from other approved anaesthetist trainers, is also an important source of evidence when assessing trainees’ performance. This means of assessment is valuable in identifying trainees who are performing above and below the standard expected for their level. Consultant feedback differs from MSF as it concerns a trainee’s progress in a specific unit of training only. MSF seeks feedback from the multidisciplinary team, including consultants, on overall professional behaviour.

7.4       Values and behaviours of practice

It is difficult to assess ‘professional attitudes and behaviours’; they cannot be directly observed but are demonstrated by actions. The taxonomy used for these is the CANMEDS classification that was developed by the Royal College of Physicians and Surgeons of Canada [http://www.royalcollege.ca/portal/page/portal/rc/canmeds/framework]. Additional information has been incorporated from the GMC guidance ‘Good Medical Practice’ and from Anaesthetic Non-Technical Skills [ANTS] [http://www.abdn.ac.uk/iprc/ants w].  The values and behaviours of practice may be evident as skills or knowledge, and may be assessed directly using the assessment system. Many, however, do not manifest themselves except in the performance of another competence; the WPBAs have been developed to allow identification of these traits.

7.5       The Annual Review of Competence Progression [ARCP]

A wide variety of information is available as evidence for the annual review. It is deemed to be the learner’s responsibility to present their reviewers with evidence of progress. Sources of information are: 

• Evidence of performance in professional examinations – if applicable;
• A reflective diary of learning experiences;
• Evidence of completion of units of training appropriate to stage of training, including 
  • WPBAs: DOPS, A-CEX, CBD and ALMAT* [Minimum of one WPBA for  each assessment type per unit of training where indicated in the assessment blueprints in the annexes or the School of Anaesthesia minimum if defined or the minimum specified for advanced pain medicine, whichever is greater; 
  • A log of clinical work undertaken;
  • Consultant feedback forms
• For intensive care medicine, WPBAs: DOPS, ICAT, I-CEX and CBD [numbers specified by the FICM and defined in Annex F];Completion of Unit of Training Form [CUT]; 
• A record of agreed targets and outcomes from interviews with their educational supervisor;
• A multi-source feedback if appropriate;
• Specific evidence of performance in areas such as research and education; and 
• Optionally: a record of a School of Anaesthesia appraisal interview

It is accepted that there is no good evidence of the validity and reliability of any of these evidences. The process of reviewing them is not arithmetic. The educational supervisor must seek to use these evidences to answer four questions:

 

 

7.6     The Workplace-Based Assessments

Training programmes in anaesthetics use workplace-based assessment as part of the assessment process for each unit of training. The workplace-based assessments are conducted using the workplace assessment tools, which consists of:

• Direct Observation of Procedural Skills (DOPS);
• Anaesthesia Clinical Evaluation Exercise (A-CEX);
• Multi-Source Feedback (MSF);
• Anaesthesia List Management Assessment Tool (ALMAT); and
• Case Based Discussion (CBD)

The DOPS, A-CEX and ALMAT are used during clinical sessions, and the assessments are based on the observed performance of the trainee’s skills, attitudes and behaviours, and knowledge. The CBD is used away from the clinical environment  – it allows the assessor to question the trainee about a clinical episode to assess the trainee’s knowledge and rationale for their actions or what they would do if presented with the clinical scenario.

7.6.1       Simulation based assessment

Simulation has an important role in teaching, particularly in rehearsing uncommon events and team training as well as a medium for demonstrating procedures and routines. The use of simulation as a means of assessment for elements of the IAC and IACOA [for example failed intubation and epidural insertion] is strongly recommended. 

7.6.2       Logbook and Portfolio 

Trainees are required to keep a record of the cases that they undertake. The level of detail of these records is described elsewhere. The RCoA has defined the categories of experience but has not stipulated the number of cases that must be undertaken. This is because it is more important to demonstrate competence than to achieve a target of experience. Self evidently a learner cannot become competent without undertaking cases and their performance must be considered in the context of their experience. In the event that assessments indicate underperformance in an area of practice the first response is to check from the logbook that the learner has had sufficient exposure to it. Incompetence in the face of what is usually sufficient exposure is a cause for concern.

The portfolio of learning is more than a logbook. It must include reflections on learning and a record of other teaching and of discussions with the educational supervisor.

7.6.3       Evidence of participation and attendance at training events

Until recently evidence of attendance at a learning session was taken to be the standard for accumulation of credits in continuing medical education. Attendance does not assure that learning has occurred but it does signify compliance with an appropriate learning plan. There are a number of aspects of training that lie on the periphery of practice such as Research Methods, Management, Evidence Based Practice, Teaching and Assessment. At present there is little focussed assessment in these areas and significant practical difficulties lie in the way of introducing summative assessment. The RCoA has at present adopted the middle ground in these areas and requires that evidence of participation in learning is presented to the ARCP. These include attendance at specific courses, evidence of presentation at local audit/quality improvement and research meetings and records, and feedback from teaching the trainee has delivered; guidance is available in the training section of the College website.

7.6.4       An Independent Appraisal

Evidence to the ARCP must include an appraisal. In many Schools of Anaesthesia this will be with the educational supervisor and will be part of the documentation relating to episodes of supervision. Some Schools conduct independent appraisal of the ARCP evidence in advance of that meeting and include this formal appraisal in the evidence for the review. This practice provides a more independent review of their training which will also include the adequacy of their educational supervision, as poor planning by the supervisor may contribute to poor outcomes by the trainee. It also provides the trainee with the opportunity to explain and expand upon the evidence they present in their portfolio.

7.7     Failure of FRCA examinations

The process to appeal against a failure of a College FRCA part examination is defined in the Primary and Final Examinations: Representations, Reviews and Appeals Regulations, available on the College website in the Examinations section [www.rcoa.ac.uk].

7.8     Assessors

7.8.1       Workplace assessors

Workplace assessments in the anaesthetics programme are conducted by consultants, speciality doctors and trainees. In accordance with GMC standard 5.9 for postgraduate training²⁷, trainees must only be assessed by someone with appropriate expertise in the area to be assessed. 

Taking into account GMC standard 5.9, it is appropriate that senior trainees be given the opportunity as part of their training [to achieve the competencies in annex G] to assess junior trainees. It is the responsibility of the Clinical Supervisor to determine whether it is appropriate for the trainee to conduct the assessment on a junior trainee. The only exception is assessments for the IAC and IACOA, where the assessments must conducted by either a consultant or specialty doctor. It is also appropriate that some assessments may be delegated to non-medical staff members who have the required expertise, for example a scrub nurse assessing a trainee scrubbing for theatre.  When a non-medical staff member performs an assessment, the actual workplace based assessment must be approved by a consultant or specialty doctor and the assessment result based on the recommendations from the non-medical staff member. When a non-medical staff member performs the assessment, their name should be included in the comments area of the workplace based assessment tool.

7.8.2       FRCA Examiners

Examiners for the Primary and Final FRCA examinations are appointed by the College. The process and criteria for the appointment of examiners is defined on the College website (www.rcoa.ac.uk).

All examiners undergo a training programme to ensure consistency of assessment in the oral examinations for both the Primary and Final; they undergo appraisal and monitoring throughout the examinations process for quality assurance purposes.