IV.    Evaluation of Resident Performance:

 

Where possible, anesthesia rotations are scheduled for a minimum of two four-week blocks to facilitate the anesthesia co-ordinator's evaluation of a resident's performance.  In the PGY2 year, thirteen four-week block rotations in anesthesia are scheduled.  In-term evaluations are provided by the resident co-ordinator for each three consecutive block rotations in anesthesia (or earlier at the co-ordinator's discretion).  Final rotation evaluations are provided at the completion of each block rotation.  During the final PGY5 year, rotations of one (four-week) block in anesthesia may be scheduled depending on a resident's specific need.  Faculty input concerning each resident's clinical performance is used to compile a composite evaluation by the anesthesia resident co-ordinator.  The evaluation is then discussed with the resident, signed by the resident, and forwarded to the office of the program director, where it becomes part of the residents training documentation.  The signature of a resident on their evaluation does not imply that they agree with the evaluation, rather it simply acknowledges that the evaluation has been discussed with the resident.  A resident may choose to enter specific statements of clarification when signing their evaluation.  An evaluation may be appealed to the Program Director and University as per the University of Ottawa Policies and Procedures for the Evaluation of Postgraduate Trainees policy document (see policy page 150).

 

It is the responsibility of each resident to ensure that they obtain an evaluation for all rotations, including ICU, medicine, elective, and all anesthesia rotations.  If the resident is unable to obtain an evaluation within two weeks of asking the rotation supervisor, the resident must notify the Anesthesia Program Director.  The rotation evaluation is the only proof that a resident has completed their rotation, and is required by the Program Director to complete a Completion of Clinical Training form (CTC) which enables a resident to proceed to their RCPSC examinations.

 

a)        Metrics In-Training Examinations: PGY2 residents complete Part I, II and III of the Metrics Examinations on their 1^st day, after 1 month, and after 6 months of their PGY2 year respectively.  These examinations provide a scoring percentile ranking of their knowledge base (using multiple choice questions) with other residents across NA at the same stage of training.  The results are individualized and identify specific knowledge areas that the resident should review.

 

b)        ABA In-Training Examinations: PGY3 and PGY4 residents complete the American Board of Anesthesia Multiple Choice In-Training Examinations in July of their PGY3 and PGY4 years.  The ABA exams provide practice at writing MCQ examinations (in preparation for the RCPSC written examination) and objective individualized feedback on their performance, highlighting areas of weakness in their knowledge base.  A resident's performance on the ABA examination has been correlated with their success on the RCPSC written and oral examinations (Kearney R., Sullivan P.  Anesthesia Resident Performance on the Metrics In-Training Examinations and Correlation with Success on the RCPSC Examinations.  RCPSC abstract Sept. 98; CJA manuscript submitted July '99).

 

c)         Core Program Examinations: There are 12 Anesthesia subject areas covered in a three year rotating Core Program.  Each Core Program concludes with a MCQ examination.  Residents are expected to obtain a mark of 70% or greater.  Each resident is provided the results of the core program examinations, including the mean mark of all residents, their mark, and their rank among the residents completing the examination.

 

d)        Oral Examinations in Anesthesia: PGY2 - PGY5 residents are given formal oral examinations in anesthesia each December and May with two examiners and standardized questions for each oral exam session.  The RCPSC oral examination format (with associated time constraints) is used to simulate a true oral examination.  Unlike the RCPSC oral examinations, residents are provided with verbal feedback from their examiners immediately following their exam.

 

II.  Evaluation of Anesthesia Faculty Teaching and Anesthesia Rotations:

 

An on-line web-based mechanism will be implemented for resident evaluation of both Faculty Teaching and the Anesthesia clinical rotations (including Anesthesia, Internal Medicine and Critical Care rotations).  Residents are required to submit their name and CPSO educational license number, when completing the on-line evaluations.  This allows our academic secretary (Lynne McHardy) to verify the data input, and identify which residents have, and have not completed their evaluations.  The information submitted will be kept strictly confidential and will not be linked to the resident when evaluation reports are generated. 

 

Residents will be requested to complete the evaluation forms of both anesthesia teaching faculty and anesthesia rotations every 6 – 12 months.  Year-end reports will be generated, to provide feedback to the faculty and teaching hospitals.  The analysis at year-end hopefully also ensures an anonymous reporting and feedback mechanism for our residents.  Residents are asked to respect their ability to provide anonymous feedback and be responsible in their comments.  Residents who have not worked with specific faculty members should not evaluate that specific faculty member. 

 

The computerized results of the residents' feedback on faculty teaching will be summarized and made available to the department chiefs, program director, and department chairman.  The computerized results of the evaluations of the anesthesia rotations will be made available to the Residency Training Committee, resident co-ordinator's, program director, and department chairman for appropriate action.

 

Evaluations for Internal Medicine and Critical Care rotations should be completed at the end of each rotation.  This information will be kept strictly confidential and used only by the Anesthesia Residency Training Committee.

It is essential that there be a fair and transparent evaluation system for residents within the Postgraduate Training Programs of the Faculty of Medicine.  As well, it is critical that we use the most appropriate and relevant evaluation tools and that every effort is made to standardize evaluation processes across programs.

 

Through our evaluation system we intend to maintain the highest possible educational standards and to continuously improve resident performance.  Evaluation practices need to be consistent with the requirements of the Royal College of Physicians and Surgeons, the College of Family Physicians of Canada, the College of Physicians and Surgeons of Ontario and other relevant bodies.

 

The procedures outlined below provide for an explicit transparent series of steps that need to be taken when evaluating a resident who encounters significant difficulties.  The procedures are intended to provide a graded response to residents with difficulties.  The vast majority of residents will not encounter significant problems in training - in fact the challenge to evaluation will be to help them develop beyond their satisfactory levels of competence.  For a few residents with difficulties, they need to be reassured that they are dealt with fairly.  The following procedures allow for close monitoring of residents with difficulties, a period of remediation where needed, followed by a period of probation if required.  However these steps need to be individualized.  Circumstances may exist in which it may not be appropriate to place a resident in a remediation program prior to probation for example.   Serious problems with professional conduct and attitude may justify probation or dismissal as an initial step.

 

It is important to emphasize that a supervisor or a Program Director can recommend an immediate suspension from clinical responsibility of a resident if it is deemed that patient care is at risk.  In this circumstance, the Postgraduate Dean should be notified immediately by the resident's Program Director.  In this case, the Postgraduate Dean, in consultation with the affected resident and Program Director, will make an inquiry of the situation and recommendation on how to proceed.  This recommendation should occur within 1 (one) month of notification, during which time the resident will be suspended from the program.  Following this, the Postgraduate Evaluation Sub-Committee should be asked to review the file.

 

PRINCIPLES OF RESIDENT EVALUATION

 

There are unique problems associated with evaluation during the Residency.  In addition to providing appropriate formative and summative assessment through objective tests, the need for accurate performance assessment through in-training evaluation and appropriate remediation represents unique challenges.  To provide a uniform evaluation system that is valid, reliable, accountable, and feasible.  The following principles should be respected:

 

1. The evaluation process must be tied to objectives.

 

a)   Objectives should be sampled as comprehensively as possible.

b)   Every training program must have written learning objectives and residents must be provided with these objectives upon entering the program.

 

c)    There must be clear Rotation specific objectives provided to residents and faculty in advance of the Rotation to guide resident learning and assessment strategies.  All essential competencies and the specific behaviours that characterize these competencies must be outlined.

 

d) The learning objectives need to be reviewed with the resident at the beginning of each rotation or educational experience.

 

e) The evaluation system should assess competence in different settings and using different modalities as appropriate.

 

2. The evaluation process must reflect the principles of learning which constitute the basis of Residency education.

 

a) Assessment should evaluate the performance in a setting as close to clinical practice as possible.  At times, more objective assessment of competence will be necessary in settings such as written exams, observed structured clinical exams (OSCE) and oral exams.  The combination of observed performance assessment and objective structured exams is ideal.

 

3. Evaluations should be both valid and reliable.

 

a) In addition to the evaluation of all rotational objectives using the most appropriate tools, input should be sought from multiple observers with different perspectives (nurses, colleagues, and patients).

 

4. All evaluations should provide useful feedback to assist Resident learning.

 

a) There should be mid-Rotation formative feedback.

 

b) The final rotation evaluation should be reviewed with the Resident individually and as close to the completion of the rotation as possible.  Strengths and weaknesses should be identified for future emphasis.

 

c)  Objective examinations should be given that reflect RCPSC and CFPC certification examinations to better prepare Residents.

 

5. The grading of Resident performance should be fair and equitable.

 

a) Systems of grading, leading to the successful completion of a rotation, year, and the program (as well as completion of the Certificate of Completion of Training form) must be clear, applied uniformly and made explicit.

b) Performance(s) that would lead to failure of a Rotation, remediation, probation, failure of a year or discharge from the program must also be clear, applied uniformly and made explicit.

 

6. Systems of Resident evaluation should be updated and continually critiqued.

 

7. The Program Director must ensure that all residents are familiar with the rules and regulations covering evaluation and promotion.

 

8. The evaluation process must be transparent and applied fairly.

 

9. The evaluation process must be confidential with respect to verbal feedback and documentation.

 

SPECIFIC STEPS IN THE EVALUATION OF RESIDENT PERFORMANCE

 

1. The evaluation process must be described for the resident at the outset of a rotation.  This includes discussion of the tools for evaluation, timing of evaluation, and identification of those who will be doing the evaluation.

 

2. There must be a written evaluation and oral feedback at the end of each rotation.

 

3. The supervisor of the rotation should provide the evaluation, with the understanding that the supervisor may receive relevant input from other health care team members who have participated in the rotation.

 

4. Written evaluations must be conducted in a timely manner and be completed and signed as close to the end of the rotation as possible.

 

5. There needs to be mid-rotation verbal feedback; if significant problems are identified, this needs to be documented and signed by the resident and person providing the feedback.

 

6. It is expected that there will be both verbal feedback to the resident as well as written evaluations which need to be discussed with the resident.  It is important that immediate timely verbal feedback be given throughout rotations and more specifically at the mid-point evaluation.

 

If concerns are noted regarding a resident's performance, it is expected that the resident will be notified in writing at once by the supervisor, with copies to the resident's Program Director.

 

7. It is expected that both the supervising physician and the resident sign the evaluation, with the understanding that the resident's signature does not necessary imply that he/she agrees with the evaluation.  The resident should have the opportunity to add written comments to the evaluation.  In the exceptional case when the resident refuses to sign, this should be documented on the evaluation.

 

8. There is an expectation that every six months there will be an evaluation made by the Program Director, or delegate, discussed with the resident and signed by both parties.

 

9. Only examination results and appropriately signed documentation or correspondence should be used in the evaluation of residents for recommendations regarding promotion, remediation, probation, and dismissal.

 

10. Clinical supervisors will make recommendations pertaining to resident evaluation to the Program Director and the Residency Program Committee.  This committee will make decisions (at times in consultation with the Assistant Dean, Postgraduate Medical Education) regarding the successful completion of a Rotation, Year or the Program as well as completion of the Confirmation of Completion of Training (CCT) form.  All decisions regarding probation or dismissal must be made by the Residency Program Committee and then ratified by the Faculty Postgraduate Evaluation subcommittee and the Assistant Dean of Postgraduate Medical Education.

 

UNSATISFACTORY PERFORMANCE

 

If an evaluation deems that a resident has performed in an unsatisfactory manner, both the supervisor and the Program Director need to discuss this with the resident.  It is important to point out that a resident can receive unsatisfactory evaluations not only for academic difficulties, but also for failure to live up to the standards of ethical professional behaviour as outlined in the document entitled "Faculty of Medicine - Standards of Ethical and Professional Behaviour" (Senate approval, March 1994, appended).

 

If a resident does not accept an unsatisfactory evaluation and recommended procedures to correct the unsatisfactory evaluation, he/she has the right to appeal according to the Postgraduate Medical Education Appeal Mechanism, Faculty of Medicine, University of Ottawa (approved March 1997, appended).

 

FAILURE OF A ROTATION

 

The Faculty of Medicine has developed the following standardized criteria, applicable to all residency training programs to ensure consistency in determining whether a resident has failed a mandatory, selective, or elective rotation.  Clinical supervisors will make recommendations pertaining to resident performance, however, the Residency Program Committee will make final pass/fail decisions for all rotations based upon all available documentation.  A resident may be deemed to have failed a rotation if he/she has met any of the following criteria:

 

1. An unsatisfactory evaluation in any domain of the rotational In-Training Evaluation report (ITER).

 

1.    Documentation that a resident, regardless of their clinical performance during the rotation, has not satisfied the standards of ethical professional behaviour (Faculty of Medicine, Standards of Ethical and Professional Behaviour, Senate approval March 1994).

 

Failure of a rotation based on criteria 1 and 2 may require remediation or probation, and the successful completion of the rotation at some point in the future.

 

FAILURE TO COMPLETE A ROTATION FOR NON-ACADEMIC REASONS

 

1. This designation applies when there is an absence for legitimate reasons from a significant component of the rotation.  The amount of time is to be at the discretion of the Program Director and relates to an inability to achieve the educational goals and objectives established for the rotation.  Absence for reasons not sanctioned by the Faculty will constitute grounds for failure.

 

Failure to complete a rotation based on this criteria alone (absence for legitimate reasons), will not require remediation or probation, but the rotation must be successfully completed at some point in the future.

 

FAILURE OF A YEAR

 

Residents must be promoted from year to year by the Residency Program Committee based upon available documentation.  A resident may be deemed to have failed a year based upon the following criteria:

 

1. The unsatisfactory completion of a remedial rotation.

 

2. The failure of a rotation where the schedule does not allow for the appropriate period of remediation.

 

3. Consistent difficulties identified throughout training that are considered by the Residency Program Committee to warrant a failure in the absence of failed rotation.

 

Failure of a year will lead to the repetition of that year or dismissal from the program under the conditions listed below, which must be ratified by the Faculty Postgraduate Evaluation subcommittee.

 

DISMISSAL FROM THE PROGRAM

 

The Residency Program Committee in conjunction with the Assistant Dean, Postgraduate Medical Education and the Faculty Postgraduate Evaluation subcommittee, will make decisions regarding expulsion from the program.  A resident may be asked to leave the program if:

 

1. A resident fails a probationary period.

 

2. A resident is felt to have a deficit that cannot be remediated.

 

3. A resident does not maintain the standards of the profession as described in the Standards of Ethical and Professional Behaviour (Senate approval, March 1994).

 

ATTESTING TO THE COMPLETION OF TRAINING:

 

The Program Director in conjunction with the Assistant Dean, Postgraduate Medical Education will make decisions pertaining to the attestation of completion of training based on all available documentation.  Satisfactory completion of all In-Training Evaluation in itself will not ensure the attestation of Completion of Training.

 

REMEDIATION

 

If a resident has failed a rotation, there may be plans clearly set out to monitor his/her performance more closely in subsequent rotations or plans made to repeat the failed rotation.  Additionally a formal period of remediation may be recommended up to (but no greater than) a year.  Remediation is a structured program aimed at helping a resident to correct identified weaknesses.  Programs should offer remediation to residents who have had failing rotations that require fine-tuning in order to successfully complete them.  It is acceptable for the decision regarding remediation to be taken at the program level; however the Assistant Dean for Postgraduate Medical Education has to be notified of the decision to arrange remediation prior to its onset for any given resident.

 

1 .     Recommendations for remediation must be brought to the Residency Program Committee by the involved supervisor(s) and the Program Director.

 

2.      The Residency Program Committee should take decisions regarding remediation only after consultation between the resident and the Committee.

 

3.      The nature and length of the remediation period need to be stated by the Residency Program Committee.

 

4.      If rotation(s) are required outside the resident's program, they should be discussed and arranged with the respective program director(s) prior to finalizing the period of remediation.

 

5.      The resident must be informed of the decision for remediation in writing, with:

 

a)      Details regarding the reasons for remediation,

b)      The specific areas of deficiency,

c)      The educational objectives during remediation, and

d)      The possible outcomes of the remediation.

 

6.      Evaluation procedures for the period of remediation must be similar to those used for all residents in all rotations, though they may be more intensive and focused.

 

7.      Remediation should be up to the same length as the failed rotation.

 

8.      At the end of a remediation period, the Program Director, after review by the Residency Program Committee, must inform the resident as to the outcome.  Possible outcomes include having successfully passed the remediation period, a recommendation for further remedial training, a recommendation for the resident to be placed on probation, or a recommendation for dismissal.

 

A resident will be allowed only two rotations for remediation in any given training program.  An additional failure or a failure to pass one of the remediation periods would lead to a recommendation for probation, or for dismissal.

 

The resident can receive credit for training which is successfully completed during a period of remediation.

 

A resident can appeal the decision requesting a period of remediation.

 

There may be instances in which failure to complete a program of remediation does not warrant

probation as the next step, but dismissal from the program may be more appropriate.

 

PROBATION

 

If a resident has not successfully completed a period of remediation, or if the Residency Program Committee determines that some academic or other difficulties are such that an immediate period of probation is more relevant, this can be instituted.  Probation is a period of training like remediation, during which the resident is expected to correct identified weaknesses and/or deficiencies.  However, unsuccessful completion of a probation period could lead to dismissal from the program.

 

Probation should be applied in cases where a resident has failed, over the period of training to successfully complete a program of remediation, has failed at least two rotations or has successfully remediated two rotations and subsequently failed a third one.

 

1 .     The proposed probation should be made on the advice of the Residency Program Committee and the Program Director and should convey recommendations in writing to the resident concerned.

 

2.      Any proposal for probation for a resident has to be ratified by the Residency Program Committee, and circulated to the resident in question, the Program Director and the Head of the Department.

 

3.      The proposed probation has to be endorsed by the Faculty Postgraduate Evaluation Sub‑Committee prior to commencement.

 

4.      Probation could be for a period up to six months.

 

5.      As with remediation, the details of probation must include the reasons for probation, specific areas of deficiency, educational objectives during probation, the evaluation method and evaluators, and the possible outcomes.

 

6.      Possible outcomes of a probationary period include full reinstatement, an additional probationary period, or dismissal from the Program. Extension of training may be required, to be negotiated on an individual basis.

 

7.      The Faculty Postgraduate Evaluation Sub-Committee must review the results in order that it may advise the Residency Program Committee on the appropriateness of the recommendations.

 

8.      Following a period of probation, the results and recommendations from the Residency Program Committee need to be conveyed to the resident and the Head of the department.

 

The resident can receive credit for training which is successfully completed during a period of probation.

 

Throughout the entire process of evaluation, a resident must be clearly informed of his/her rights to appeal.