Yes, I failed the Specialty Examination of the Royal College of Physicians and Surgeons of Canada (Royal College). This is a fact. But that does not preclude me from expressing what I consider to be wrong.
You may feel this is the story of someone that cannot tolerate failing, but I can assure you that this is not the case. Based on discussions with many colleagues in different specialties, it is likely that my story is shared by others, either because they faced the same situation or because their own reflections led to similar perceptions of the process.
My score at the written examination was between 65 and 69.9 per cent with a pass rate at 70 per cent, leaving me between one and seven questions short of passing. However, since I had identified a factually inaccurate question, I decided to request a Formal Review. The Summary of Performance given to candidates does not detail which questions were failed, meaning I had a legitimate concern that what was an accurate answer was improperly marked and led to my failure. In addition, in my request for the Formal Review, I highlighted concerns that if one question was inaccurate, others may have been as well. I was indeed surprised when sitting for the examination that there were some inadequacies in the examination questions list. Despite providing evidence, I was not granted a Formal Review.
I had identified a factually inaccurate question.
As I gathered information regarding the processes in place at the Royal College, I came to the conclusion that the processes do not follow the standard that the college imposes on other institutions. The opacity of the examination process, the lack of transparency and the duty to fairness must be addressed.
The examination process and appeal do not permit anyone to review the content of the examination as it is deemed “confidential to the Royal College and not shared with candidates.” There is therefore no possibility for the candidates to check the accuracy of the expected answers, not even for the sake of understanding where they failed.
The Royal College insists the questions are designed by “recognized content experts” and that a committee selects the questions deemed suitable for the examination. When a request for appeal is made, there is limited chance of success since candidates do not have access to anything tangible. Moreover, the Royal College does not accept appeal requests for what it considers to be an “alleged error in content.” Despite providing detailed information to the Royal College to support my claim as evidence-based, this did not suffice.
The Royal College also indicated that psychometric analyses help identify errors. I do not share that opinion since psychometric analyses do not have the possibility to identify inaccuracies, but rather only a pattern of response from the candidates. During a review course prior to the examination, a participant asked about the question that I reported as factually inaccurate (in a slightly altered stem but with the same factually inaccurate concept). If the psychometric analyses had performed as adequately as the Royal College argues, this question would have been removed from my examination question list.
Lack of transparency
Passing or failing the specialty examination has tremendous professional impact on candidates. Thus, the Royal College has a responsibility to engage in a trustworthy relationship with candidates who aim to become fellows of this institution. Unfortunately, I do not believe that such a relationship exists.
No clear and detailed feedback is provided to the candidates to allow them to improve.
The Royal College examination process is not transparent, which is inconsistent with the rules it makes when accrediting the Residency Program and external Continuing Professional Development (CPD) programs. In terms of the Residency Program, it stipulates that “the central policies and processes that address […], assessment, […], and appeals are regularly reviewed, transparent, and effectively applied.” Unfortunately, there is no similar call for transparency in its own examination and appeal processes. For the CPD programs developed by other agencies, the college states that “the self-assessment program must provide detailed feedback to participants on their performance to enable the identification of any areas requiring improvement through the development of a future learning plan. Providing specific feedback on which answers were correct and incorrect with references […].” But in the college’s case, it does not provide clear and detailed feedback to its candidates to allow them to improve. It only provides a Summary of Performance that details a percentage by subcategories in the Specialty. Since some of the questions were quite inadequate to test the readiness to practice in my opinion, I do not really see how this summary can be of any particular use for future improvement.
Duty to fairness
As I was researching the Royal College, I was surprised that I could not identify a monitoring entity that oversees the activities of the Royal College and ensures that it follows ethical and legal practices. However, I did come across a landmark case in administrative law that provides guidance to ensure duty to fairness in the decision-making process of administrations. In my opinion, the Royal College, in the context of the examination process, does not follow any of the five criteria established in Baker v. Canada (1999).
A positive decision made by the Royal College gives a candidate the ability to work independently. Thus, the decision-making process resembles an administrative judicial process. Therefore, the principles of procedural protections should be in place. In other words, the Royal College should not have the power to restrict someone’s practice without giving the candidate a fair appeal process. The fact that it denied my claim without any explanation is not in keeping with these principles. Moreover, not being given access to any evidence to support an appeal claim does not meet the principles detailed in Baker v. Canada (1999).
A fair and transparent process needs to be in place.
When analyzing the Royal College’s processes in the light of this landmark case, it is important to consider the impact of passing the examination. Remaining a non-FRCP/SC practitioner not only has a significant impact on someone’s professional, but also on a personal level. Notably, having to spend hours studying again for this examination without any clarity as to how to study differently is not a trivial issue.
In my opinion, it is important that an agency such as the Royal College act as a gatekeeper of Specialty Medicine in Canada. It permits us to have high quality and homogeneous training and practice across the country. However, a fair and transparent process needs to be in place.
I am not going to debate the utility of the written examination; others already have done it. But I can doubt the accuracy of some questions (beside the one that was factually inaccurate) since a unique answer is not always possible. The Royal College has control over information that it will share, leaving candidates with little to contest. In an answer to one of my communications, the college expressed the fact that in its view, its processes are fair. Unfortunately, I continue to question that claim given how the Royal College governs its processes.
In my case, I know that I do not have any choice other than registering for the examination again. I just hope that in the meantime, the concerns I raised are heard and that the processes are improved. I also hope that presenting my case will help those who may, like me, have felt defeated; maybe some will feel that they can also speak up.
I am sharing a longer version of this article with the Minister of Health, the Minister of Mental Health and Addictions and other Parliamentarians. You can read that version here. Publicly voicing concerns may not be sufficient to allow for a change in process; engaging the decision-makers who have the power to impose a change may be necessary.