Picture two relatively healthy female patients, both in their 50s. Neither “Mrs. Smith” nor “Mrs. Johnson” has a family history of breast cancer or any other significant risk factor for this disease. Each woman has been a patient at the same family practice and, over time, each has established a positive, trusting relationship with their doctor, whom we’ll call “Dr. Williams.”
During separate health visits, Williams takes the opportunity to discuss the benefits and downsides of mammography for breast cancer screening. But in the end, Smith schedules a mammogram, while Johnson does not.
Did Williams provide better quality care to Smith?
These kinds of measurable indicators of primary care performance are a core component of the Ontario government’s controversial Patients First bill, reintroduced on October 6.
Most would agree that understanding primary care performance is an important undertaking. Asking these kinds of questions helps providers know if they’re doing a good job, where deficiencies exist and where best to focus their improvement efforts. They help researchers advance our understanding of the best way to deliver services. They help those paying for care, including governments, know whether they’re facilitating beneficial services while reaping efficiencies and saving costs.
And for patients, the focus of all these efforts, they help determine whether the care they receive is consistent with the highest quality evidence available.
But it’s important that these performance indicators reflect the nuances of what it means to provide quality primary care, which is heavily influenced by the patient-doctor relationship. Quality primary care depends on mutual trust, an understanding of individual values and preferences, and a shared decision-making approach.
In this context, what is right for one patient is not always necessarily right for another. Williams provided no better care to Smith than to Johnson, even though the outcome of each discussion was different.
These qualitative concepts are difficult to disaggregate and measure. However, we gravitate instead toward more easily definable, readily accessible metrics, such as mammography rates.
But are these “measurables” actually meaningful?
Many of the metrics we use to evaluate primary care are based on dated evidence, no longer align with current guidelines, or have been invalidated altogether.
Examples from Health Quality Ontario’s individualized primary care performance reports include the 30-day hospital readmission rate, physician visit within seven days after hospital discharge, emergency department visits for conditions best managed elsewhere, renal protection for all patients with diabetes, statins for all patients with diabetes and mammography for breast cancer screening.
Even the blood sugar target (A1C) we use to monitor diabetes, widely accepted as the cornerstone of favourable health outcomes, has come under recent, paradigm-shifting criticism.
Aggressively chasing metrics we know are meaningless (or at best, not as meaningful as we once thought) not only risks overburdening our patients and our health system, it also threatens to undervalue dimensions of primary care that, while difficult to measure, are consistently associated with quality care and positive patient outcomes: patient-centredness, the patient-doctor relationship, appropriate access, patient empowerment, social determinants, and continuity of care.
Just because these attributes are not easily extractable from an existing database doesn’t mean we can’t study them.
Some clinics are already integrating tablet-based software with their existing electronic health record (EHR) systems to collect experiential data from patients attending their appointments. EHR appointment schedulers could be used to trigger electronic post-visit surveying as well.
Practitioners can participate in efforts to survey their practices’ organizational attributes, which is important for understanding how care is best delivered. The recent QUALICO PC (Quality & Costs of Primary Care) study is an example.
Social determinants of health – critical to understanding many disparities in technical care and health outcomes – are currently not encoded uniformly (or at all) into EHRs. Perhaps they should be.
Many validated instruments already exist to assist in studying these more complex dimensions of primary care, such as the PDRQ-9 questionnaire, the World Health Organization’s health-related quality of life questionnaire, and the Short-Form 36 Health Survey. Ongoing mixed-methods research should help provide further direction.
It’s imperative we find inexpensive and unobtrusive ways to accurately and comprehensively understand primary care performance. Otherwise, we’ll continue to miss opportunities to convince policymakers to invest where disparities truly exist.
Decisions based on only a partial understanding will result in funds flowing into the wrong areas of primary care, or preferentially into acute care settings where governments can more readily see the results of their investments.
We all want to improve primary care. We know that a strong primary care system makes for a healthier, more productive society. But it seems the current performance indicators are not serving us as well as we’d hoped, and collectively, it’s time to undertake the difficult task of measuring the things that really matter.
The comments section is closed.
The concerns raised in the article certainly need to be addressed in developing indicators for primary health care.
Although we are in early days, Alberta Health is addressing these concerns through a consultative, collaborative process to develop a starting set of indicators for Primary Care Networks (PCNs). The indicators are intended to measure desired outcomes and support improvements the elements that lead to excellence in primary health care delivery, including leadership and patient care. The indicators are at the system level, and align with the Alberta’s Primary Health Care strategic plan and evaluation framework. All PCNs are required to report on the indicator set in their annual reports.
The work is led by the Primary Health Care Measurement & Evaluation Working Group that includes Alberta Health, Primary Care Networks, Alberta Health Services, the Health Quality Council of Alberta, primary care researchers, physicians, and evaluators. Content experts are included on short term task groups that propose refinements to indicators and create toolkits to support the reporting process. As toolkits are implemented, they are accompanied by webinars and other appropriate educational opportunities.
For more information on Alberta’s Primary Health Care Indicator Set, please contact Rick Leischner at rick.leischner@gov.ab.ca .
Rick Leischner
Senior Advisor, Performance Measurement and Evaluation,
Alberta Health
Thank you to Drs Hogg and Premji for bringing forward such an important issue – that is, the need for meaningful measurable data in primary care. For a few years now, AFHTO (Association of Family Health Teams of Ontario) members have participated in the Data To Decisions (D2D) initiative which shows that quality can be measured according to what matters to patients AND that higher quality in primary care is associated with lower costs to the health care system. Nearly two-thirds of AFHTO members are VOLUNTARILY participating in D2D which gives insight into the care of close to 2 million Ontarians. Primary care providers are already well underway in defining primary care meaningful measurement and are active participants in the conversations around using performance data by framing measurement to improving quality of care for our patients. I would highly encourage readers to learn more about this ground-up voluntary performance measurement initiative by visiting the AFHTO website: http://www.afhto.ca/category/measurement/.
Kavita Mehta, CEO, Association of Family Health Teams of Ontario (AFHTO)
Thanks Drs. Premji and Hogg for highlighting the continuing need to exam health system performance measures.
For disclosure purposes: Dr Hogg and I both sit as Directors of The North American Primary Care Research Group. In addition to being the Primary Care Lead at Health Quality Ontario, I chair the Community Clinician Advisory Group of NAPCRG.
Family physicians have found it helpful to receive some feedback at the practice level about the care they provide to their patients. As has been discussed, the existing measures were chosen not only based on availability of data and ability to benchmark but also to provide us a comparison internationally.
While we need to “start where we are,” I think that Drs. Premji and Hogg’s analysis also highlights the need to bring together practicing clinicians, researchers and health policy makers in developing our own complex system measures 2.0. Moreover, we need engaged patients to help us co-design measures that are meaningful to Ontarians.
I have already been speaking with several Ontario MDs after a social media discussion following the updated Primary Care Practice Reports release 10 days ago (https://twitter.com/davidkaplanmd).
Apropos, The Institute for Health Care Improvement has recently released Measure that Matter – Whole System Measures 2.0 (http://www.ihi.org/resources/Pages/IHIWhitePapers/Whole-System-Measures-Compass-for-Health-System-Leaders.aspx)
I look forward to working with everyone to help shape our system-level performance metrics and highlight our practice level quality improvement/practice management opportunities.
David M Kaplan
Associate Professor, Family & Community Medicine, Joint Centre for Bioethics, University of Toronto
Primary Care Clinical Lead, Health Quality Ontario
Thought provoking article. Drs Hogg and Premji, have you worked on encoding social determinants of health into your EHR? Or are you using one of the validated tools yet, with your patients?
I would be interested to know where you are on this path. Regards Wendy Hamilton Ottawa family doc
Thanks, Wendy. Our EHR doesn’t have the capability of encoding many important social determinants of health, including income and education. Our group includes this information in the CPP section of our charts in free-text form, which is useful in the context of individual patient-doctor encounters but unfortunately not useful for data analysis. It would be great to see EHR vendors develop this capability.
Very insightful article and I think speaks clearly in sync with many family physicians. It is unfortunate that with the availability of technology and the ability to collect data ( which is not the same as understanding data- ie just because it is measurable does not mean it has value) we get all caught up in the numbers- and in the process lose that spark we get from having the privilege to connect with our patients in a most unique humanistic and personal manner. Even if we find a way to measure those “soft” qualities, should we and does it not jeopardize that personal relationship?
A very well put reminder that there is much more to the patient-family physician relationship than measured indicators in primary care. In practice it is far more valuable to spend extra time trying to get a COPD patient to receive an influenza vaccine or a pneumonia vaccine than to use that time trying to convince a healthy young person, in the office for an unrelated matter, to receive an influenza vaccination. Family physician time, particularly rural family physician time, is a very scarce resource and needs to be used wisely.
While polio, measles and smallpox vaccinations have demonstrated the value of ‘herd immunity’, we need to treat individuals as individuals and respect their capacity to manage their own health.
Brilliant piece which highlights the cornerstone of primary health care – the essential components of providing patient – centered care!
Thanks Kamila and Bill for your insightful comment. I fully agree with you. As we more and more understand that person centered care is happening in the context of grey one decision making (i.e. how much data reflect how different treatments stand up evidence-based medicine see http://clinicalevidence.bmj.com/x/set/static/cms/efficacy-categorisations.html), we need to focus more on the quality of the processes. Therefore, I would like to add the following additional indicators of personalized patient centered decision support: decisional conflict (i.e. how much the patient feels he/she is comfortable with the decision because of being informed by best evidence, knowing what matters most to them and being supported), perception of patients of being engaged meaningfully in the decision making process, decisional regret, informed decision, etc. In other words, PROMs that captured the true nature of the patient centered care processes can be easily integrated in EMRs. More measures can be found at https://decisionaid.ohri.ca/ . Thanks again and best regards. France
Thank you so much for this thought provoking article. I think that the question of which indicators are best is really important, and we need to find a balance between patient experience metrics and patient outcome metrics. I am intrigued by some of the work in the US to look at measurement in primary care and family medicine through the lens of the “patient medical home” and the core features of effective primary care delivery. These metrics would touch on the degree to which care is patient centred, comprehensive, and coordinated and the degree to which there is continuity of care with the MRP (or the “teamlet” – MD/RN). I have also been interested in Scott Giesel’s thinking (Dartmouth) about the things that matter to patients which he considers as the “3 C’s”. These are: capability, comfort and calm. Using metrics that drive at these, we would ask questions (both experience and outcome) about the degree to which we have improved patients’ capability (or function), improved their comfort (minimized their symptoms) or made their treatment more “calm” (more coordinated, simpler to manage).
Having said all that, I agree with Darren’s comment, that it is important to ensure that we have a process through which we evolve metrics so that they are dynamic and responsive to evolving evidence. I also think that we need to rethink to what extent we measure and report, and to what extent we measure and incent.
Thank you again for a thought provoking article.
Einstein said it best “Not everything that can be counted counts, and not everything that counts can be counted”. We have consistently measured the convenient in health care, not the important. It is difficult to measure important information given how poor our databases are, and how much of what we measure is designed deliberately for political purposes, or just to reduce costs without regard for real human outcomes. Thanks for pointing this out and keep up the fight for meaningful data in primary care.
I have nothing to add to the discussion but I feel I have to say something: Very insightful article.
Bill (and Kamila),
While I totally agree that “Not everything that counts can be counted, and not everything that can be counted counts”, I also think that when it comes to Primary Care Measurement the best place to start is where we are. The most important concept to note is that the indicators chosen were never meant to be static. The set of measures we use should adapt to new knowledge and be modernized over time. As data starts to be collected this becomes possible and then primary care measurement tools will be better positioned in this complex adaptive health care system. As we were both involved in the selection of the first 220+ PCPM indicators with HQO, you will remember that we started with 1500+ possibilities and these were narrowed down to the set we see based on availability of data, ability to benchmark and compare both provincially and internationally, validity and veracity, ability to show improvement and ultimately best evidence available at the time. These were the start not the final end point of this journey.
Measuring the very important relationships we hold dear in primary care, as well as social determinants, patient empowerment and appropriateness are all evolutions of this first tranche of work, in my opinion. Such measures may be “softer” in terms of hard data, and less available at the system level, but because this wisdom is more difficult to obtain it is much more difficult to engage our peers in understanding its value. Thanks for raising this important point.
Looking forward to the research and insight that comes from your work in this domain!
I think the work that HQO does has the potential to be of use. It has the potential to add knowledge as we figure out how to prove to those outside the doctor-patient relationship that a specific doctor-patient relationship has value.
The problem is that HQO’s reports will be used by lots of people — and some of them are neither trained in research nor have an appreciation of how research and medicine are evolving fields. The nuances will not be appreciated. The idea that same-day/ next-day appointments serve as a measure of accessibility is one: I almost never fill up my same-day/ next day appointments. However, many of my patients go to WICs or ER for reasons that have nothing to do with me and everything to do with their own preferences and schedules. Yet according to the interpretation offered by HQO, MOH and AG, that makes me an inaccessible doctor. As it turns out, many of my patients have my cell phone number for after-hours telephone consults. Yet even that level of access has made no dents in the number of patients who go to WIC or ER. So at the end of the day, by same-day/next-day metrics, I am inaccessible. And if Bill 41 passes, I will be punished for inaccessibility. Short of dragging those patients into my clinic or sitting in my office 24h a day, there is very little else I can do to make myself more accessible.
The MOH has used HQO stats without context. Today, the AG report did the same. And the public relies on these sources to translate research into layman’s terms. So right now, the public is getting the wrong impression from over-simplification of a very complex scenario.
I respectfully suggest that HQO and other organizations that chase metrics be cautious of how they advertise metrics in the public sphere. I would also suggest that when the government or the AG gets the metrics wrong, HQO publicly disagree with and correct them. The law of unintended consequences holds particularly true here – and those consequences could be devastating to docs such as I.
Are you saying that the HQO primary care metrics are “based on dated evidence, no longer align with current guidelines, or have been invalidated altogether”? That is the way your article reads so I would like to see the evidence for your statement.
Thanks for your comment, Tom. We’ve included links within the article to evidence questioning some of the metrics currently used by HQO. Not all of the metrics currently reported by HQO are evidence-basesd for all patients. For example, according to the CDA guidelines, not all patients with diabetes benefit from being on a statin or renal protective agent, but as of the writing of this article, the reported metric doesn’t tease out these various subpopulations. Other reported indicators are less black-and-white than the metric alone suggests, such as mammography for breast cancer screening (declining mammography may be a very reasonable decision for an individual patient who values the potential harms of overdiagnosis and false positives over the potential benefits of early cancer detection). As we see increasing interest in tying funding to performance, it’s important we recognize the limitations of some of the metrics we currently use and continue refining how we define and measure quality within primary care.
I would add that no evidence based guidelines exist in situations of multiple comorbidities.
The article states many, not all of the metrics, and then goes on to give examples with links to studies that question their validity as appropriate metrics.