Let’s not confuse measurable with meaningful in primary care


Leave a Comment

Enter the debate: reply to an existing comment
18 comments

  1. Tom Closson

    Are you saying that the HQO primary care metrics are “based on dated evidence, no longer align with current guidelines, or have been invalidated altogether”? That is the way your article reads so I would like to see the evidence for your statement.

    • Kamila Premji

      Thanks for your comment, Tom. We’ve included links within the article to evidence questioning some of the metrics currently used by HQO. Not all of the metrics currently reported by HQO are evidence-basesd for all patients. For example, according to the CDA guidelines, not all patients with diabetes benefit from being on a statin or renal protective agent, but as of the writing of this article, the reported metric doesn’t tease out these various subpopulations. Other reported indicators are less black-and-white than the metric alone suggests, such as mammography for breast cancer screening (declining mammography may be a very reasonable decision for an individual patient who values the potential harms of overdiagnosis and false positives over the potential benefits of early cancer detection). As we see increasing interest in tying funding to performance, it’s important we recognize the limitations of some of the metrics we currently use and continue refining how we define and measure quality within primary care.

      • Dr. Bob Bernstein

        I would add that no evidence based guidelines exist in situations of multiple comorbidities.

    • Jeffrey Mascarenhas

      The article states many, not all of the metrics, and then goes on to give examples with links to studies that question their validity as appropriate metrics.

  2. Darren Larsen

    Bill (and Kamila),

    While I totally agree that “Not everything that counts can be counted, and not everything that can be counted counts”, I also think that when it comes to Primary Care Measurement the best place to start is where we are. The most important concept to note is that the indicators chosen were never meant to be static. The set of measures we use should adapt to new knowledge and be modernized over time. As data starts to be collected this becomes possible and then primary care measurement tools will be better positioned in this complex adaptive health care system. As we were both involved in the selection of the first 220+ PCPM indicators with HQO, you will remember that we started with 1500+ possibilities and these were narrowed down to the set we see based on availability of data, ability to benchmark and compare both provincially and internationally, validity and veracity, ability to show improvement and ultimately best evidence available at the time. These were the start not the final end point of this journey.

    Measuring the very important relationships we hold dear in primary care, as well as social determinants, patient empowerment and appropriateness are all evolutions of this first tranche of work, in my opinion. Such measures may be “softer” in terms of hard data, and less available at the system level, but because this wisdom is more difficult to obtain it is much more difficult to engage our peers in understanding its value. Thanks for raising this important point.

    Looking forward to the research and insight that comes from your work in this domain!

    • Nadia Alam

      I think the work that HQO does has the potential to be of use. It has the potential to add knowledge as we figure out how to prove to those outside the doctor-patient relationship that a specific doctor-patient relationship has value.

      The problem is that HQO’s reports will be used by lots of people — and some of them are neither trained in research nor have an appreciation of how research and medicine are evolving fields. The nuances will not be appreciated. The idea that same-day/ next-day appointments serve as a measure of accessibility is one: I almost never fill up my same-day/ next day appointments. However, many of my patients go to WICs or ER for reasons that have nothing to do with me and everything to do with their own preferences and schedules. Yet according to the interpretation offered by HQO, MOH and AG, that makes me an inaccessible doctor. As it turns out, many of my patients have my cell phone number for after-hours telephone consults. Yet even that level of access has made no dents in the number of patients who go to WIC or ER. So at the end of the day, by same-day/next-day metrics, I am inaccessible. And if Bill 41 passes, I will be punished for inaccessibility. Short of dragging those patients into my clinic or sitting in my office 24h a day, there is very little else I can do to make myself more accessible.

      The MOH has used HQO stats without context. Today, the AG report did the same. And the public relies on these sources to translate research into layman’s terms. So right now, the public is getting the wrong impression from over-simplification of a very complex scenario.

      I respectfully suggest that HQO and other organizations that chase metrics be cautious of how they advertise metrics in the public sphere. I would also suggest that when the government or the AG gets the metrics wrong, HQO publicly disagree with and correct them. The law of unintended consequences holds particularly true here – and those consequences could be devastating to docs such as I.

  3. Gerald Goldlist

    I have nothing to add to the discussion but I feel I have to say something: Very insightful article.

  4. Dr. Bob Bernstein

    Einstein said it best “Not everything that can be counted counts, and not everything that counts can be counted”. We have consistently measured the convenient in health care, not the important. It is difficult to measure important information given how poor our databases are, and how much of what we measure is designed deliberately for political purposes, or just to reduce costs without regard for real human outcomes. Thanks for pointing this out and keep up the fight for meaningful data in primary care.

  5. Sarah Newbery

    Thank you so much for this thought provoking article. I think that the question of which indicators are best is really important, and we need to find a balance between patient experience metrics and patient outcome metrics. I am intrigued by some of the work in the US to look at measurement in primary care and family medicine through the lens of the “patient medical home” and the core features of effective primary care delivery. These metrics would touch on the degree to which care is patient centred, comprehensive, and coordinated and the degree to which there is continuity of care with the MRP (or the “teamlet” – MD/RN). I have also been interested in Scott Giesel’s thinking (Dartmouth) about the things that matter to patients which he considers as the “3 C’s”. These are: capability, comfort and calm. Using metrics that drive at these, we would ask questions (both experience and outcome) about the degree to which we have improved patients’ capability (or function), improved their comfort (minimized their symptoms) or made their treatment more “calm” (more coordinated, simpler to manage).

    Having said all that, I agree with Darren’s comment, that it is important to ensure that we have a process through which we evolve metrics so that they are dynamic and responsive to evolving evidence. I also think that we need to rethink to what extent we measure and report, and to what extent we measure and incent.

    Thank you again for a thought provoking article.

  6. France Légaré

    Thanks Kamila and Bill for your insightful comment. I fully agree with you. As we more and more understand that person centered care is happening in the context of grey one decision making (i.e. how much data reflect how different treatments stand up evidence-based medicine see http://clinicalevidence.bmj.com/x/set/static/cms/efficacy-categorisations.html), we need to focus more on the quality of the processes. Therefore, I would like to add the following additional indicators of personalized patient centered decision support: decisional conflict (i.e. how much the patient feels he/she is comfortable with the decision because of being informed by best evidence, knowing what matters most to them and being supported), perception of patients of being engaged meaningfully in the decision making process, decisional regret, informed decision, etc. In other words, PROMs that captured the true nature of the patient centered care processes can be easily integrated in EMRs. More measures can be found at https://decisionaid.ohri.ca/ . Thanks again and best regards. France

  7. Judy Belle Brown

    Brilliant piece which highlights the cornerstone of primary health care – the essential components of providing patient – centered care!

  8. Paul Coolican

    A very well put reminder that there is much more to the patient-family physician relationship than measured indicators in primary care. In practice it is far more valuable to spend extra time trying to get a COPD patient to receive an influenza vaccine or a pneumonia vaccine than to use that time trying to convince a healthy young person, in the office for an unrelated matter, to receive an influenza vaccination. Family physician time, particularly rural family physician time, is a very scarce resource and needs to be used wisely.

    While polio, measles and smallpox vaccinations have demonstrated the value of ‘herd immunity’, we need to treat individuals as individuals and respect their capacity to manage their own health.

  9. Salina Jaffer

    Very insightful article and I think speaks clearly in sync with many family physicians. It is unfortunate that with the availability of technology and the ability to collect data ( which is not the same as understanding data- ie just because it is measurable does not mean it has value) we get all caught up in the numbers- and in the process lose that spark we get from having the privilege to connect with our patients in a most unique humanistic and personal manner. Even if we find a way to measure those “soft” qualities, should we and does it not jeopardize that personal relationship?

  10. Wendy Hamilton

    Thought provoking article. Drs Hogg and Premji, have you worked on encoding social determinants of health into your EHR? Or are you using one of the validated tools yet, with your patients?
    I would be interested to know where you are on this path. Regards Wendy Hamilton Ottawa family doc

    • Kamila Premji

      Thanks, Wendy. Our EHR doesn’t have the capability of encoding many important social determinants of health, including income and education. Our group includes this information in the CPP section of our charts in free-text form, which is useful in the context of individual patient-doctor encounters but unfortunately not useful for data analysis. It would be great to see EHR vendors develop this capability.

  11. David Kaplan

    Thanks Drs. Premji and Hogg for highlighting the continuing need to exam health system performance measures.

    For disclosure purposes: Dr Hogg and I both sit as Directors of The North American Primary Care Research Group. In addition to being the Primary Care Lead at Health Quality Ontario, I chair the Community Clinician Advisory Group of NAPCRG.

    Family physicians have found it helpful to receive some feedback at the practice level about the care they provide to their patients. As has been discussed, the existing measures were chosen not only based on availability of data and ability to benchmark but also to provide us a comparison internationally.

    While we need to “start where we are,” I think that Drs. Premji and Hogg’s analysis also highlights the need to bring together practicing clinicians, researchers and health policy makers in developing our own complex system measures 2.0. Moreover, we need engaged patients to help us co-design measures that are meaningful to Ontarians.

    I have already been speaking with several Ontario MDs after a social media discussion following the updated Primary Care Practice Reports release 10 days ago (https://twitter.com/davidkaplanmd).

    Apropos, The Institute for Health Care Improvement has recently released Measure that Matter – Whole System Measures 2.0 (http://www.ihi.org/resources/Pages/IHIWhitePapers/Whole-System-Measures-Compass-for-Health-System-Leaders.aspx)

    I look forward to working with everyone to help shape our system-level performance metrics and highlight our practice level quality improvement/practice management opportunities.

    David M Kaplan
    Associate Professor, Family & Community Medicine, Joint Centre for Bioethics, University of Toronto
    Primary Care Clinical Lead, Health Quality Ontario

  12. Kavita Mehta

    Thank you to Drs Hogg and Premji for bringing forward such an important issue – that is, the need for meaningful measurable data in primary care. For a few years now, AFHTO (Association of Family Health Teams of Ontario) members have participated in the Data To Decisions (D2D) initiative which shows that quality can be measured according to what matters to patients AND that higher quality in primary care is associated with lower costs to the health care system. Nearly two-thirds of AFHTO members are VOLUNTARILY participating in D2D which gives insight into the care of close to 2 million Ontarians. Primary care providers are already well underway in defining primary care meaningful measurement and are active participants in the conversations around using performance data by framing measurement to improving quality of care for our patients. I would highly encourage readers to learn more about this ground-up voluntary performance measurement initiative by visiting the AFHTO website: http://www.afhto.ca/category/measurement/.

    Kavita Mehta, CEO, Association of Family Health Teams of Ontario (AFHTO)

  13. Rick Leischner

    The concerns raised in the article certainly need to be addressed in developing indicators for primary health care.
    Although we are in early days, Alberta Health is addressing these concerns through a consultative, collaborative process to develop a starting set of indicators for Primary Care Networks (PCNs). The indicators are intended to measure desired outcomes and support improvements the elements that lead to excellence in primary health care delivery, including leadership and patient care. The indicators are at the system level, and align with the Alberta’s Primary Health Care strategic plan and evaluation framework. All PCNs are required to report on the indicator set in their annual reports.
    The work is led by the Primary Health Care Measurement & Evaluation Working Group that includes Alberta Health, Primary Care Networks, Alberta Health Services, the Health Quality Council of Alberta, primary care researchers, physicians, and evaluators. Content experts are included on short term task groups that propose refinements to indicators and create toolkits to support the reporting process. As toolkits are implemented, they are accompanied by webinars and other appropriate educational opportunities.
    For more information on Alberta’s Primary Health Care Indicator Set, please contact Rick Leischner at rick.leischner@gov.ab.ca .

    Rick Leischner
    Senior Advisor, Performance Measurement and Evaluation,
    Alberta Health

Submit a comment