by Lt. Col. Liam Toth, Deputy CMIO

2020 Clinical Informatics Fellowship Graduate

"When a measure becomes a target, it ceases to be a good measure." 

I was slightly surprised by the amount of feedback I received last week from the discussion on metrics.  Unfortunately, I also realized that I brought up metrics in the absolute wrong order.  I broke one of my cardinal rules.  You should always start with “why”.  And instead I started with “what”.  So even though it is out of order, I’m going to go back and hit they purpose of metrics this week before moving on towards the “how” of metrics in MHS GENESIS.  

I think there are two main groups that are probably reading this.  First are the younger physicians that have an interest in improving the process but may not feel empowered to do so.  I hope this is an introduction to how to talk about metrics in a way that people can hear. I hope this is a tool to carry out a good conversation and challenge the status quo.  The second are older physicians that are in leadership positions.  I hope this is an opportunity to reconsider every metric at every meeting.  There are more metrics than ever, but I rarely see people ask enough “whys” to get to the real reasons for many of those metrics.

Why are and why are not metrics?

Effective metrics are NOT a mechanism for comparing people.  They are NOT a way to measure “productivity”.  They don’t even represent the organizational goals.  Rather, the effective use of metrics ARE a way to determine if we are meeting our strategic goals of better health, better care, better value, and increased readiness.  They are a way to understand and measure if improvements are being made.  But most importantly, effective metrics ARE a mechanism for motivating individuals and teams to better meet our strategic goals.  But what makes a metric effective?

A bad metric will ONLY reinforce the behaviors that will meet that metric – and nothing more.   Bad metrics are the quintessential definition of Goodahart’s Law, which states that once people know what is being measured, they will adapt their behavior accordingly - and the results are unlikely to be what was originally intended.   That is because bad metrics only focus on reinforcing behaviors that meet a metric instead of recognizing that metrics are the best way to improve an organization and motivate your people.

Metrics = Motivation?

Entire books have been written about motivation, but for metrics it can probably be distilled to an understanding of the two major aspects of motivation – internal vs external.  External motivation is generally stimulated by either the desire to get a reward (primarily money) or a desire not to get a punishment.  External motivation can be highly effective for tasks which require lower cognitive focus.  However, when external motivation is applied to tasks which require high cognitive focus, the effects are disastrous.  In these instances, when cognition is required, external motivations deteriorate performance and outcomes.  Even worse, when applied to tasks which have inherently high internal motivation, it actually eliminates the internal motivation further eroding outcomes.

This generally happens in two ways.  First, external motivators “crowd-out” the intrinsic motivation.  Previously interesting and rewarding tasks are undermined with a subsequent increase in selfish behaviors and decreased teamwork.  Second, it takes a previous moral imperative and transforms it into a transaction regulated by price or time.  Part of the very reason that many physicians are referring to burnout as a “moral injury” is because the application of external motivators directly conflict with their internal motivation for why they went into medicine to begin with. 

Metrics as Motivation in Medicine

 This concept is highly applicable to the determination of appropriate metrics for the medical community. Medicine has a high cognitive requirement and we should avoid external motivations at all costs.  DMHRSi is the perfect example of a bad metric.  Several studies have demonstrated that when you start measuring time for high cognitive tasks, people invariably decrease the amount of time that they work on those tasks.  In other words, we are purposefully reducing peoples moral desire to work by requiring them to account for their time.  “Patients seen a week” is another terrible metric.  First, by the same mechanism, it will actually cause people that previously saw more people a week out of moral obligation to reduce the number of patients they see.  Second, it degrades the relationship between a patient and physician to an impersonal transaction that meets a number requirement instead of seeing their value as a person.  It gets to the very root of the problem of medicine as a business.  It is a business, but we need to treat it differently if we want to keep people motivated.

How Should Metrics be Used?

Although often more difficult to capture, metrics should instead reinforce the social and professional norms and thereby increase internal motivation.  For instance, availability of the next appointment within 72 hours has demonstratable improvements in patient care.   By creating a metric with improves the care and outcome of the patient, we can reinforce internal motivations of a highly intelligent workforce.  Further implementation of this concept would require not “punishing” those that fail to meet that standard, but rather assisting them to understand both why it is important and how the organization can better support them to meet their patients’ needs to improve outcomes.

Unfortunately, external metrics have become the norm.  They are easy to capture.  They are usually easy to explain.  They are easy to compensate.  They are also extremely damaging to the long-term success of an organization and undermine the internal motivators for which people sought both medicine and military service.  Hopefully we have an opportunity to reconsider the “why” of metrics and realign them with our strategic goals and how they can reinforce instead of undermine the internal motivation of our dedicated, determined, yet downtrodden staff. 

GENESIS 101:  I’m going to give a shout out this week to the Site Integrators.  The site integrators are the MTFs main information avenue for all things GENESIS and are a wealth of knowledge and organization.  They are the local processing for any issue tracking and improvements.  They are the communication experts for getting everything out.