Goodhart's Law and Gaming the Metrics

As I noted last time, Goodhart's Law is the observation by Charles Goodhart that when we express an organizational goal in terms of a metric, the metric loses its value as a measure of anything. [Goodhart 1975] To be clear, metrics are quantifiable measures of attributes of business processes. Goodhart's observation, in essence, is that any such measurement supposedly indicates the difference between the current value and the goal value, but when the goal value is widely known, the current value is subject to distortions that make the current value unreliable. It's likely that several mechanisms account for this phenomenon, and some have been studied better than others.

I proposed last time that one factor that contributes to the loss of reliability of metrics is our tendency to believe that we can "measure" human behavior. Because something as complex as human behavior is bound to include abstractions, the exercise of "measurement" is likely less objective than, say, weighing a sack of potatoes. Consequently, our "metrics" are subject to distortions, which eventually erode their usefulness.

Gaming the metrics

Another source of distortions is a human behavior that goes by various names, including "gaming the metrics" and "juking the stats." It works like this.

When the goal value of a metric is widely known, members of the population whose behavior is supposedly represented by the metric in question begin adjusting their behavior so as to achieve the goal value of the metric. What is problematic about these adjustments is that organizations have difficulty enforcing limits on behavioral adjustments. Some adjustments are acceptable and welcome; others are intended to — and do — drive the metric value toward the goal, but not in a way that achieves the desired value.

Here's an example.

In several of these domains, we see how the people who are pressed for improved metrics respond to pressure. The police, for example, engage in a practice they call "juking the stats." To juke the stats, you adjust your behavior to produce better values of the metric (in this case crime data), without necessarily improving what that metric is supposed to "measure" (in this case crime). [Revankar 2016] .

Patterns of gaming metrics

The When the goal value of a metric is
widely known, people whose behavior is
represented by the metric adjust their
behavior to achieve the goal valueprocess of gaming metrics can occur wherever we use metrics. One well-studied area is misconduct in academic research. [Biagioli 2020] That work suggests several patterns of gaming metrics.

Counterfeit the data

Whatever is the process for collecting data for the metric, it probably relies on manual or automated data collection. In this pattern, someone intervenes in the data production process, providing false data that makes the metric report better results than actual data would. Audit trails can deter this activity, but wily counterfeiters can always hide their tracks.

Misstate categories

Some metrics consist of counts of issues binned according to a set of binning definitions. In this pattern, the populations in some of the categories are misstated. For example, if there are 22 issues in category "Severe" the report might state that there are only 15. One way to accomplish this is to make a snapshot of the category populations at an advantageous time. And, of course, simple falsifying is also effective.

Redefine categories

Another way to improve the populations in categories is to redefine the categories. Subtle changes in category definitions can appear to be intended to create a "more accurate representation of our status," when what they actually do is conceal the number of issues that are in the most problematic categories.

Assign highest priority to the least difficult issues

By consistently avoiding investing effort and resources in addressing difficult issues, the organization can create a long list of issues addressed. They might have little positive effect on the health of the organization, but their numbers convey a very different impression — one of substantial progress.

Restart the clock

To game metrics that measure elapsed time between events, restarting the clock is a handy tactic. For example, consider a metric that measures the time it takes to resolve a support ticket at a help desk. One way to game this metric is to declare a ticket resolved and close it before it's actually resolved. When the ticket's submitter objects, the help desk opens a "replacement" ticket, thereby restarting the clock. Another way to achieve a similar result is to declare a ticket unclear or ambiguous, close it, and send the submitter a document describing how to submit a ticket. There are probably dozens of ways to restart the clock.

Last words

When all else fails, there is one last option for those intent on misrepresenting the state of the organization: introduce new metrics. If the metrics in use are likely to produce an uncomfortable representation of the process in question, perhaps a different set of metrics might make a more favorable impression. Naturally, the new metrics must be accompanied by a justification of the claim that they produce a more accurate view of the process status. The justification can be the most difficult piece of the exercise.   First issue in this series   Top    Next Issue

Are your projects always (or almost always) late and over budget? Are your project teams plagued by turnover, burnout, and high defect rates? Turn your culture around. Read 52 Tips for Leaders of Project-Oriented Organizations, filled with tips and techniques for organizational leaders. Order Now!

Footnotes

Your comments are welcome

About Point Lookout

Thank you for reading this article. I hope you enjoyed it and found it useful, and that you'll consider recommending it to a friend.

This article in its entirety was written by a human being. No machine intelligence was involved in any way.

Point Lookout is a free weekly email newsletter. Browse the archive of past issues. Subscribe for free.

Support Point Lookout by joining the Friends of Point Lookout, as an individual or as an organization.

Do you face a complex interpersonal situation? Send it in, anonymously if you like, and I'll give you my two cents.

Related articles

More articles on Workplace Politics:

Ten Reasons Why You Don't Always Get What You Measure: III

The phrase "You get what you measure," has acquired the status of "truism." Yet many measurement-based initiatives have produced disappointing results. Here's Part III of an examination of the idea — a look at management's role in these surprises.

The Perils of Political Praise

Political Praise is any public statement, praising (most often) an individual, and including a characterization of the individual or the individual's deeds, and which spins or distorts in such a way that it advances the praiser's own political agenda, possibly at the expense of the one praised.

Suppressing Dissent: I

In some groups, disagreeing with the majority, or disagreeing with the Leader, can be a personally expensive act. Here is Part I of a set of tactics used by Leaders who choose not to tolerate dissent.

Implicit Interrogations

Investigations at work can begin with implicit interrogations — implicit because they're unannounced and unacknowledged. The goal is to determine what people did or knew without revealing that an investigation is underway. When asked, those conducting these interrogations often deny they're doing it. What's the nature of implicit interrogations?

They Don't Reply to My Email

Ever have the experience of sending an email message to someone, asking for information or approval or whatever, and then waiting for a response that comes only too late? Maybe your correspondent is an evil loser, but maybe not. Maybe the problem is in your message.

See also Workplace Politics and Workplace Politics for more related articles.

Forthcoming issues of Point Lookout

Coming March 12: Embedded Technology Groups and the Dunning-Kruger Effect

Groups of technical specialists in fields that differ markedly from the main business of the enterprise that hosts them must sometimes deal with wrong-headed decisions made by people who think they know more about the technology than they actually do. Available here and by RSS on March 12.

And on March 19: On Lying by Omission

Of the many devious strategies of workplace politics, deception is among the most commonly used. And perhaps the most commonly used tactic of deception is lying. Since getting caught in a lie can be costly, people try to lie without lying. Available here and by RSS on March 19.

Coaching services

I offer email and telephone coaching at both corporate and individual rates. Contact Rick for details at rbrenjTnUayrCbSnnEcYfner@ChacdcYpBKAaMJgMalFXoCanyon.com or (650) 787-6475, or toll-free in the continental US at (866) 378-5470.

Get the ebook!

Past issues of Point Lookout are available in six ebooks:

• Get 2001-2 in Geese Don't Land on Twigs (PDF, )
• Get 2003-4 in Why Dogs Wag (PDF, )
• Get 2005-6 in Loopy Things We Do (PDF, )
• Get 2007-8 in Things We Believe That Maybe Aren't So True (PDF, )
• Get 2009-10 in The Questions Not Asked (PDF, )
• Get all of the first twelve years (2001-2012) in The Collected Issues of Point Lookout (PDF, )

Are you a writer, editor or publisher on deadline? Are you looking for an article that will get people talking and get compliments flying your way? You can have 500-1000 words in your inbox in one hour. License any article from this Web site. More info