Point Lookout: a free weekly publication of Chaco Canyon Consulting
Volume 23, Issue 10;   March 8, 2023: Goodhart's Law and Gaming the Metrics

Goodhart's Law and Gaming the Metrics

by

Goodhart's Law is an observation about managing by metrics. When we make known the metrics' goals, we risk collapse of the metrics, in part because people try to "game" the metrics by shading or manufacturing the data to produce the goal result.
A Wurlitzer "One More Time" jukebox, circa 1950

A Wurlitzer "One More Time" jukebox, circa 1950. The term "juking the stats" has an interesting history. As Joseph T. Shipley writes, "In the mountains of southern United States, many Elizabethan words, that have died out in England, are preserved. Thus jouk, to dodge, to move quickly, was applied to the places where liquor was sold, in prohibition times, hence, any cheap drinking place. When the automatic phonograph swept to popularity in such shops, it came to be called a juke box." [Shipley 1967] Image by Victoria_Watercolor, courtesy .

As I noted last time, Goodhart's Law is the observation by Charles Goodhart that when we express an organizational goal in terms of a metric, the metric loses its value as a measure of anything. [Goodhart 1975] To be clear, metrics are quantifiable measures of attributes of business processes. Goodhart's observation, in essence, is that any such measurement supposedly indicates the difference between the current value and the goal value, but when the goal value is widely known, the current value is subject to distortions that make the current value unreliable. It's likely that several mechanisms account for this phenomenon, and some have been studied better than others.

I proposed last time that one factor that contributes to the loss of reliability of metrics is our tendency to believe that we can "measure" human behavior. Because something as complex as human behavior is bound to include abstractions, the exercise of "measurement" is likely less objective than, say, weighing a sack of potatoes. Consequently, our "metrics" are subject to distortions, which eventually erode their usefulness.

Gaming the metrics

Another source of distortions is a human behavior that goes by various names, including "gaming the metrics" and "juking the stats." It works like this.

When the goal value of a metric is widely known, members of the population whose behavior is supposedly represented by the metric in question begin adjusting their behavior so as to achieve the goal value of the metric. What is problematic about these adjustments is that organizations have difficulty enforcing limits on behavioral adjustments. Some adjustments are acceptable and welcome; others are intended to — and do — drive the metric value toward the goal, but not in a way that achieves the desired value.

Here's an example.

The Peabody-Award-winning HBO series, "The Wire," explores how several segments of society and their institutions deal with crime and its consequences in the city of Baltimore, Maryland, USA, in the early 2000s. Elected officials press the police for lower crime statistics; the media press the elected officials for the same; the state presses the board of Education and teachers for higher test scores; the harbor unions press elected officials for better shipping facilities and more traffic; dealers in illegal drugs press their suppliers for more reliable "product" deliveries; and so on.

In several of these domains, we see how the people who are pressed for improved metrics respond to pressure. The police, for example, engage in a practice they call "juking the stats." To juke the stats, you adjust your behavior to produce better values of the metric (in this case crime data), without necessarily improving what that metric is supposed to "measure" (in this case crime). [Revankar 2016] .

Patterns of gaming metrics

The When the goal value of a metric is
widely known, people whose behavior is
represented by the metric adjust their
behavior to achieve the goal value
process of gaming metrics can occur wherever we use metrics. One well-studied area is misconduct in academic research. [Biagioli 2020] That work suggests several patterns of gaming metrics.

Counterfeit the data
Whatever is the process for collecting data for the metric, it probably relies on manual or automated data collection. In this pattern, someone intervenes in the data production process, providing false data that makes the metric report better results than actual data would. Audit trails can deter this activity, but wily counterfeiters can always hide their tracks.
Misstate categories
Some metrics consist of counts of issues binned according to a set of binning definitions. In this pattern, the populations in some of the categories are misstated. For example, if there are 22 issues in category "Severe" the report might state that there are only 15. One way to accomplish this is to make a snapshot of the category populations at an advantageous time. And, of course, simple falsifying is also effective.
Redefine categories
Another way to improve the populations in categories is to redefine the categories. Subtle changes in category definitions can appear to be intended to create a "more accurate representation of our status," when what they actually do is conceal the number of issues that are in the most problematic categories.
Assign highest priority to the least difficult issues
By consistently avoiding investing effort and resources in addressing difficult issues, the organization can create a long list of issues addressed. They might have little positive effect on the health of the organization, but their numbers convey a very different impression — one of substantial progress.
Restart the clock
To game metrics that measure elapsed time between events, restarting the clock is a handy tactic. For example, consider a metric that measures the time it takes to resolve a support ticket at a help desk. One way to game this metric is to declare a ticket resolved and close it before it's actually resolved. When the ticket's submitter objects, the help desk opens a "replacement" ticket, thereby restarting the clock. Another way to achieve a similar result is to declare a ticket unclear or ambiguous, close it, and send the submitter a document describing how to submit a ticket. There are probably dozens of ways to restart the clock.

Last words

When all else fails, there is one last option for those intent on misrepresenting the state of the organization: introduce new metrics. If the metrics in use are likely to produce an uncomfortable representation of the process in question, perhaps a different set of metrics might make a more favorable impression. Naturally, the new metrics must be accompanied by a justification of the claim that they produce a more accurate view of the process status. The justification can be the most difficult piece of the exercise. First in this series  Go to top Top  Next issue: Fear/Anxiety Bias: I  Next Issue

52 Tips for Leaders of Project-Oriented OrganizationsAre your projects always (or almost always) late and over budget? Are your project teams plagued by turnover, burnout, and high defect rates? Turn your culture around. Read 52 Tips for Leaders of Project-Oriented Organizations, filled with tips and techniques for organizational leaders. Order Now!

Footnotes

Comprehensive list of all citations from all editions of Point Lookout
[Shipley 1967]
Joseph Twadell Shipley. Dictionary of Word Origins. New York: Philosophical library, 1945. 1967 Edition. Order from Amazon.com. Back
[Goodhart 1975]
Charles Goodhart. "Problems of Monetary Management: The U.K. Experience", in Courakis, Anthony S. (ed.), Inflation, Depression, and Economic Policy in the West. Totowa, New Jersey: Barnes and Noble Books (1981), p. 116. Order from Amazon.com. Back
[Revankar 2016]
Roshan Revankar. "Juking the Stats: Engineering lessons from HBO's The Wire," Medium.com, January 20, 2016. Available here. Retrieved 17 February 2023. Back
[Biagioli 2020]
Mario Biagioli and Alexandra Lippman, eds. Gaming the metrics: Misconduct and manipulation in academic research. MIT Press, 2020. Available here. Order from Amazon.com. Back

Your comments are welcome

Would you like to see your comments posted here? rbrenIyeJIiAfnGdKlUXrner@ChacsxirZwZlENmHUNHioCanyon.comSend me your comments by email, or by Web form.

About Point Lookout

This article in its entirety was written by a 
          human being. No machine intelligence was involved in any way.Thank you for reading this article. I hope you enjoyed it and found it useful, and that you'll consider recommending it to a friend.

This article in its entirety was written by a human being. No machine intelligence was involved in any way.

Point Lookout is a free weekly email newsletter. Browse the archive of past issues. Subscribe for free.

Support Point Lookout by joining the Friends of Point Lookout, as an individual or as an organization.

Do you face a complex interpersonal situation? Send it in, anonymously if you like, and I'll give you my two cents.

Related articles

More articles on Personal, Team, and Organizational Effectiveness:

Bottleneck road signWhen Your Boss Is a Micromanager
If your boss is a micromanager, your life can be a seemingly endless misery of humiliation and frustration. Changing your boss is one possible solution, but it's unlikely to succeed. What you can do is change the way you experience the micromanagement.
Governor Scott Walker of WisconsinIndicators of Lock-In: I
In group decision making, lock-in occurs when the group persists in adhering to its chosen course even though superior alternatives exist. Lock-in can be disastrous for problem-solving organizations. What are some common indicators of lock-in?
Masonry archesPerformance Issues for Nonsupervisors
If, in part of your job, you're a nonsupervisory leader, such as a team lead or a project manager, you face special challenges when dealing with performance issues. Here are some guidelines for nonsupervisors.
A wall of stoneRed Flags: II
When we find clear evidence of serious problems in a project or other collaboration, we sometimes realize that we had overlooked several "red flags" that had foretold trouble. In this Part II of our review of red flags, we consider communication patterns that are useful indicators of future problems.
A home officeAnticipating Absence: Quarantine and Isolation
When the pandemic compels some knowledge workers to quarantine or isolate, we tend to treat them as if they were totally unavailable. But if they're willing and able to work, even part-time, they might be able to continue to contribute. To make this happen, work out conditions in advance.

See also Personal, Team, and Organizational Effectiveness and Problem Solving and Creativity for more related articles.

Forthcoming issues of Point Lookout

What most of us think of when we think of checklistsComing February 28: Checklists: Conventional or Auditable
Checklists help us remember the steps of complex procedures, and the order in which we must execute them. The simplest form is the conventional checklist. But when we need a record of what we've done, we need an auditable checklist. Available here and by RSS on February 28.
Adolf Hitler greets Neville Chamberlain at the beginning of the Bad Godesberg meeting on 24 September 1938And on March 6: Six More Insights About Workplace Bullying
Some of the lore about dealing with bullies at work isn't just wrong — it's harmful. It's harmful in the sense that applying it intensifies the bullying. Here are six insights that might help when devising strategies for dealing with bullies at work. Example: Letting yourself be bullied is not a thing. Available here and by RSS on March 6.

Coaching services

I offer email and telephone coaching at both corporate and individual rates. Contact Rick for details at rbrenIyeJIiAfnGdKlUXrner@ChacsxirZwZlENmHUNHioCanyon.com or (650) 787-6475, or toll-free in the continental US at (866) 378-5470.

Get the ebook!

Past issues of Point Lookout are available in six ebooks:

Reprinting this article

Are you a writer, editor or publisher on deadline? Are you looking for an article that will get people talking and get compliments flying your way? You can have 500-1000 words in your inbox in one hour. License any article from this Web site. More info

Follow Rick

Send email or subscribe to one of my newsletters Follow me at LinkedIn Follow me at X, or share a tweet Subscribe to RSS feeds Subscribe to RSS feeds
The message of Point Lookout is unique. Help get the message out. Please donate to help keep Point Lookout available for free to everyone.
Technical Debt for Policymakers BlogMy blog, Technical Debt for Policymakers, offers resources, insights, and conversations of interest to policymakers who are concerned with managing technical debt within their organizations. Get the millstone of technical debt off the neck of your organization!
Go For It: Sometimes It's Easier If You RunBad boss, long commute, troubling ethical questions, hateful colleague? Learn what we can do when we love the work but not the job.
303 Tips for Virtual and Global TeamsLearn how to make your virtual global team sing.
101 Tips for Managing ChangeAre you managing a change effort that faces rampant cynicism, passive non-cooperation, or maybe even outright revolt?
101 Tips for Effective MeetingsLearn how to make meetings more productive — and more rare.
Exchange your "personal trade secrets" — the tips, tricks and techniques that make you an ace — with other aces, anonymously. Visit the Library of Personal Trade Secrets.
If your teams don't yet consistently achieve state-of-the-art teamwork, check out this catalog. Help is just a few clicks/taps away!
Ebooks, booklets and tip books on project management, conflict, writing email, effective meetings and more.