What is the Goodhart’s Law?

When the focus is set in only one measure, people optimize that single measure.

A good example is the so-called “Cobra Effect”. In India, the government offered money in exchange for each dead Cobra that was turned in, to reduce the abundance of loose cobra snakes on the Indian streets. At first, the policy seemed to be successful: people killed loose cobra snakes for the reward.

But after some successful time, people began to house breed cobras and hand them to the government to receive the bounty.

After the government became aware of this strategy, they decided to scrap the cobra bounty program. Guess what happened: people released cobras free. Suddenly, the number of loose cobras on the street increased: the government policy failed!

What does this teach us?: When an optimization measure is set, people can manipulate it to meet the target. Goodhart’s law has been popularized as follows:

Goodhart’s Law says that when a measure becomes a target, it ceases to be a good measure.

Who is Charles Goodhart? Charles Goodhart is a British economist born in 1936. He was a member of the Bank of England’s Monetary Policy Committee and professor at the London School of Economics. The idea of the Goodhart’s Law was first advanced in a 1975 paper. That paper was later used to criticize the monetary policy of the government of Margaret Thatcher.

The original formulation of Goodhart, made in 1975, was:

Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.

Goodhart’s Law and related ideas are used in many areas of economics. The Law it’s implied by the idea of rational expectations: people are aware of the implications of its actions and act according to them.

Goodhart’s Law has been beautifully formulated by Jón Danı́elsson (an economist teaching at the London School of Economics):

Any statistical relationship will break down when used for policy purposes.

Examples of Goodhart’s Law

Goodhart’s Law has implications across many fields, not only in economics but also in business policies, data science, etc:

Some examples:

- Search Engine Optimization: For many years, Google used a system called PageRank to sort it search results. PageRank used the number of backlinks each webpage had as a strong proxy of the quality of it. Webmaster started to implement several tactics to increase the number of backlinks, instead of increasing the quality of the content. For example, they exchanged backlinks with each other or built many websites with a lot of backlinks between them (Backlinks Farms). Google modified it’s search results sorting algorithm to overcome the Goodhart’s Law. Many aspects of the current system are not made public, to avoid the Goodhart’s Law.

- Sales targets: Many businesses set sales targets to increase the productivity of salespeople. For example a car salesman needs to sell 20 cars per month to receive a bonus payment. At the end of the month, they will make a lot of phone calls and usually offer discounts or perks to reach the 20 cars target. This strategy may be detrimental to the business if the discounts reduce profits.

- Coupons strategy: When companies regularly offer discount coupons, people can delay purchases to get future discounts.

- Company perks: many startups offer nice perks to attract star employees. But maybe perks attract only employees interested in perks and not necessarily the best employees.

- Call centers: Many call centers set average call time targets, like three minutes per call. This policy can be detrimental to the quality of services: many customer support specialists can be unhelpful to people to reduce the average time per call.

- School and university notes as a proxy for quality learning. Students are memorizing for test grades instead of actually deep learning content.

- Research papers in academic world: researchers are eager to get their papers published: the quantity of published research is usually used as a proxy for academic productivity. But this has lead to many scientists manipulating the data to achieve statistical significance. For example, by using data subsets. This has also lead to the bias of the publication of only impressive results, while many papers on the same subject that didn’t achieve statistical significance were not published.

Implications in Data Analysis

When a model based on past data is deployed to the real world, people may start altering their behavior. This could invalidate the model. In Data Science, the Goodhart’s Law can also be expressed as follows: “the behavior may change because of the models presence”.

We can test for the presence of Goodhart’s Law using time series and comparing the model fit before and after the implementation of the model.

How to Overcome the Goodhart’s Law?

A first option is to use better measurements. Measurements that take multiple factors into consideration.

In cases where the rewards do not need to be explicitly communicated, agents will not have a proxy to optimize. They will use human discretion and try to optimize the overall outcome based on their common sense.

Avoiding large groups and hierarchies can also avoid the need for introduction of targets or KPIs. Small groups will naturally take into account many indicators as a measure of success.

Breaking Down the Goodhart’s Law

Goodhart’s Law is present in many scientific and business fields. Scientist, policy makers and business managers need to be aware of the Law and avoid falling in the trap of using a single KPI (Key Performance Indicator) as a measure for success.

About the Author: Federico Anzil is an economist and analyst.