Economic oracles: how markets solve complex problems

When we talk about making sensible plans, we're really asking: how do we figure out the best way to use what we have, to achieve what we want to achieve? It seems simple if we have all the right information, start with certain preferences, and know everything there is to know about the resources at our disposal. But that's not the whole story. The central challenge in planning is getting the right information to the right people. We've developed many mathematical tools and techniques to help us logically determine the best way to organise a set of resources efficiently. But these tools don't quite solve the economic puzzle. Why? Because they depend on having all relevant information available at the point of analysis. In reality, the information we need isn't neatly packaged for us. It's scattered among many individuals, and often it's incomplete or conflicting.

Markets as oracles

In 1945, Frederich Hayek posited that this challenge could be solved by the market price mechanism.¹ He imagined market prices as a kind of omniscient collective opinion, one that forms over countless interactions between self-interested individuals acting in accordance with their independent beliefs. In this way, the knowledge embedded in prices represents an aggregation of all relevant information that is initially scattered among many people. The intuition is compelling, because it suggests that market prices bring to light a kind of collective knowledge that surpasses what can be known by any one individual.

Hayek imagined market prices carrying information about changes in the supply and demand of valuable commodities like tin, or copper. However, market prices could also be specified on outcomes that are abstractly defined––like the solution to a complicated problem. Prices would then play the role of an oracle,² revealing information about potential solutions to the underlying problem.

A proxy for complex problems

Our research puts Hayek's intuition to the test by assessing whether markets can spread knowledge about solutions to the Knapsack problem³: a canonical example of a complex problem. The problem revolves around a simple premise: given a set of items, each with a specific weight and value, determine the combination of items that maximises the value while staying within a limited weight capacity. It's like the problem you face when you try to find the best way to pack your suitcase for a vacation, while keeping the weight of your luggage below the airline's limit.

The Knapsack problem is interesting because it belongs to a class of NP-hard problems that are exceptionally difficult to solve. These problems are difficult because known algorithms to identify solutions guarantee no greater efficiency than naive trial and error. As a result, the process of finding solutions becomes prohibitively time-consuming when the solution to the problem is just one among of millions of possible incorrect alternatives. It's like being asked to find a needle in a haystack, but the needle is so small and the haystack is so large that a lifetime of searching would only scratch the surface.

Omniscient prices

If markets can succeed in revealing solutions to complex instances of the Knapsack problem, we conjecture that they could also help solve other complex problems of interest to humanity. To test this conjecture, we ran an experiment in which groups of individuals attempted to solve difficult instances of the Knapsack problem while exchanging information about their solutions by trading in an experimental marketplace.

Over a series of sessions, we presented groups of individuals with the same Knapsack problem, allowing them ten minutes to individually attempt to find the best solution. While solving the problem, we gave everyone the opportunity to trade units of an experimental asset whose payoff depended on the collective performance of all individuals addressing the Knapsack problem. At the end of each session, each unit of this experimental asset promised its owner a payout equal to the best solution achieved by any participant during that session. In this way, people could earn money by purchasing the experimental asset at prices below their estimation of the best solution found, or by selling it at prices above the best solution.

Although everyone began with the same information about the underlying Knapsack problem, some were faster than others at finding superior solutions. By Hayek's intuition, the trading behaviour of these 'expert' individuals would cause the market price to converge towards the value of their superior solutions. In this way, prevailing market prices would act as an oracle: revealing information about the solutions of the best performing participants, and guiding others to improve their own solutions.

Just as Hayek predicted, our experiments corroborate the remarkable ability of markets to act as efficient devices to aggregate and disseminate information. Across all sessions, the market succeeded not only in spreading the best solution attained among participants, but also in revealing the true optimal solution. This was true even in the most challenging sessions, in which the optimal solution was just one out of almost 17 million possible alternatives.

Implications

Our results demonstrate an intriguing possibility: markets incentivise groups of people to try their best at solving hard problems, in such a way that enables individuals to benefit from the collective knowledge of others. This provides a solution to the primary issue with patent-based incentivies, where individuals are discouraged from sharing their knowledge with others, for fear of losing their competitive advantage.⁴

If markets can reveal optimal solutions to the Knapsack problem, they might also offer insights into more intricate issues, such as optimising resource allocation for climate change mitigation, drug-discovery, or large-scale infrastructure planning. Of course, the scope of market-based problem-solving is still not fully understood. Many questions remain: How do factors like participant expertise, the complexity of the problem, or the rules of market design affect the ability of prices to function as oracles?

One thing, however, is clear: the potential for markets to incentivise collective problem-solving is an exciting frontier. The implications extend well beyond traditional economics, opening new avenues for leveraging human interaction to tackle some of the world's most pressing challenges.