Markets Are Knowledge Ecosystems

How do prices aggregate, coordinate, and create knowledge, and why can't AI substitute for cognition?

Markets are knowledge ecosystems. I say that a lot, to many people from diverse backgrounds, and at first most people are a little puzzled until they think about it for a bit and we talk about what prices are and what function they serve in the open, intertemporal, complex systems of economy and society. But what does it mean for markets to be knowledge ecosystems? The answer to that question is relevant to MIT economist Daron Acemoglu writing this week about knowledge and information in the context of AI's effects on work and his new book with Simon Johnson, Power and Progress.

Markets as ecosystems

Source: National Geographic

Think about an ecosystem, say a wetlands with myriad plants and animals, including moose. When we think about ecosystems we think about them as organic systems with interdependent plants and animals that thrive through a network of feedback effects that move information around about where the yummiest grasses are growing, how many baby moose there are this spring, and whether there's enough or too little water or too much heat or cold. Darwin's understanding of the dynamics of ecosystems and of evolution still holds – the existing environment and those feedback loops and interdependencies create a network through which changes in the environment lead to changes in the emergent outcomes, changes in animal birth and death rates, in food availability along the food chain, and changes in the architecture of the overall ecosystem. The context provides information, and the members of the system take actions that can change the context, feeding into the actions of others. Dynamic fitness with the changing environment, not perfect optimization, is the key to survival and growth.

That's what markets do with humans, with our exchange interactions, and with our decisions about what to invest and how much and how to innovate and create new market ecosystems. As in the wetlands, the context provides information, and members of the economy and society are interdependent and their choices change the context and feed into the feedback effects on other participants. What's salient in the human market ecosystem, though, is the role that the price system plays as this information network to inform the feedback loops. Markets are knowledge ecosystems because prices connect individuals in the network of exchange and communicate some of their dispersed, private, subjective, tacit knowledge through the actions they take. The emergent outcomes we observe are a consequence of making choices and taking actions

The more we represent markets as ecosystems, and the price system as the nervous system of that ecosystem, the less we think mechanistically about markets, and that's a good thing.

Knowledge, information, prices

Economists do not always define knowledge explicitly and often conflate knowledge with information, although they are different in meaningful ways. My working definition of knowledge is based on F.A. Hayek's work (and is the reason I use the name Knowledge Problem), and I wrote about it in my chapter on the knowledge problem in the Oxford Handbook of Austrian Economics (2015). If they think about Hayek's work on knowledge, most people think of his seminal paper The Use of Knowledge in Society (AER 1945). There Hayek defines different categories of knowledge that are relevant to economic decision-making, one being scientific knowledge (what today we might call data, or information), and the other being a larger, more abstract category:

Today it is almost heresy to suggest that scientific knowledge is not the sum of all knowledge. But a little reflection will show that there is beyond question a body of very important but unorganized knowledge which cannot possibly be called scientific in the sense of knowledge of general rules: the knowledge of the particular circumstances of time and place. It is with respect to this that practically every individual has some advantage over all others in that he possesses unique information of which beneficial use might be made, but of which use can be made only if the decisions depending on it are left to him or are made with his active cooperation. We need to remember only how much we have to learn in any occupation after we have completed our theoretical training, how big a part of our working life we spend learning particular jobs, and how valuable an asset in all walks of life is knowledge of people, of local conditions, and special circumstances. To know of and put to use a machine not fully employed, or somebody's skill which could be better utilized, or to be aware of a surplus stock which can be drawn upon during an interruption of supplies, is socially quite as useful as the knowledge of better alternative techniques. And the shipper who earns his living from using otherwise empty or half-filled journeys of tramp-steamers, or the estate agent whose whole knowledge is almost exclusively one of temporary opportunities, or the arbitrageur who gains from local differences of commodity prices, are all performing eminently useful functions based on special knowledge of circumstances of the fleeting moment not known to others. (1945, pp. 521-522)

This dispersed, private knowledge of "time and place" is both knowledge about local circumstances that can only be acquired experientially and a person's knowledge of their own preferences, their own opportunity costs, their own evaluation of tradeoffs. It's different from scientific knowledge, or information, which is knowable and discoverable by others (although perhaps at a (search) cost). Hayek goes on to argue that this diverse, diffuse, private knowledge plays a pivotal role in economic decision-making that allows coordination across the actions and plans of strangers scattered around the world to emerge. That coordination, he says, is the fundamental economic problem in a dynamic and complex system of systems.

The complexity knowledge problem is superficially computational but actually cognitive

Following Esteban Thomsen (Prices and Knowledge, 1992), in my handbook chapter I called the challenge of making use of dispersed private knowledge the complexity knowledge problem: "the difficulty of coordinating individual plans and choices in the ubiquitous and unavoidable presence of dispersed, private, subjective knowledge" (p. 46). Aggregating that knowledge is costly, if not impossible.

Prices, and the price system generally, emerged over the human history of exchange as the most robust and parsimonious way to grapple with this complexity:

Prices economize on the communication and interpretation of knowledge among dispersed agents.

How do individuals learn the plans of others? How do they learn when they are wrong and take action accordingly? Prices and market processes provide feedback channels. Feedback loops, learning, adaptation to a changing environment and changing actions and plans of others, interdependence of agents and their actions in a complex system, and how prices and markets serve as feedback loops making a complex system adaptive are all important implications of Hayek’s argument. Prices provide profit opportunities and realized profits, and those realized profits serve as feedback that can spur the discovery of new products, services, business models, or other ways to create value through economic activity. Alert entrepreneurs see these opportunities, learn from observed and realized feedback, and adapt their plans accordingly. ... Markets are processes for social learning and provide feedback channels for entrepreneurial alertness. (Kiesling 2015, pp. 49-50)

Prices turn some, but not all, knowledge into actionable information. Since prices reflect only peoples' decisions to buy or not to buy/sell or not to sell, they do not and cannot reflect everything about their preferences, opportunity costs, and experiential knowledge of time and place.

This complexity-coordination aspect of knowledge has, to a large extent, been incorporated into economics more generally, and if an economist is aware of Hayek's knowledge problem, it's usually restricted to this sense. In economics it has led to the development of the field of mechanism design from Leonid Hurwicz's theoretical work on the question of "what is the minimum amount of information required for markets to achieve competitive outcomes?" As Brian Albrecht pointed out in a really great post on Tuesday, Hurwicz had been a student of Hayek's at the London School of Economics, and through his work with Stan Reiter and others, Hurwicz integrated Hayek's complexity and coordination analysis with work coming out of systems theory and information theory (Herb Simon, Claude Shannon) in the 1960s and 1970s. Mechanism design takes the dispersed, subjective knowledge of individuals, represents it mathematically, and then examines institutional design to enable market participants to create the most surplus/gains from trade possible, given that private knowledge is not available to others as knowable and actionable information.

Here the complexity knowledge problem intersects with both the origins of Hayek's knowledge problem concept and the modern questions around increased computation and AI and their ability to substitute for humans and markets. Hayek started working on this topic in the 1930s in the context of the socialist calculation debate that he and Ludwig von Mises were having with Oskar Lange and Abba Lerner, among others. In the 1920s Mises had argued that central planning could not replace decentralized market processes because the computation required to replicate the prices that emerge from market processes was impossible. Over the next two decades, though, he and Hayek expanded the argument beyond computation to incorporate the complexity knowledge problem. Since that time, people working in cybernetics and others have argued that improved computation does enable central planning where it was not possible before.

Which brings us to Acemoglu's Twitter thread from earlier this week delving into this computational question and concentration in the age of AI. He draws some connections to concentration of power to connect his thread to his book, but I want to focus on his interpretation of the knowledge problem:

Hayek’s argument offers an original and ingenious “computational” critique of central planning. His basic premise is that there is a huge amount of dispersed knowledge in society about a very large number of goods and services (e.g., people’s preferences). ... Hayek argues that the market system efficiently aggregates information via the price system. Prices will adjust when there is excess supply or demand for some goods, and when this process stops, the relevant economic aspects of this dispersed information is taken into account. ... Coming back to Hayek’s argument, there was another aspect of it that has always bothered me. What if computational power of central planners improved tremendously? Would Hayek then be happy with central planning?

This is an incorrect reading of Hayek, although it's more accurate as a reading of Mises' early contributions to the socialist calculation debate. Hayek was not making a computational argument, he was making a cognitive neuroscience argument about the limitations on our ability to access the minds of others. Much of Hayek's work synthesized his background in law and biology with his economics, most evident in his article Economics and Knowledge (Economica 1937) and his book The Sensory Order (1952). Superficially the problem is computational, but even better-faster-cheaper computation cannot overcome the cognitive limitations that humans have because our perceptions of the world are subjective and filtered through our own minds, and because we cannot access the minds of others.

The knowledge problem is not that dispersed knowledge is costly to compute, it's that we are cognitively bounded away from being able to compute it.

Individuals make decisions based on perception, which can be distinct and subjective because of the particulars of individual experience but which also has consistency across people because of the evolutionary process.

Furthermore, the human mind has cognitive limitations, because in its entirety, the human mind cannot be comprehended fully by the human mind; an individual cannot grasp all of the relevant factual, institutional, and cognitive knowledge for decision-making because of the inescapable embeddedness of the human mind in the system. Hayek reaches this conclusion because he conceives of the mind as a self-organizing, emergent order; through evolutionary processes involving both the brain’s physical structure and the cultural and empirical experiences of individuals, the mind’s capacity to classify and distinguish emerges, beyond the conscious control of any one person. This theory of mind has significant implications for the definition of knowledge and the use of knowledge: “What this amounts to is that all the ‘knowledge’ of the external world which such an organism possesses consists in the action patterns which the stimuli tend to evoke, or, with special reference to the human mind, that what we call knowledge is primarily a system of rules of action assisted and modified by rules indicating equivalences or differences or various combinations of stimuli” (Hayek "Primacy of the Abstract", 1969, p. 41). The mind categorizes information and inputs (stimuli) and uses perception in the process, resulting in knowledge. (Kiesling 2015, pp. 50-51)

Note also that if Hayek is correct about the knowledge of time and place and “practically every individual has some advantage over all others in that he possesses unique information of which beneficial use might be made” of it, if that knowledge is important in economic decision-making that will attenuate or prevent some, if not all, of the deleterious effects of power concentration that Acemoglu worries about in his Twitter thread and book.

The contextual knowledge problem

Brian Albrecht's post offers a solid critique of Acemoglu's interpretation of Hayek, and it moves us in the direction of the second type of knowledge problem that Hayek addresses: in addition to the complexity knowledge problem, there is the contextual knowledge problem: "The epistemic fact that some knowledge relevant to such coordination does not exist outside of the market context; such knowledge is either created in the process of market interaction, tacit knowledge that is not consciously known, or inarticulate knowledge that is difficult to express or aggregate." (Kiesling 2015, p. 46) Not all knowledge that's relevant to economic decision-making is even knowable by individuals in advance – some of our most useful knowledge emerges only within a context in which we confront a choice.

If I asked you, out of the blue, what's your preference/willingness to pay for a can of LaCroix pamplemousse sparkling water (the best flavor!), how would you answer? Would you even know how to answer? Probably not, because knowledge like that is really only created in a context where we have to make a choice: I'm at the airport, and I can either fill my water bottle at the fountain at zero price or buy a can of LaCroix for $1.99. Or, I've just walked 5 miles along the lakefront in Chicago and roll up at the Diversey golf driving range's cafe, and it's 80 degrees out and sunny, do I pay the $1.49 for the can of LaCroix? That knowledge only emerges within the very personal and local context of my situation and my preferences in that situation.

Knowledge that we use to make economic decisions is also often tacit. Every day we use knowledge that we don't know we possess to make choices and to engage in exchange.

Consider, as [economist Don] Lavoie does, the problem of riding a bicycle: most people ride bicycles even without knowing the physics underlying how to maintain balance on a bicycle, without solving the equations for the bicycling equilibrium. Knowing how to ride a bike is an example of inarticulate knowledge, the difference between “knowing that” if I pedal fast enough the wheels will turn to maintain balance and “knowing how” the physics works. Similarly, inarticulate knowledge informs economic decision-making, and that knowledge emerges from our actions and interactions in the process of exchange. An important implication of the pervasiveness of inarticulate knowledge is that an ex ante, nonmarket, decentralized mechanism cannot replicate either the efficiency or the knowledge-creating or knowledge-revealing effects of prices and market processes. Decentralized market processes and price mechanisms can elicit and make use of inarticulate knowledge where other institutions cannot. (Kiesling 2015, p. 55)

In his 1945 article Hayek had not yet fully developed this contextual knowledge problem (which he did in later work in the 1960s), but he starts to point in that direction. Through an overly-narrow, isolated, and inaccurate reading of Hayek (1945), Acemoglu mistakenly categorizes the knowledge problem as a computational problem when in fact it's a deep and inescapable cognitive characteristic of being human.

We can analyze patterns in data after the fact, and we can use AI and machine learning to help us make out-of-sample predictions of future outcomes using those historical data, but that is different from being able to access and take action on private, subjective, contextual, and tacit knowledge in real time. Even sophisticated computation and AI cannot substitute for decentralized market processes and for humans taking action and making choices.