The Knowledge Problem and Institutions

What is the knowledge problem and why does it matter?

Photo by Alina Grubnyak on Unsplash

When I kicked off this Substack in January 2023 to reinvigorate my Knowledge Problem blog, I didn't explain the name's origin. Some of you probably recognize it, but many of you probably wonder what the name signifies. The knowledge problem is a fundamental and pervasive characteristic of people, individually and socially. About a decade ago I was pleased to be invited to write the chapter on the knowledge problem for the Oxford Handbook of Austrian Economics (here's the manuscript version of the chapter); today I'm going to edit and riff off of that chapter to lay the groundwork for how the knowledge problem does and should influence institutions, the formal and informal “rules of the game” that structure our incentives and our interactions (Douglass North, Institutions, Journal of Economic Perspectives, 1991). I discussed the knowledge problem in some depth as it relates to prices in a newsletter in May, in response to Daron Acemoglu’s inaccurate reading of Hayek’s The Use of Knowledge in Society (1945). Today I'll elaborate on what the knowledge problem is and what it implies for institutions.

Markets Are Knowledge Ecosystems

In “The Use of Knowledge in Society”, Hayek argued that the fundamental economic problem is not resource allocation based on a given set of preferences and technical capabilities. Instead, coordinating decisions and actions among interacting individual agents with diffuse private knowledge and plans is the fundamental problem. Knowledge"‘s diffuse and private nature hampers such coordination, but out of human interaction, institutions emerge that enable decentralized coordination. Prices and market processes are such an institution. Hayek also argued that knowledge is not the same as “scientific” information, there is no given and uniform set or distribution of data, and such information fails to capture all knowledge relevant to both static and dynamic decision-making and coordination.

The Complexity Knowledge Problem

Hayek’s article Economics and Knowledge (1937) examined the information assumptions associated with the neoclassical economic concept of equilibrium. Information (or data), knowledge, and foresight undergird the equilibrium state, a state in which agents have coordinated, mutually compatible plans from which no agent deviates. What knowledge is necessary for achieving equilibrium, and how do agents acquire it? [In the 1930s Hayek’s colleague Oskar Morgenstern, famous for developing game theory with John von Neumann, was also working on the implications of common knowledge and perfect foresight assumptions in economics.] In the full information assumptions underlying the neoclassical equilibrium model, an individual knows his/her preferences or cost of production and there is full common knowledge about market prices across everyone.

Hayek’s counter to this argument is twofold. First, the knowledge relevant to decision-making goes beyond market prices; it also includes knowledge such as individual skills and alertness to or awareness of profit opportunities. Second, the knowledge relevant to decision-making is dispersed among agents in the economy, but a market economy enables the “spontaneous interaction” of these agents with diffuse private knowledge to bring about an outcome “in which prices correspond to costs” (pp. 50-51). Analyzing the importance of the dispersion of knowledge marked the beginning of Hayek’s extensive work on the knowledge problem.

Hayek’s initial arguments focused on what Esteban Thomsen (Prices and Knowledge, 1992) called the “complexity knowledge problem”. The “man on the spot” (1945, p. 524) has subjective, private knowledge of “the particular circumstances of time and place” (p. 522). That knowledge is decision-relevant and cannot be accessed by anyone else, except through a decentralized system of prices and a market process of exchange to discover those prices. Prices economize on the communication and interpretation of knowledge, and market processes provide feedback channels. Feedback loops, learning, adaptation to a changing environment and changing actions and plans of others, are all important implications of Hayek’s argument. Prices provide profit opportunities and realized profits, which serve as feedback to spur the discovery of new products, services, business models, or other ways to create value through economic activity.

Markets are processes for social learning and provide feedback channels for entrepreneurship. But prices cannot convey full knowledge among individuals. Instead they communicate the consequences of the actual actions and interactions of many agents. For that reason we should not think of prices as containing full knowledge, but instead as knowledge surrogates.

Full knowledge is unrealistic and not achievable by any person or group of people.

The Contextual Knowledge Problem

The complexity knowledge problem is not the only knowledge problem. The other, that Thomsen calls the “contextual knowledge problem”, is the epistemic fact that some knowledge relevant to coordination does not exist outside of the market context. Such knowledge is either created in the process of market interaction (such as learning what your preferences are in that moment given the context), tacit knowledge that is not consciously known, or inarticulate knowledge that is difficult to express or aggregate. Riding a bicycle is a classic example of tacit knowledge, with inarticulate knowledge as a sub-category. While you can explain the basic mechanics of how a bicycle works and how to ride one, the actual skill of balancing, pedaling, and steering is difficult to convey fully through words or diagrams. It's something that is best learned through direct experience and practice. People who know how to ride a bike may not be able to describe in detail how they maintain balance or make adjustments to stay upright; it"‘s a skill that becomes second nature through practice and muscle memory.

The contextual knowledge problem incorporates the idea of knowledge generation within the market process. It also contributes to understanding feedback effects in social systems. For example, contextual knowledge means that individuals and the patterns that emerge from their aggregate interactions in social systems are prone to amplification through positive feedback, such as riots, bubbles, and bank panics (and more innocuous effects, like doing the wave at sporting events or participating in a standing ovation). But contextual knowledge also helps us understand the dampening, or equilibrating, negative feedback effects in markets that take the form of entrepreneurial alertness (Kirzner 1997).

Later scholars have also synthesized and developed further both the complexity and contextual knowledge problems. In Knowledge and Decisions, Thomas Sowell developed these ideas and used them to analyze the social processes and institutions to aggregate fragmentary knowledge, coordinate decisions across individuals possessing this fragmentary knowledge, and generate beneficial outcomes and enable a complex society to emerge and function.

The unifying theme of Knowledge and Decisions is that the specific mechanics of decision-making processes and institutions determine what kinds of knowledge can be brought to bear and with what effectiveness. In a world where people are preoccupied with arguing about what decisions should be made on a sweeping range of issues, this book argues that the most fundamental question is not what decision to make but who is to make it – through what processes and under what incentives and constraints, and with what feedback mechanisms to correct the decision if it proves to be wrong. (1980 [1996], p. xxii)

Sowell used the reality of the knowledge problem as a starting point to analyze the institutions humans use to make collective decisions. In my own work I use Sowell’s framework for grounding institutional analysis of regulation and technological change in the pervasiveness of the knowledge problem.

Implications for Institutional Design

The knowledge problem is particularly relevant in political economy, or the comparative analysis of economic and social institutions and their performance. Some of the most important implications of the knowledge problem arise in the area of institutional design. All economic activity takes place within an institutional context, as North (1991) described. The important determinants of an individual's actions and interactions are his or her preferences, the environment, and the institutional framework (a modeling framework that also informs research and practice in experimental economics (video)). Institutions interact with preferences and the environment to determine individual actions and interactions with others. Thus economists can evaluate the relationship of institutions to outcomes, taking into account the incentives they generate and the features of the environment in which the institutions operate. Using the knowledge problem idea that the fundamental economic problem is the coordination of actions and plans among heterogeneous agents with dispersed, private, contextual knowledge, the institutional question is one of evaluating the emergence of and the deliberate design of institutions that maximize social cooperation.

Institutions often emerge out of history, experience, and trial-and-error experimentation, and in fact formal legal institutions have often been codifications of informal norms and emergent informal legal systems, some of which had existed for centuries. In this sense we can think of, for example, the English common law as an emergent legal order, or an organic, grown legal order grounded in basic principles of property rights, commutative justice to defend negatively-defined rights, rights of contract, and consent (Hayek, The Constitution of Liberty, 1960). The long evolution of precedent and the roles of judges in interpreting the common law provide feedback channels that enable this legal institution to change over time, to adapt to changes in the economic, social, and physical environment and changes in preferences. The experiences and information flows that inform this evolution feed into the legal system in a bottom-up manner, through individual interactions and experimentation.

The common law is compatible with both the complexity and the contextual aspects of the knowledge problem. The common law does not require aggregation of diffuse private knowledge in order for a community to meet and enforce its principles, and its grounding in these basic principles enables individuals to prosper from the exercise of their division of knowledge through mutually beneficial exchange. It also does not restrict the ability of individuals to act on their contextual knowledge, as long as those actions do not harm the rights of others with respect to life, property, contract, and consent. In the course of creating and exploiting such opportunities, people actually create a more complex economic and social order as they innovate and take entrepreneurial actions to create new value propositions; our modern society is an example of this general pattern in economic history.

Contrast this concept of emergent institutions and the example of the English common law with the idea of deliberately designed institutions. Such institutions typically are created through legislative and administrative procedures, imposed in a top-down fashion to achieve a specific objective. Economic regulation frequently takes this form, whether at the local level (e.g., zoning regulations) or at the national level (e.g., regulations on the business activity of the financial industry).

Designed institutions do not always facilitate outcomes consistent with taking into account the knowledge problem, and thus do not necessarily maximize social cooperation. They often limit the ways that individuals could create mutual benefit by making use of diffuse private knowledge. A concrete example of this problem is the cost-based regulation of retail electricity prices, which for most customers for over a century in the United States have been fixed to reflect the average cost per unit (Kiesling 2008). This unvarying regulated price and the operation of the retail provider as a government-granted monopoly preclude any competition or product differentiation such as the ability to choose contracts with differentiated pricing. Such product differentiation could benefit both producer and consumer, and could enable signaling the relative value of electricity in that hour as a knowledge surrogate. The top-down imposition of price controls stifles the exercise of private knowledge and the creation of contextual knowledge, resulting in an inferior outcome to the extent that consumer preferences deviate from the regulated price and service combination. The regulated monopoly status means that the firm never faces a market test of its offerings. Price controls flatten the knowledge landscape because they truncate both the communication of dispersed knowledge and the creation of contextual knowledge.

The institutional and political economy implications of the knowledge problem are significant, because the knowledge problem strikes at the core of some of the most essential assumptions underlying government action and regulation. At some level all arguments for government intervention, and the associated theoretical models and institutional designs, rely on the presumption of the existence, knowability, and stability of an optimal outcome. The complexity knowledge problem suggests that optimal outcomes are not knowable ex ante, while the contextual knowledge problem means the stability of an outcome's optimality is context-dependent; and if such an outcome does exist, its existence may be fleeting. The institutional context will affect the knowledge and the actions available to individuals who are trying to coordinate their actions for mutual gain. Thus a regulatory plan or institutional design based on models that ignore the knowledge problem is grounded in unrealistic assumptions, and is prone to unintended consequences and an inability to achieve the desired outcome.

Knowledge is inherently imperfect because it is dispersed, private, local, often tacit, frequently inarticulate, sometimes ephemeral, and usually contextual. Economic models based on assumptions of perfect knowledge thus do a poor job of capturing the informational and epistemological factors most relevant to both static and dynamic decision-making and economic calculation. Economic and social institutions designed deliberately based on those models are unlikely to perform well at generating prosperity, as research in robust political economy indicates, and as Hayek suggested in his Nobel address (1974):

If man is not to do more harm than good in his efforts to improve the social order, he will have to learn that in this, as in all other fields where essential complexity of an organized kind prevails, he cannot acquire the full knowledge which would make mastery of the events possible. He will therefore have to use what knowledge he can achieve, not to shape the results as the craftsman shapes his handiwork, but rather to cultivate a growth by providing the appropriate environment, in the manner in which the gardener does this for his plants. … The recognition of the insuperable limits to his knowledge ought indeed to teach the student of society a lesson of humility which should guard him against becoming an accomplice in men's fatal striving to control society – a striving which makes him not only a tyrant over his fellows, but which may well make him the destroyer of a civilization which no brain has designed but which has grown from the free efforts of millions of individuals.