Coase GPT

Do large language models "understand" The Problem of Social Cost?

This week a couple of work tasks have given me an opportunity to experiment with the new AGI tools ChatGPT and Google Bard. I've played around with asking them factual questions to fill in gaps in my knowledge, which I then double checked with (non-Google) search engines to confirm, and they performed quite well. My "conversation" with ChatGPT about digitalization of electric systems was amusing, but I'll save that for another time ...

These large language models (LLMs) don't handle academic research questions very well. When I asked ChatGPT to provide citations to references for seminal articles in auction theory, it gave me references that don't exist, a phenomenon that some of my friends have also reported. When I asked Bard to cite its sources, it told me "as a large language model I am not designed to provide that information", so at least it's more honest about it than ChatGPT is. At the same time, though, when I asked Bard to summarize Vickrey (1961), the paper that kick-started modern auction theory and demonstrated the theoretical superiority of the second-price sealed-bid auction, it gave me an accurate, pretty good non-technical summary.

So I decided to test the two LLMs by asking for interpretations of a seminal piece of academic research that has had profound and wide-ranging effects on both research and practice, research that in my opinion is difficult to interpret and has many poor interpretations circulating in academic research, in practice, and in student essays.

I asked ChatGPT and Bard to summarize and interpret Ronald Coase's The Problem of Social Cost (1960). This transformational work is one of the most important articles ever written in economics and at the synthesis of law and economics, a synthesis that Coase's work catalyzed into an important interdisciplinary research field. In 2021 I wrote The Essential Ronald Coase, a book about Coase's work and its impact for the Fraser Institute's Essential Scholars series, and between that and decades of teaching environmental economics, I have a deep familiarity with the paper but still find that I don't fully understand its depths and implications. Still, I have a summary of it and my own interpretation in Chapter 3 of the book.

Coase's analysis is simultaneously simple and complex – simple in the sense that it says clarity in the assignment of property rights enables private bargaining to resolve resource use conflicts, complex in many senses including its characterization of external costs as reciprocal in nature (i.e., the paper mill's use of the river to discharge waste imposes a cost on the water bottling plant, and the water bottling plant's water quality requirement imposes a cost on the paper mill, they have conflicting uses of a scarce resource with ill-defined property rights). Coase used a series of legal cases to lay out his critique of Pigou's neoclassical externality theory, which he saw as incorrectly single-sided by focusing solely on the cost that the paper mill creates by discharging its waste into the river. Coase expands beyond Pigou's overly narrow "polluter pays" framework of internalizing externalities through taxation, and he does so by characterizing the problem of externality as one of ill-defined property rights due to high transaction costs. If you think of the problem as one of exchanging rights/compensating for harms, it becomes a more dynamic question of identifying which party is the least-cost avoider of the harm as well as figuring out how much of the harm should continue to occur/is not worth eliminating. Then the role of the law is to reduce transaction costs and assign liability to that least-cost avoider, although if transaction costs are zero the parties will bargain to the efficient outcome. In that case the legal assignment of rights/harms affects the distribution of outcomes, but not the achievement of an efficient outcome. This last point is Stigler's "Coase Theorem", which too many people (including too many academics who should know better) identify as "the result" of Coase (1960), a result that Coase did not think was important at all because in reality transaction costs are never zero.

I first asked the LLMs to "summarize Ronald Coase's 1960 paper "The Problem of Social Cost". ChatGPT started with

In his 1960 paper "The Problem of Social Cost," economist Ronald Coase addresses the issue of externalities, or the unintended costs or benefits that arise from economic transactions and affect people who are not directly involved in those transactions. Coase argues that when externalities exist, the allocation of resources is not optimal and that the market mechanism alone may not provide an efficient solution.

and Bard with

Ronald Coase's 1960 paper "The Problem of Social Cost" is a seminal work in law and economics. In the paper, Coase argues that the allocation of resources is efficient regardless of who owns the property rights, as long as there are no transaction costs.

Not great. Definitely too focused on the "so-called Coase Theorem". They both do a better job of identifying Coase's development of bargaining as a process to resolve conflicts:

[ChatGPT] Coase suggests that the traditional approach to dealing with externalities, which involves government intervention through regulation or taxes, may not be the most effective solution. He proposes an alternative approach that involves the affected parties negotiating with each other to reach an agreement that will result in the most efficient allocation of resources. This negotiation process, he argues, can lead to an optimal outcome if property rights are well defined and transaction costs are low.

[Bard] Coase begins by discussing the problem of externalities, which are costs or benefits that are not borne by the parties directly involved in a transaction. He then gives the example of a factory that emits smoke, which harms the people who live nearby. Coase argues that the optimal solution to this problem is not to regulate the factory, but to allow the factory owner and the homeowners to bargain with each other. If the factory owner is willing to pay the homeowners enough money, the homeowners will be willing to tolerate the smoke. Conversely, if the homeowners are willing to pay the factory owner enough money, the factory owner will be willing to reduce the amount of smoke it emits.

I then asked them "what role does the assignment of legal liability play in Coase's argument?"

[ChatGPT] Coase suggests that assigning legal liability to one party over the other will affect the incentives for negotiation and the distribution of costs and benefits.

In the absence of legal liability, the party who causes the harm (the "polluter") has no obligation to compensate the affected party for the harm done. As a result, the polluter has no incentive to reduce the harm, and the affected party has no incentive to negotiate a solution, as they have no legal claim to the damages caused.

On the other hand, if legal liability is assigned to the polluter, they will have an incentive to reduce the harm they cause to the affected party, either by reducing their activities or by compensating the affected party for the damages caused. This incentive to reduce harm will lead to negotiations between the parties, resulting in an optimal allocation of resources.

However, if legal liability is assigned to the affected party, the polluter will have no incentive to reduce the harm they cause. Instead, they will have an incentive to negotiate with the affected party to continue their activities, as they have no legal obligation to reduce the harm caused.

[Bard] Coase argues that the optimal allocation of resources will be achieved regardless of who owns the property rights, as long as there are no transaction costs. However, he also argues that the assignment of legal liability can affect the distribution of costs and benefits between the parties involved in an externality.

For example, if the factory owner is held liable for the harm caused by the smoke, then the factory owner will have to pay the homeowners for the damage. This will increase the cost of production for the factory owner, and it may lead to the factory owner reducing the amount of smoke it emits. Conversely, if the homeowners are held liable for the harm caused by the smoke, then the homeowners will have to pay the factory owner for the damage. This will increase the cost of living for the homeowners, and it may lead to the homeowners moving away from the factory.

Coase argues that the optimal assignment of legal liability is the one that leads to the most efficient allocation of resources. However, he also acknowledges that there may be other factors that need to be considered when assigning legal liability, such as fairness and equity.

Still not great. The characterization of one of the parties as "polluter" does not capture how deeply Coase's analysis is a critique of Pigou's single-sided "polluter pays" analysis. The ChatGPT response does articulate an important point that Coase identifies: the bargaining process is an important way to operationalize incentives to reduce harms/costs and is thus a mechanism driving toward improved outcomes through a reallocation of harms and which party pays for them. The Bard response doesn't reflect that. Responses from both LLMs are very contingent on how I worded my queries.

When asked "what does Coase mean when he says that the problem is a reciprocal problem?", ChatGPT completely missed it (so much so that I won't impose on your time to read it), but Bard did a pretty good job:

[Bard] When Coase says that the problem is a reciprocal problem, he means that the problem of externalities is not one-sided. It is not simply a problem of one party causing harm to another. Rather, it is a problem that involves two parties, each of whom is both a cause and a victim of the harm.

For example, in the case of the factory that emits smoke, the factory owner is causing harm to the homeowners. However, the homeowners are also causing harm to the factory owner, by demanding that the factory reduce its emissions. In this case, the factory owner and the homeowners are both both causes and victims of the harm.

Coase argues that this reciprocity is important, because it means that the optimal solution to the problem of externalities is not to simply regulate one party or the other. Rather, the optimal solution is to find a way to compensate both parties for the harm that they are causing and suffering.

I would quibble with some of this and if this were a student answer on a problem set or exam I'd ask them to sharpen their point and be more precise and give them partial credit, but the basic idea is there.

For my final query I asked them to interpret this quote from the article, after Coase's extended example of a railroad's sparks causing fires in farmers' crops:

How is it that the Pigovian analysis seems to give the wrong answer? The reason is that Pigou does not seem to have noticed that his analysis is dealing with an entirely different question. The analysis as such is correct. But it is quite illegitimate for Pigou to draw the particular conclusion he does. The question at issue is not whether it is desirable to run an additional train or a faster train or to install smoke-preventing devices; the question at issue is whether it is desirable to have a system in which the railway has to compensate those who suffer damage from the fires which it causes or one in which the railway does not have to compensate them. When an economist is comparing alternative social arrangements, the proper procedure is to compare the total social product yielded by these different arrangements. The comparison of private and social products is neither here nor there. (p. 34)

ChatGPT provided a superficially correct response, with the middle paragraph simply being a rewording of the quote:

[ChatGPT] In this quote, Coase is criticizing Pigou's analysis of externalities, arguing that Pigou has failed to recognize that his analysis is dealing with a different question than the one at issue. Coase is suggesting that Pigou's analysis is correct in itself, but it is illegitimate for him to draw the conclusion that government intervention, such as taxation or regulation, is always necessary to correct market failures caused by externalities.

Coase argues that the question at issue is not whether it is desirable to run an additional train or a faster train or to install smoke-preventing devices, but rather whether it is desirable to have a system in which the railway has to compensate those who suffer damage from the fires which it causes or one in which the railway does not have to compensate them. Coase suggests that the proper procedure for an economist when comparing alternative social arrangements is to compare the total social product yielded by these different arrangements, rather than simply comparing the private and social products.

In essence, Coase is suggesting that Pigou's analysis fails to consider the potential for bargaining between the parties involved in the externality and the reciprocal nature of the problem. Coase is arguing that government intervention is not always necessary to correct market failures caused by externalities, and that a negotiated solution may be more efficient in some cases.

Bard's reponse was a bit more substantive and not as reliant on the actual quote, but more importantly, it got to the main point which is that part of the benefit of bargaining is discovering the optimal level of the harm:

[Bard] Coase is arguing that Pigou's analysis is wrong because it focuses on the private and social costs of an activity, rather than the total social product. Coase argues that the proper procedure is to compare the total social product yielded by different social arrangements. This means that we should compare the total benefits and costs of an activity, regardless of who bears those benefits and costs.

Coase gives the example of a railway that emits smoke, which causes damage to nearby properties. Pigou would argue that the government should tax the railway to internalize the externality, so that the railway would bear the full cost of its activities. Coase argues that this is the wrong approach, because it would lead to a reduction in the total social product. The railway would be less likely to run trains, and the people who live near the railway would be worse off.

Coase argues that the better approach is to allow the railway and the homeowners to bargain with each other. If the railway is willing to pay the homeowners enough money, the homeowners will be willing to tolerate the smoke. Conversely, if the homeowners are willing to pay the railway enough money, the railway will be willing to reduce the amount of smoke it emits.

Coase argues that this approach will lead to a more efficient outcome, because it will take into account the interests of both the railway and the homeowners. The railway will only be willing to pay the homeowners as much money as it is worth to them to tolerate the smoke. The homeowners will only be willing to pay the railway as much money as it is worth to them to reduce the amount of smoke.

One conclusion I've drawn from this exercise is that these LLMs can be useful in helping you understand an article in an area that's new to you, and the strategy I would follow is what I did in the final prompt – pick out a quote from the article and ask it to interpret the quote. The overall "summarize Coase (1960)" is too generic to be useful (rather like summarize Proust). There is no depth of understanding in these models, which reflect a more superficial understanding culled from internet aggregation, but if you are looking for a simpler way to articulate something, being more specific in your query will get you better results. My results are also consistent with the hypothesis that these LLMs are not an existential threat, for what that's worth, so I will not be joining the moral panic of the week, but will use these LLMs as a tool.