2020-07-26

EA ideas 3: uncertainty

2.0k words (7 minutes)
 
Posts in this series:
FIRST | PREVIOUS | NEXT

Moral uncertainty is uncertainty over the definition of good. For example, you might broadly accept utilitarianism, but still have some credence in deontological principles occasionally being more right.

Moral uncertainty is different from epistemic uncertainty (uncertainty about our knowledge, its sources, and uncertainty over our degree of uncertainty about these things). In practice these often mix – uncertainty over an action can easily involve both moral and epistemic uncertainty – but since is-ought confusions are a common trap in any discussion, it is good to keep these ideas firmly separate.


Dealing with moral uncertainty

Thinking about moral uncertainty quickly gets us into deep philosophical waters.

How do we decide which action to take? One approach is called “My Favourite Theory” (MFT), which is to act entirely in accordance to the moral theory you think is most likely to be correct. There are a number of counterarguments, many of which involve around problems of how we draw boundaries between theories: if you have 0.1 credence in each of 8 consequentialist theories and 0.2 credence in a deontological theory, should you really be a strict deontologist? (More fundamentally: say we have some credence in a family of moral systems with a continuous range of variants – say, differing by arbitrarily small differences in the weights assigned to various forms of happiness – does MFT require we reject this family of theories in favour of ones that vary only discretely, since in the former case the probability of a particular variant being correct is infinitesimal?). For a defence of MFT, see this paper.

If we reject MFT, when making decisions we have to somehow make comparisons between the recommendations of different moral systems. Some regard this as non-sensical; others write theses on how to do it (some of the same ground is covered in a much shorter space here; this paper also discusses the same concerns with MFT that I mentioned in the last paragraph, and problems with switching to “My Favourite Option” – acting according to the option that is most likely to be correct, summed over all moral theories you have credence in).

Another less specific idea is the parliamentary model. Imagine that all moral theories you have some credence in send delegates to a parliament, who can then negotiate, bargain, and vote their way to a conclusion. We can imagine delegates for a low-credence theory generally being overruled, but, on the issues most important to that theory, being able to bargain their way to changing the result.

(In a nice touch of subtlety, the authors take care to specify that though the parliament acts according to a typical 50%-to-pass principle, the delegates act as if they believe that the percent of votes for an action is the probability that it will happen, removing the perverse incentives generated by an arbitrary threshold.)

As an example of other sorts of meta-ethical considerations, Robin Hanson compares the process of fitting a moral theory to our moral intuitions to fitting a curve (the theory) to a set of data points (our moral intuitions). He argues that there’s enough uncertainty over these intuitions that we should take heed of a basic principle of curve-fitting: keep it simple, or otherwise you will overfit, and your curve will veer off in one direction or another when you try to extrapolate.


Mixed moral and epistemic uncertainty

Cause X

We are probably committing a moral atrocity without being aware of it.

This is argued here. The first argument is that past societies have been unaware of serious moral problems and we don’t have strong enough reasons to believe ourselves exempt from this rule. The second is that there are many sources of potential moral catastrophe – there are very many ways of being wrong about ethics or being wrong about key facts – so though we can’t point to any specific likely failure mode with huge consequences, the probability that at least one exists isn’t low.

In addition to an ongoing moral catastrophe, it could be that we are overlooking an opportunity to achieve a lot of good for cheap. In either case there would be a cause, dubbed Cause X, which would be a completely unknown but extremely important way of improving the world.

(In either case, the cause would likely involve both moral and epistemic failure: we’ve both failed to think carefully enough about ethics to see what it implies, and failed to spot important facts about the world.)

“Overlooked moral problem” immediately invites everyone to imagine their pet cause. That is not what Cause X is about. Imagine a world where every cause you support triumphed. What would still be wrong about this world? Some starting points for answering this are presented here.

If you say “nothing”, consider MacAskill’s anecdote in the previous link: Aristotle was smart and spent his life thinking about ethics, but still thought slavery made sense.


Types of epistemic uncertainty

I use the term "epistemic uncertainty" because the concept is broader than just uncertainty over facts. For example, our brains are flawed in predictable ways, and dealing with this is different from dealing with being wrong or having incomplete information about a specific fact.

Flawed brains

A basic cause for uncertainty is that human brains make mistakes. Especially important are biases, which consistently make our thinking wrong in the same way. This is a big and important topic; the classic book is Kahneman’s Thinking, Fast and Slow, but if you prefer sprawling and arcane chains of blog posts, you’ll find plenty here. I will only briefly mention some examples.

The most important bias to avoid when thinking about EA may be scope neglect. In short, people don’t automatically multiply. It is the image of a starving child that counts in your brain, and your brain gives this image the same weight whether the number you see on the page has three zeros or six after it. Trying to reason about any big problem without being very mindful of scope neglect is like trying to captain a ship that has no bottom: you will sink before you move anywhere.

Many biases are difficult to counter, but occasionally someone thinks of a clever trick. Status quo bias is a preference for keeping things as they are. It can often be spotted through the reversal test. For example, say you argue that we shouldn’t lengthen human lifespans further. Ask yourself: should we then decrease life expectancy? If you think that we should have neither more nor less of something, you should also have a good reason for why it just so happens that we have an optimum amount already. What are the chances that the best possible lifespan for humans also happens to be the highest one that present technology can achieve?


Crucial considerations

A crucial consideration is something that flips (or otherwise radically changes) the value of achieving a general goal.

For example, imagine your goal is to end raising cows for meat, because you want to prevent suffering. Now say there’s a fancy new brain-scanner that lets you determine that even though the cow ends up getting chucked into a meat grinder, on average the cow’s happiness is above the threshold for when non-existence is preferable to existence (assume this is a well-defined concept in your moral system). Your morals are the same as before, but now they’re telling you to raise more cows for meat.

An example of a chain of crucial considerations is whether or not we should develop some breakthrough but potentially dangerous technology, like AI or synthetic biology. We might think that the economic and personal benefits make it worth the expense, but a potential crucial consideration is the danger of accidents or misuse. There might be another crucial consideration that it’s better to have the technology developed internationally and in the open, rather than have advances made by rogue states.

There are probably many crucial considerations that are either unknown or unacknowledged, especially in areas that we haven’t thought about for very long.


Cluelessness

The idea of cluelessness is that we are extremely uncertain about the impact of every action. For example, making a car stop as you cross the street might affect a conception later that day, and might make the difference between the birth of a future Gandhi or Hitler later on. (Note that many non-consequentialist moral systems seem even more prone to cluelessness worries – William MacAskill points this out in this paper, and argues for it more informally here.)

I’m not sure I fully understand the concerns. I’m especially confused about what the practical consequences of cluelessness should be on our decision-making. Even if we’re mostly clueless about the consequences of our actions, we should base them on the small amount of information we do have. However, at the very least it’s worth keeping in mind just how big uncertainty over consequences can be, and there are a bunch of philosophy paper topics here.

For more on cluelessness, see for example:


Reality is underpowered

Imagine we resolve all of our uncertainties over moral philosophy, iron out the philosophical questions posed by cluelessness, confidently identify Cause X, avoid biases, find all crucial considerations, and all that remains is the relatively down-to-earth work of figuring out which interventions are most effective. You might think this is simple: run a bunch of randomised controlled trials (RCTs) on different interventions, publish the papers, and maybe wait for a meta-analysis to combine the results of all relevant papers before concluding that the matter is solved.

Unfortunately, it’s often the case that reality is underpowered (in the statistical sense): we can’t run the experiments or collect the data that we’d need to answer our questions.

To take an extreme example, there are many different factors that affect a country’s development. To really settle the issue, we might make groups of, say, a dozen countries each, give them different amounts of the development factors (holding everything else fairly constant), watch them develop over 100 years, run a statistical analysis of the outcomes, and then draw conclusions about how much the factors matter. But try finding hundreds of identical countries with persuadable national leaders (and at least one country must have a science ethics board that lets this study go forwards).

To make a metaphor with a different sort of power: the answers to our questions (on what effects are the most important in driving some phenomenon, or which intervention is the most effective) exist, sharp and clear, but the telescopes with which we try to see them aren’t good enough. The best we can do is interpret the smudges we do see, inferring as much as we can without the brute force of an RCT.

This is an obvious point, but an important one to keep in mind to temper the rush to say we can answer everything if only we run the right study.


Conclusions?

All this uncertainty might seem to imply two conclusions. I support one of them but not the other.

The first conclusion is that the goal of doing good is complicated and difficult (as is the subgoal of having accurate beliefs about the world). This is true, and important to remember. It is tempting to forget analysis and fall back on feelings of righteousness, or to switch to easier questions like “what feels right?” or “what does society say is right?”

The second conclusion is that this uncertainty means we should try less. This is wrong. Uncertainties may rightly redirect efforts towards more research, and reducing key uncertainties is probably one of the best things we can do, but there’s no reason why they should make us reduce our efforts.

Uncertainty and confusion are properties of minds, not reality; they exist on the map, not the territory. To every well-formed question there is an answer. We need only find it.

 

Next post

2020-07-25

EA ideas 2: expected value and risk neutrality

2.6k words (9 minutes)
 
Posts in this series:
PREVIOUS | NEXT


The expected value (EV) of an event / choice / random variable is the sum, over all possible outcomes, of {value of outcome} times {probability of that outcome} (if all outcomes are equally likely, it is the average; if they’re not, it’s the probability-weighted average).

In general, a rational agent makes decisions that maximise the expected value of the things they care about. However, EV reasoning involves more subtleties than its mathematical simplicity suggests, in both the real world and in thought experiments.

Is a 50% chance of 1000€ exactly as good as a certain gain of 500€ (that is, are we risk-neutral?), or a 50% chance of 2000€ with a 50% chance of a 1000€ loss instead?

Not necessarily. A bunch of research (and common sense) says people put decreasing value on an additional unit of money: the thousandth euro is worth more than the ten-thousandth. For example, average happiness scales roughly logarithmically with per-capita GDP. The thing to maximise in a monetary tradeoff is not the money, but the value you place on money; with a logarithmic relationship, the diminishing returns mean that more certain bets are better than naive EV-of-money reasoning implies. A related reason is that people weight losses more than gains, which makes the third case look worse than the first even if you don’t assume a logarithmic money->value function.

However, a (selfish) rational agent will still maximise EV in such decisions – not of money, but of what they get from it.

(If you’re not selfish and live in a world where money can be transferred easily, the marginal benefit curve of efficiently targeted donations is essentially flat for a very long time – a single person will hit quickly diminishing returns after getting some amount of money, but there are enough poor people in the world that enormous resources are needed before you need to worry about everyone reaching the point of very low marginal benefit from more money. To fix the old saying, albeit with some hit to its catchiness: “money can buy happiness only (roughly) logarithmically for yourself, but (almost) linearly in the world at large, given efficient targeting”.)

In some cases, we don’t need to worry about wonky thing->value functions. Imagine the three scenarios above, but instead of euros we have lives. Each life has the same value; there’s no reasonable argument for the thousandth life being worth less than the first. Simple EV reasoning is the right tool.


Why expected value?

This conclusion easily invites a certain hesitation. Any decision involving hundreds of lives is a momentous one; how can we be sure of exactly the right way to value these decisions, even in simplified thought experiments? What’s so great about EV?

A strong argument is that maximising EV is the strategy that leads to the greatest good over many decisions. In a single decision, a risky but EV-maximising choice can backfire – you might take a 50-50 bet of saving 1000 lives and lose, in which case you’ll have done much worse than picking an option of certainly saving 400. However, it’s a mathematical fact that given enough such choices, the actual average value will tend towards the EV. So maximising EV is what results in the most value in the long run.

You might argue that we’re not often met with dozens of similar momentous decisions. Say that we’re reasonably confident the same choice will never pop up again, and certainly not many times; doesn’t the above argument no longer apply? Take a slightly broader view though, and consider which strategy gets you the most value across all decisions you make (of which there will realistically be many, even if no single decision occurs twice): the answer is still EV maximisation. We could go on to construct crazier thought experiments – toy universes in which only one decision ever occurs, for example – and then the argument really begins to break down (though you might try to save it by some wild scheme of imagining many hypothetical agents faced with the same choice and consider a Kantian / rule-utilitarian principle of deciding by answering the question of which strategy would be right if it were the one adopted across all countless hypothetical instances of this decision).

There are other arguments too. Imagine 1000 people are about to die of a disease, and you have to decide between a cure that will certainly cure 400 versus an experimental one that will either cure everyone or save no-one. Imagine you are one of these people. In the first scenario, you have a 40% chance of living; in the second, a 50% chance. Which would you prefer?

On a more mathematical level, von Neumann (an all-around polymath) and Morgenstern (co-founder of game theory with von Neumann) have proved that under fairly basic assumptions of what is rational behaviour, a rational agent acts as if they’re maximising the EV of some preference function.


Problems with EV

Diabolical philosophers have managed to dream up many challenges for EV reasoning. For example, imagine there’s two dollars on the table. You toss a coin; if it’s heads you take the money on the table, if it’s tails the money on the table doubles and you toss again. You have a 1/2 chance of winning 2 dollars, 1/4 chance of winning 4, 1/8 chance of winning 8, and so on, for a total EV of 1/2 x 2 + 1/4 x 4 + … = 1 + 1 + … . The sequence diverges to infinity.

Imagine a choice: one game of the “St. Petersburg lottery” described above, or a million dollars. You’d be crazy not to pick the latter.

Is this a challenge to the principle of maximising EV? Not in our universe. We know that whatever casino we’re playing at can’t have an infinite amount of money, so we’re wise to intuitively reject the St. Petersburg lottery. (This section on Wikipedia has a very nice demonstration of why, even if the casino is backed by Bill Gates’s net worth, the EV of the St. Petersburg game is less than $40.)

The St. Petersburg lottery isn’t the weirdest EV paradox by half, though. In the Pasadena game, the EV is undefined (see the link for a definition, analysis, and an argument that such scenarios are points against EV-only decision-making). Nick Bostrom writes about the problems of consequentialist ethics in an infinite universe (or a universe that has a finite probability of being infinite) here.

There’s also the classic: Pascal’s wager, the idea that even if the probability of god existing is extremely low, the benefits (an eternity in heaven) are great enough that you should seek to believe in god and live a life of Christian virtue.

Unlike even Bostrom’s infinite ethics, Pascal’s wager is straightforwardly silly. We have no reason to privilege the hypothesis of a Christian god over the hypothesis – equally probable given the evidence we have – that there’s a god who punishes us exactly for what the Christian god rewards us for, or that god is a chicken and condemns all chicken-eaters to an eternity of hell. So even if you accept the mathematically dubious multiplication of infinities, Pascal’s wager doesn’t let you make an informed decision one way or another.

However, the general format of Pascal’s wager – big values multiplied by small probabilities – is the cause of much of EV-related craziness, and dealing with such situations is a good example of how naive EV reasoning can go wrong. The more general case is often referred to as Pascal’s mugging, and exemplified by the scenario (see link) where a mugger threatens to torture an astronomical amount of people unless you give them a small amount of money.


Tempering EV extremeness with Bayesian updating

Something similar to Pascal’s mugging easily happens if you calculate EVs by multiplying together very rough guesses involving small probabilities and huge outcomes.

The best and most general approach to these sorts of issues is laid out here.

The key insight is to remember two things. First, every estimate is a probability distribution: if you measure a nail or estimate the effectiveness of a charity, the result isn’t just your best-guess value, but also the uncertainty surrounding it. Second, Bayesian updating is how you change your estimates when given new evidence (and hence you should pay attention to your prior: the estimate you have before getting the new information).

Using some maths detailed here, it can be shown that if your prior and measurement both follow normal distributions, then your new (Bayesian) estimate will be another normal distribution, with a mean (=expected value) that is an average of the prior and measurement means, weighted by the inverse variance of the two distributions. (Note that the link does it with log-normal distributions, but the result is the same; just switch between variables and their logarithms.)

Here’s an interactive graph that lets you visualise this.

The results are pretty intuitive. Let’s say our prior for the effectiveness of some intervention has a mean of zero. If we take a measurement with low variance, our updated estimate probability distribution will shift most of the way towards our new measurement, and its variance will decrease (it will become narrower):

Red is the probability distribution of our prior estimate. Green is our measurement. Black is our new belief, after a Bayesian update of our prior with the measurement. Dotted lines show the EV (=average, since the distributions are symmetrical) for each probability distribution. You can imagine the x-axis as either a linear or log scale.

If the same measurement has greater variance, our estimates shift less:


And if we have a very imprecise measurement – for example, we’ve multiplied a bunch of rough guesses together – the estimate barely shifts even if the estimate is high:


Of course, we can argue about what our priors should be – perhaps, for many of the hypothetical scenarios with potentially massive benefits (for instance concerning potential space colonisation in the future), the variance of our prior should be very large, in which case even highly uncertain guesses will shift our best-guess EV a lot. But the overall point still stands: if you go to your calculator, punch in some numbers, and conclude you’ve discovered something massively more important than anything else, it’s time to think very carefully about how much you can really conclude.

Overall, I think this is a good example of how a bit of maths can knock off quite a few teeth from a philosophical problem.

(Here’s a link to a wider look at pitfalls of overly simple EV reasoning with a different framing, by the same author as this earlier link. And here is another exploration of the special considerations involved with low-probability, high-stakes risks.)


Risk neutrality

An implication of EV maximisation as a decision framework is risk neutrality: when you’ve measured things in units of what you actually care about (e.g. converting money to the value it has for you as discussed above), you should be neutral about the choice between 10% chance of 10 value units and 100% chance of 1, and you really should prefer a 10% chance of 11 “value units” over a 100% chance of 1 “value unit”, or a 50-50 bet between losing 10 and gaining 20 over a certain gain of 14.

This is not an intuitive conclusion, but I think we can be fairly confident in its correctness. Not only do we have robust theoretical reasons for using EV, but we can point to specific bugs in our brains that makes us balk at risk-neutrality: biases like scope neglect, which makes humans underestimate the difference between big and small effects, or loss aversion, which makes losses more salient than gains, or a preference for certainty.

$$$%%IF YOU SEE DOLLAR SIGNS IN THE NEXT SECTION, EQUATION RENDERING VIA MATHJAX IS NOT WORKING IN YOUR BROWSER$$$

Stochastic dominance (an aside)

Risk neutrality is not necessarily specific to EV maximisation. There’s a far more lenient, though also far more incomplete, principle of rational decision making that goes under the clumsy name of “stochastic dominance”: given options $$A$$ and $$B$$, if the probability of a payoff of $$X$$ or greater is more under option $$A$$ than option $$B$$ for all values of $$X$$, then $$A$$ “stochastically dominates” option B and should be preferred. It’s very hard to argue against stochastic dominance.

Consider a risky and a safe bet; to be precise, call them option $$A$$, with a small probability $$p$$ of a large payoff $$L$$, and option $$B$$, with a certain small payoff $$S$$. Assume that $$pL > S$$, so EV maximising says to take option $$A$$. However, we don’t have stochastic dominance: the probability of getting a small amount of value $$v$$ ($$v < S$$) is greater with $$B$$ than $$A$$, whereas the probability of getting a large amount of value ($$S < v < L$$) is greater with option $$A$$.

The insight of this paper (summarised here) is that if we care about the total amount of value in the universe, are sufficiently uncertain about this total amount, and make some assumptions about its distribution, then stochastic dominance alone implies a high level of risk neutrality.

The argument goes as follows: we have some estimate of the probability distribution $$U$$ of value that might exist in the universe. We care about the entire universe, not just the local effects of our decision, so what we consider is $$A + U$$ and $$B + U$$ rather than $$A$$ and $$B$$. Now consider an amount of value $$v$$. The probability that $$A + U$$ exceeds $$v$$ is the probability that $$U > v$$, plus the probability that $$(v - L) < U < v$$ and $$A$$ pays off $$L$$ (we called this probability $$p$$ earlier). The probability that $$B + U$$ exceeds $$v$$ is the probability that $$U > v - S$$.

Is the first probability greater? This depends on the shape of the distribution of $$U$$ (to be precise, we’re asking whether $$P(U > v) + p P(v - L < U < v) > P(U > v - S)$$, which clearly depends on $$U$$). If you do a bunch of maths (which is present in the paper linked above; I haven’t looked through it), it turns out that this is true for all $$v$$ – and hence we have stochastic dominance of $$A$$ over $$B$$ – if the distribution of $$U$$ is wide enough and has a fat tail (i.e. trails off slowly as $$v$$ increases).

What’s especially neat is that this automatically excludes Pascal’s mugging. The smaller the probability $$p$$ of our payoff is, the more stringent the criteria get: we need a wider and wider distribution of $$U$$ before $$A$$ stochastically dominates $$B$$, and at some point even the most stringent Pascalian must admit $$U$$ can’t plausibly have that wide of a distribution.

It’s far from clear what $$U$$’s shape is, and hence how strong this reasoning is (see the links above for that). However, it is a good example of how easily benign background assumptions introduce risk neutrality into the problem of rational choice.


Implications of risk neutrality: hits-based giving

What does risk neutrality imply about real-world altruism? In short, that we should be willing to take risks.

A good overview of these considerations is given in this article. The key point:

[W]e suspect that, in fact, much of the best philanthropy is likely to fail.

For example, GiveWell thinks that Deworm the World Initiative probably has low impact, but still recommends them as one of their top charities because there’s a chance of massive impacts.

Hits-based giving comes with its own share of problems. As the article linked above notes, it can provide a cover for arrogance and make it harder to be open about decision-making. However, just as high-risk high-reward projects make up a disproportionate share of successes in scientific research and entrepreneurship, we shouldn’t be surprised if the bulk of returns on charity comes from a small number of risky bets.

 

Next post

EA ideas 1: rigour and opportunity in charity

2.2k words (8 minutes)

Effective altruism (EA) is about trying to carefully reason how to do the most good. On the practical side, EA has inspired the donation of hundreds of millions of dollars to impactful charities, and lead to many new organisations focused on important causes. On the theoretical side, it has lead to rigorous and precise thought on ethics and how to apply it in the real world.

The intellectual work that has come out of EA is valuable, especially in two ways.

First, much EA work is exceptional in the breadth and weight of the matters it considers. It is interdisciplinary, including everything from meta-ethics to interpreting studies on the effectiveness of vaccination programs in developing countries. Because of its motivation – finding and exploring the most important problems – it zeros in on the weightiest issues in any particular area. EA work is a goldmine of interesting writing, particularly if you find yourself drawn in a discipline-agnostic way to all the biggest questions.

Second, EA often has a scientific precision of argument that is often missing from discussions on abstract things (e.g. meta-ethics) or emotionally charged issues (e.g. saving lives).

This post explains the motivations behind EA, and has a table of contents for this post series.


Altruism, impartial welfarist good, and cause neutrality

I will have more to say in a later post about specific philosophical issues in defining what is moral. For now I will hope that the idea of an impartial welfare-oriented definition of good is sufficiently defensible that I will not be mauled to death by moral philosophers before that post (though if it doesn’t happen by then, it will certainly happen afterwards).

Impartial (in the sense of considering everyone fairly, and giving the same answer regardless of who’s doing the judging) and welfare-oriented (in the sense of valuing happiness, meaning, fulfilment of preferences, and the absence of suffering) good is an intuitive and fairly unobjectionable idea. Yet if we take it as a goal, it points towards a different idea of charity than the current norm.

Most charities are single-issue charities. This generally makes sense: better to have one organisation be really good at distributing malaria nets and one really good at advocating for taking nuclear weapons off high alert, than to have one organisation doing a mediocre job at both (malaria net delivery via ICBM?).

But the siloing of causes often goes further. If the effectiveness of an intervention is considered, it is often after choosing a cause area. To weigh cause areas against each other, to judge the needs of African children against, say, factory farmed pigs, seems like a faux pas at best, and a sin at worst (for a particularly incendiary tirade on the topic, see this article).

However, if we hold ourselves to an impartial welfarist idea of good, this judgement must be made. An artist might choose what to paint based on how they want to express themselves or on a sudden flash of inspiration. A would-be altruist refusing to weigh causes against each other and instead selecting them on the basis of passion or inspiration is acting like our artist. In the artist’s case it doesn’t matter, but the altruist, in doing so, implicitly values their own choice and/or self-expression over the good that their actions might do. This is not altruism by our definition of good.

Of course, people differ in their knowledge and talents, and these tend to align with inspiration. In the real world, it may well be that your greater ability, drive, and/or knowledge in one area outweighs the greater efficiency at which results convert to goodness in some other area. We will also see arguments for not placing all our bets on the same cause, and explore the enormous uncertainties that come in trying to compare causes. But the idea of cause-neutrality – that causes are comparable, and that making these comparisons is an important part of the job of any would-be altruist – remains.


Effectiveness

Focusing on the idea of impartial welfarist good also makes it clear that, in trying to do good, we should focus on the good our actions result in. This may seem like an obvious statement, but it is not true of much charitable work.

For example, we tend to emphasise the sacrifices of the donor over the benefits of the recipients. Consider old tales of people like Francis of Assisi. Their claim to virtue (and sainthood) comes from giving away all their possessions, but the question of how much good this did to the beggars doesn’t come up. This attitude continues in the many modern charity evaluators that focus on metrics like percentage of money spent on overhead costs. Paying big salaries to recruit the best management and administration may genuinely be a cost-effective way of increasing the total good done, but it conflicts with our stereotype of self-sacrificing do-gooders. Of course, there is virtue in selfless sacrifice, but we should remember that the goal of charity is to make recipients better off, not to rank donors.

As with many things humans do, acts of charity often aren't based on rational calculation. Some consider this a good thing: altruistic acts should come from hearts, not spreadsheets. This is wrong – if you care about impartial welfarist good.

It is a fact about our world that good charity is hard, and that charities have vast differences in cost-effectiveness. When one charity results in ten or a hundred times more healthy years of life per dollar spent than another, boring details of statistical effectiveness become important moral facts. (This is true not just of charities, but most kinds of projects that might impact many people – government policy, activism, and so on.)

When the difference in effectiveness between different interventions is often greater than the difference to doing nothing at all, and when these differences are often measured in lives, effectiveness considerations are critical in any attempt to do good.

There is a role for simple, comforting altruism, but this role isn’t making big decisions over how to benefit others. These decisions deserve more than goodwill. They deserve to be made right.


Opportunity

Debates over charitable giving often centre on questions of moral duty and obligation (a good example is Famine, Affluence, and Morality, Peter Singer’s classic paper that laid some of the foundations of what later became EA).

Another framing is to think of it as an opportunity. To someone who cares about impartial welfarist good, altruistic acts are not a burden but an opportunity to achieve valuable things. In particular, there are many reasons to think that we (as in developed-world humans of the early 21st century) have an exceptionally large opportunity to do good.

First, our values are better than those of people in preceding eras. This statement implies many philosophically contentious points, but for the time being I will not defend them, instead appealing to what I hope to be a common sense conviction that human morality isn’t nearly relative enough that it is impossible to differentiate modern secular humanist values from values that support war, slavery, and boundaries on personhood that exclude most people.

(Of course, this statement also suggests that our current moral views are far from perfect too. This is important, very likely true, and will be discussed at length in future posts. The fact that this is increasingly recognised is hopefully a hint that we are at least on the right track.)

Second, we have more resources than people in previous eras. There is also large variation in global income, meaning that if you happen to live in a rich country, you can help many others for cheap. A 2-adult, 1-child UK household with a total income of £30,000 is in the top 10% of the world income distribution and 7 times richer than the median global household.

Third, knowledge on what is effective has increased and technology make it easier to apply this knowledge. Today GiveWell’s thorough charity research can multiply the impact of giving. Twenty years ago, there was no GiveWell. Two hundred years ago, donation guidance, if it existed, might have consisted of the church telling you to donate to them so they can convert people and push their social values.

Fourthly, we may have an unprecedented ability to affect where civilisation is headed (for thoughts on this topic, see for example this link). The steepness of technical advancement increases the variance of possible future outcomes: in the next few decades we might nuke each other or engineer a pandemic – or we can set ourselves on a trajectory towards becoming a sustainable civilisation with billions of happy inhabitants that lasts until the stars burn down. Past eras didn’t have similar power, and if the future goes well humanity will no longer be as vulnerable to catastrophe as we are today, so people living roughly today might have exceptional leverage.


Common EA cause areas

The cause areas most frequently seen as important, and most specific to EA relative to what other charities focus on, are:

  • Global poverty, because the developing world is big, poor, and has many tractable problems with well-researched solutions.
  • Animal welfare, because it is largely ignored, and potentially huge in scope (depending on how much animal lives are valued).
  • Existential risk: focusing on avoiding human extinction or other irrevocable civilisational collapses, because new technologies (AI and biotech in particular) make them scarily plausible. (Sometimes this is motivated even more strongly by long-termism: specifically caring about the overwhelming number of happy future lives that may come to exist over the long-term future if we don't mess things up).

These are far from the only cause areas discussed in EA. Many EA-affiliated people argue either against some of the above, for the overwhelming importance of one relative to the other, or for entirely different causes.


Effective altruism in practice

In practice, EA can seem weird and theoretical.

The main reason for EA weirdness is that it casts a wide net. Everyone agrees that international peacekeeping is an important project, and also a serious one: it doesn’t get much more serious than world leaders intervening to get men with big guns to have big talks about their big disputes. On the other hand, the colonisation of space is important, but seems to have very little gravitas indeed; it’s something out of a science fiction novel. However, just as it’s a brute fact about the world that there are lots of violent people with big guns, it’s also a brute fact that space is big; both of these facts should be taken seriously when considering the long-run future. There might be a clear line between sci-fi and current affairs in a bookshop, but reality doesn't care about genre.

More generally, it’s important to keep in mind that every moral advance started out as a weird idea (for example, it was once considered crazy to suggest that women should get to vote).

Parts of EA are very theoretical. This, too, is by design. Future posts will show many cases where which way we resolve a very abstract issue has a big impact on what the right practical action is – and in many of these cases it is unclear what the right resolution is. Finding out clearly matters.

If EA seems too theoretical or mathematical to you, consider two points. First, whatever the field, doing complex things in the real world tends to involve (or be built on) theoretical heavy lifting. Second, most charity efforts don’t pay much attention to theoretical issues; EA is at very least a helpful counterweight, and likely to uncover missed opportunities.

Whenever the goal is to do good, it is easy to be overwhelmed by feelings of righteousness and forget theoretical scruples. Unfortunately we don’t live in the simple world where what feels right is the same as what is right.

The core of effective altruism is not any particular moral theory or cause area, but a conviction that doing good is both important and difficult, and hence worthy of thought.


This post series:

  1. Rigour and opportunity in charity: this post.
  2. Expected value and risk neutrality: a rational agent maximises the expected value of what it cares about. Expected value reasoning is not free of problems, but, outside extreme thought experiments and applied carefully, it clears most of them, including "Pascal's mugging" (high-stakes, low-probability situations). Expected value reasoning implies risk neutrality. The most effective charity may often be a risky one, and gains from giving may be dominated by a few risky bets.
  3. Uncertainty: we are uncertain about both what is right and what is true (being mindful of the difference is often important). Moral uncertainty raises the question of how we should act when we have credence in more than one moral theory. Uncertainty about truth has many sources, including ones broader than uncertainty about specific facts, such as our biases or the difficulty of confirming some facts. These uncertainties suggest we are unaware of huge problems and opportunities.
  4. Utilitarianism: while not a necessary part of EA thinking, utilitarianism is the most successful description of the core of human ethics so far. In principle (if not practice, due to the complexity of defining utility), it is capable of deciding every moral question, an important property for a moral system. Our moral progress over the past few centuries can be summarised as a transition to more utilitarian morality.


(More coming)

2020-05-09

Short reviews: fiction

Cryptonomicon (Neal Stephenson)


Cryptonomicon is a hard novel to summarise. It is about World War II code-breakers and 1990s tech entrepreneurs, but also manages to concern itself with most other things as well.

I first read Cryptonomicon over two years ago. However, it is a massive book, and since it happens in the same universe as The Baroque Cycle, I assumed reading it again would reveal many new things. I was not wrong.

Neal Stephenson has a humorously extravagant (baroque?) writing style that is always entertaining to read, but in Cryptonomicon it is taken to an extreme. Stephenson turns mundane activities like writing a business plan, eating cereal, taking a car ride in the Philippines, and visiting a dentist into lengthy but hilarious tangents. Do they contribute to the plot? Who cares!

As this is a Neal Stephenson novel, certain vices will also be present. A printed version of the book, dropped from a bomber, would punch a hole through the deck of a Japanese warship. The plot meanders to an extent that puts most rivers to shame. And some things are just plain weird.

But overall, Cryptonomicon makes for a great read for anyone with the time to spare, and an interest in codebreaking, history, war, mathematics, the Internet, the financial industry, or technology.


Exhalation (Ted Chiang)


“Exhalation”, this short story collection’s titular work, is the greatest short story I have ever read. (You may read it online for free – and legally, as far as I can tell – here). The careful setup builds to a beautiful and intuitive analogy that make the philosophical points at the end hit hard.

Based on the strengths of “Exhalation” (the short story), I bought Exhalation (the short story collection). None of the other stories surpass “Exhalation”, though they are mostly good and sometimes excellent.

Reading a Ted Chiang story is like watching an eerily intricate machine in action, or listening to a Bach fugue: the feeling is one of orderliness and precision combined with an almost casual ease. The premise of each story is fundamentally a thought experiment; a “what-if” question knocks down one domino and the story follows its consequences all the way down the chain. Nothing is wasted or in excess, and the beats of the pacing come like metronome beats. In the best of the stories, these beats are almost undetectable at first, gradually building up into dawning revalation as the pieces fall together and the story reaches its climax.

Aside from “Exhalation”, there are two stories that stand out.

“The Truth of Fact, the Truth of Feeling” is a thoughtful exploration of the effect of the medium on what is seen as true (a topic that Neil Postman would feel right at home with). The story cleverly parallels the story of a person in an African village being introduced to literacy in the past, and a person in the future grappling with the consequences of technology that records everything people see. In a world of cautionary tales about technology stealing our identities, destroying our communities, or letting dinosaurs loose in the park, Chiang’s take on this issue is surprisingly forward-looking.

In “Omphalos” (an Ancient Greek word for “navel”, as in the expression “navel of the world”), the what-if question is: what if creationism were true, but humanity was a side-effect rather than the pinnacle of creation? The story is told in the form of prayers to god. Chiang takes the reader on a tour of what the scientific facts of this world look like: old trees with no growth rings in the middle, mummified people without navels, and so on, until finally a physics discovery, while confirming without doubt the existence of miracles, also leads inevitably to the conclusion that we are not the purpose of god’s creation. All this takes place in parallel with the emotional arc of the central character, which is told in a sympathetic and realistic manner.


Summerland (Hannu Rajaniemi)


The year is 1938. The Spanish Civil War rages on, Europe braces for war, Queen Victoria reigns from the afterlife, and the Soviets are merging souls into a godlike overmind, starting with Lenin’s.

In the alternative universe of Summerland, Marconi discovered more than he bargained for when working with radio transmission, and soon enough ectotanks and other supernatural weaponry were being deployed in World War I. Since then much of early-1900s spiritualism has been proven right.

Most significant is Summerland, an afterlife where souls can lodge themselves (provided they have a ticket) and even interact to a limited extent with the living.

In terms of plot, Summerland is a fairly straightforward spy novel. This is executed well (though my judgement may not be representative of those who know more about spy novels), but the premise is what makes Summerland special.

(Rajaniemi is best known for his far-future science fiction trilogy, which starts with The Quantum Thief; this is also recommended.)


The Curse of Chalion (Lois McMaster Bujold)


At the time of writing, the “Reception” section of the Wikipedia page for this book tells me nothing but “The book has received a number of reviews”.

This rather underwhelming (though doubtlessly accurate) statement does not do the book justice. The Curse of Chalion shines not through outstanding excellence in one respect, but rather by bringing a variety of good elements together: characters that feel like real people, an atmospheric setting, and above all a hard-to-pin-down tastefulness where nothing is in excess.

If I had to critique something, some of the turning points in the plot are rather deus ex machina. However, overall the book is a great example of fantasy built on literary merits rather than genre props, and makes for a very enjoyable story to get lost in.

(The introduction of the Wikipedia article, however, is little but a list of all the awards the book has won.)


Unsong (Scott Alexander)


In the beginning God created the heavens and the Earth. For a while, everything was fine. Then Thamiel, the left hand of God, appears in the centre of the Earth and corrupts a third of the angelic host. A war begins between the angels and demons, in which the demons gain the upper hand. Their victory is averted only when the mathematically talented archangel Uriel initiates his backup plan: switching the world from running on divine light to running on mathematical laws. Angels and demons both are reduced to mere metaphors, and the world is saved.

Saved, that is, until humans get very good at harnessing those laws and send Apollo 8 on a trip around the moon in 1968. Unfortunately all space beyond the moon is simply an illusion to make the universe seem consistent with the physics that now reigns on Earth. Instead of looping around the moon, Apollo 8 crashes into the edge of the world, damaging the delicate celestial machinery that Uriel put into place to maintain his conversion.

Various glitches start to show up in the working of the world. Angels and demons begin returning: Uriel reappears in a hurricane in the Mexican Gulf, from where he plays the role of an overworked sysadmin issuing a constant stream of patches to prevent physics from crashing, while demons spring up from Lake Baikal and start invading Russia.

The backstory of Unsong, told in various short excerpts throughout the book, continues with a very clever account of how the world reacts to this turn of events. Cold War politicking continues; for example, at one point Henry Kissinger successfully convinces President Nixon to ally with Hell in order to keep the Russians in check.

The main plot line begins in 2017. In this universe, kabbalah works. In particular, it makes possible the discovery of Names of God – words which have magical powers, but whose distribution is controlled by strict copyright laws. The main character, Aaron Smith-Teller, is a gifted kabbalist, but works a low-paid job helping a company find Names: he reads potential Names off a computer screen all day long, and if he finds a Name, gives it over to the company. The process cannot be automated because computers lack a soul and hence can’t detect which words are Names, necessitating this sort of low-skill work.

Unsong is remarkable not just for its crazy premise, but for the consistency and ruthlessness of its internal logic (which characters do not fail to exploit). Imagine you stumble across a Name that grants souls to inanimate objects. What do you do? That’s obvious: use it on a computer, have it start searching for new Names at superhuman speed, sell the Names for profit, buy more computers, and continue in this vein until you have magic powers beyond your dreams and can take over the world. If the Bible is literally true, what is the overriding moral priority? Simple: end the existence of hell; countless people suffering eternal torture for vague reasons cannot be part of a just universe.

The central question that many of Unsong’s characters grapple with is the problem of theodicy: why would a good god create a world with so much evil? This question does not have direct relevance to our own world, but it leads to other interesting questions (as well as giving the author a chance to flaunt their ingenuity; the book actually has a plausible answer). Together with characters who are often both idealistic and ruthless – I’m particularly fond of Jalaketu West, AKA “The Comet King” – this makes the book a good exploration of many moral themes.

Be warned, though: Unsong is about a universe where words, rather than equations, are the building blocks of reality. This leads to a lot of perverse verbal ingenuity, including more puns than can possibly be healthy. If you don’t want to read about characters who protest at the World’s Fair by waving signs saying “No it isn’t!”, or how atheists also include a leviathan in their mythology by calling the whole world a giant fluke, stay away.

Unsong was published online, chapter by chapter. This means two things, one bad and one good. First, it is a bit less polished than a published novel might be. Second, you can read it for free online.

Short reviews: non-fiction

The Feynman Lectures on Physics (Richard Feynman)


The Feynman Lectures on Physics (FLOP) is an incredible resource on basic physics. Feynman has an inimitable style: he is always clear, never the slightest bit pretentious, and has an eerie ability to cut through tangles of models, assumptions, and equations to get at the fundamental point. Often you can feel Feynman’s infectious enthusiasm through the page.

There are some issues with trying to learn physics from FLOP. There are no exercises, so you cannot test your understanding very easily.

Another reason is that the easy flow and elegant arguments make it less structured. If a typical textbook is like taking an official tour through a city, methodically exploring everything there is to see, the general feel of FLOP is more of chasing after a boundlessly enthusiastic tour guide as he zips from place to place using various shortcuts, leaving you with the nagging feeling that, while it was certainly very fun, you might not be able to retrace the route afterwards.

Feynman has a remarkable ability to introduce just enough background to pull off some proof or argument. This makes for some brilliant arguments that are fun to follow, but, particularly when it comes to mathematical tricks, left me with the feeling that if I didn’t have more background than Feynman introduces, I would be lost.

(Given a solid understanding of calculus and complex numbers, there are no great leaps required to follow the mathematics in volume 1. Volume 2 deals mainly with electromagnetism, which relies on vector calculus; at the time I was reading it, I didn’t have a solid grasp on that and this made parts difficult to follow. I still haven’t had a chance to read through everything in the second half of volume 2, and have read nothing from volume 3 and so cannot comment on it.)

Overall, FLOP is a brilliant resource. Perhaps it works best as a reference volume; there are many arguments that I do not remember off the top of my head, but which I remember are presented with extreme clarity in FLOP. Of course, without reading through at least once, how will you know what’s in it?


The Character of Physical Law (Richard Feynman)


The Character of Physical Law is based on another series of lectures Feynman that gave. It attempts to squeeze out maximum understanding and reflection about what physics is about from a minimum of abstruse maths.

It succeeds.

The focus is not on what the laws themselves are, but rather on the common themes in many of them: conservation principles, symmetry, and, of course, maths. The combination of clear explanation and reflection without pretence or overstretched philosophy is unbeatable.

If you read one popular physics book, make it this one. It is as close to the heart of physics as you can get without heavy mathematics.

If you are serious about physics, you will of course have to dive into the maths. But read this book anyways.


Origin Story: A Big History of Everything (David Christian)


I have rarely agreed with the purpose of a book as much as I do with the purpose of Origin Story.

The idea is that an origin story explaining where the world came from and what humanity’s place in it is has been a foundational part of most human cultures in history. Ironically, just as our civilisation is now figuring out the real answers to these questions, a collective understanding of our “origin story” is missing. This is the gap that Origin Story – and the field of big history in general – aims to plug.


The Great Leveler: Violence and the History of Inequality (Walter Scheidel)


Inequality is a trendy topic. Coherent insights into its history and how to quantify it are notably less trendy.

The Great Leveler provides both in spades. Optimism is in somewhat shorter supply. Scheidel identifies “Four Horsemen of Leveling” that have historically driven large decreases in inequality: total war, violent revolution, state collapse, and pandemics. If, as Scheidel cautions, welfare democracies probably won’t buck this trend, it looks like coronavirus is our only chance.


The Strategy of Conflict (Thomas Schelling)


You are in a car, driving directly towards another car. You will soon crash. The rules of the game are simple: the first one to swerve loses. How do you win? You close your eyes, throw away the steering wheel - basically, anything that both removes your ability to act and credibly signals this to your opponent.

The Strategy of Conflict is all about delightfully - and sometimes scarily - counterintuitive problems in game theory, in particular conflict of the nuclear sort. The general theme is that reducing your ability to make choices and committing to irrational acts can be the most powerful tools at your disposal. If you can commit to something in advance, regardless of whether it is in your rational interest to do it when the time comes, you can change the payoffs for your opponent, and hence possibly change what they calculate their best action to be.


The Doomsday Machine (Daniel Ellsberg)


My brief notes on this book snowballed into a full review, which you can find here. If you’re getting tired of the coronavirus pandemic, why not put things into perspective by reading about nuclear war?


Founders at Work (Jessica Livingston)


Founders at Work is a collection of interviews with startup founders. The book doesn’t try to be anything fancy, or make any deep conclusions about how the technology industry works. Its main value – and this is not a trivial thing – is as a source of “virtual experience” that you can download into your brain. Reading dozens of founders reflecting on their experiences with the guidance of a knowledgeable interviewer is the second-best thing to having that experience yourself.
Perhaps the two most basic and recurring themes are:
  1. In a (good) startup, everything is as barebones, minimalist, and plain as possible. The working place might be the stereotypical garage, someone’s apartment, or there might not even be one. Money is saved in endlessly creative ways. At most, you occasionally might have to dress up or pretend to have a normal office to impress investors. This theme is summarised by a story told in the introduction: some people tried to figure out how to make a sports car go faster, and eventually realised the key was to remove everything that makes it look like it goes fast.
  2. In the early stages, no one has any idea what they’re doing


Security Engineering (Ross Anderson)


The lecturer for my current software & security engineering course is publishing the third edition of his security engineering textbook online chapter by chapter (“like Dickens’ novels”, as he describes it). The textbook is extremely readable, and many of the case studies are both illuminating and funny. Read it here.

Be warned that most of the chapters will disappear from the website for several years after the book is published. However, they will return afterwards, and the same page linked above also has all the chapters from the second edition, which has already passed this period and is free online forever.

2020-04-23

Review: The Doomsday Machine

 Book: The Doomsday Machine: Confessions of a Nuclear War Plannerby Daniel Ellsberg (2012).
This review is 4.6k words (≈15 minutes).
 

Here’s what former RAND Corporation nuclear strategy analyst (and later, Pentagon Papers leaker) Daniel Ellsberg and his colleague thought about the movie Doctor Strangelove – a dark and brilliant comedy about accidental nuclear war – after watching it in 1964:
"We came out into the afternoon sunlight, dazed by the light and the film, both agreeing that what we had just seen was, essentially, a documentary."

Doctor Ellsberg, or: How I Learned to Start Worrying and Hate the Bomb

The age of mass slaughter of civilians as war strategy did not start with Hiroshima, but rather years before with British and, later, American bombing campaigns. No new moral or strategic choice was made in the decision to drop the atomic bombs on Japan; it was the natural outgrowth of the policies that had already incinerated Dresden and Tokyo.

Of course, nuclear technology meant an escalation of its scale. A single plane carrying an atomic bomb is more efficient at delivering mass death than a bomber fleet. Hydrogen weapons, in which the atomic bomb is a mere detonating cap for a fusion reaction, scale up the destructive power a thousandfold. Thanks to missiles that can strike anywhere on Earth within an hour and the insistence of many nuclear countries in keeping weapons on high alert, each nuclear power has a loaded gun trained at the civilian population of the others.

The perverse logic of this hostage situation leads to the sorts of insanities that make Ellsberg call Doctor Strangelove a documentary.

For example, the lack of any way to recall bombers was a true and deliberate part of the US nuclear response mechanism. The fictional horror scenario of unintentionally launched bombers continuing towards their targets while the rest of the world spends its final hours waiting powerlessly was at most fifteen minutes from becoming reality throughout the early Cold War.

(Thankfully the switch from bombers to faster missiles later removed this anxiety-inducing pre-Armageddon wait.)

Why? Presumably because a recall code could be stolen by the enemy and used to misdirect an attack. The logic of mutually assured destruction demands certain response without delay.
When the US Air Force was told to place electronic locks on Minuteman missiles to prevent unauthorised launch, they decided that the unlock code would always be set at 0000 0000, so that a launch would never be blocked because the code was missing (or because a nervous launch officer couldn’t punch in anything more complicated).

Delegating the authority to launch weapons is another way of ensuring launch readiness. If the president, vice president, and everyone else in the line of succession right down to the White House chef are nuked into oblivion by a surprise strike, it can’t interfere with the ability to retaliate, or else that’s exactly what the Soviets would immediately start planning to do. And so (as Ellsberg carefully investigated) Eisenhower discreetly gave the admiral of the US Indo-Pacific Command the right to start the nuclear war plan by his own initiative. Communication links across the Pacific can be unreliable, so the admiral further delegated launch authority down the chain. To Ellsberg it is unclear if even the president was aware of the further delegation, but perfectly clear that this is insanity: all it would take is a geopolitical crisis, some bad weather over Hawaii, and suddenly some over-eager general on a distant Pacific island thinks that nuclear war has broken out and it is their duty to join the fun.

This is not the end of bureaucratic madness. Ellsberg recounts his surprise after learning that the US had no war plan involving just the Soviet Union. Any nuclear attack would hit China as well. The admirals Ellsberg asked about this were incensed, leading Ellsberg to conclude:
"Thus, if the president gave an order to attack only Soviet targets, CINCPAC [US Indo-Pacific Command, now called USINDOPACOM] forces, having destroyed Vladivostok and a few other minor targets in eastern Russia, would essentially have to sit out the war as observers—“on the sidelines,” as they thought of it—during the big game."
This was something that the admirals thought intolerable.

It gets worse:
"[I]t had long been clear to me that if the highest authorities did give [an order that excluded China] it would be virtually impossible to implement that order quickly in the Pacific. That was true for technical as well as bureaucratic reasons. CINCPAC planners were working extremely hard, around the clock each year, just to produce one single plan for nuclear war against the Sino-Soviet bloc, and they simply didn’t have the ability to produce a second plan for war with the Soviet Union alone."
Why was it so difficult to create a nuclear war plan? A major reason was the enormous number of calculations needed to schedule the bombers so that they wouldn’t be swatted out of the sky by nukes dropped by other bombers:
"Plans specified that a particular explosion would go off at time-over-target, or TOT (for example, 117 minutes and 32 seconds after the Execute order), and then a nearby explosion would go off 2 minutes and 12 seconds later, and so forth. If everything went according to plan, no plane would be struck down by the explosion from a bomb dropped by another plane; no “fratricide” would occur."
Practical inconveniences, like the fact that not all planes would manage to get themselves in the air equally fast, or the existence of weather, were ignored.
'I pointed these two problems out to a planner once. 
“Yes, I’ve thought of these problems before,” he said. 
“Well, doesn’t that make you question the value of making all these calculations and plans?” 
“These men are risking their lives flying out there. We’ve got to do what we can to save their lives.” 
“But it doesn’t seem that this plan has any chance to save any lives at all. It would save lives only if the execution followed the plan down to the second, and there’s not even the remotest possibility of that happening.” 
“Well, we’re ordered to make these calculations, so that’s what we do.”'
No sane person can think to themselves “let’s plan to kill three hundred million Chinese peasants in order to protect the egos of a few admirals, and because someone wants to make really detailed spreadsheets”. A badly designed bureaucracy won’t even blink.
"How to describe that, other than insanity? Should the Pentagon officials and their subordinates have been institutionalized? But that was precisely the problem: they already were. Their institutions not only promoted this insanity, they demanded it. And still do. As do comparable institutions in Russia."

Cuban roulette

Ellsberg’s account of the Cuban missile crisis is particularly haunting. Both Kennedy and Khrushchev were eager to avoid war, more cautious than many of their advisors, and willing to step down the bravado even at steep political cost.

Yet at the peak of the crisis on October 27th, 1962, there were two occasions when nuclear war was averted by chance. The first occasion was when the captain of a Soviet submarine being hounded by American destroyers decided to launch a nuclear torpedo at the destroyers. On most submarines the agreement of the captain and political officer would have sufficed, but flotilla commander Vasili Arkhipov happened to be onboard this particular submarine, and had the authority to overrule the captain and the political officer.

Had Arkhipov been stationed on a different submarine:
"The source of [the explosion caused by the nuclear torpedo] would have been mysterious to other commanders in the [US] Navy and officials on the ExComm, since no submarines known to be in the region were believed to carry nuclear warheads. The clear implication on the cause of the nuclear destruction of this antisubmarine hunter-killer group would have been a medium-range missile from Cuba whose launch had not been detected. That is the event that President Kennedy had announced on October 22nd would lead to a full-scale nuclear attack on the Soviet Union."
Perhaps Kennedy would have decided to take back his redline, and maybe the conflict might have deescalated even then. But the odds would have been long.

The second time was above Siberia and the Bering Sea. An American U-2 spy plane had wandered off-course into Soviet airspace. MiGs were scrambled to intercept it (perhaps believing it to be a reconnaissance plane for a larger attack), and American F-102As scrambled in turn to intercept the MiGs before they could get to the U-2. The F-102As were armed only with nuclear air-to-air weapons, since they were meant to be used against Russian nuclear bomber formations.

Secretary of Defense Robert McNamara reportedly ran out of a Pentagon meeting hysterically yelling “this means war with the Soviet Union” upon hearing the news. Kennedy, however, was calmer:
"In a panic, [the chief of the Bureau of Intelligence and Research] rushed in to tell the president there was a U-2 over Russia being pursued by MiGs. Kennedy, very cool, responded from his rocking chair (as Hilsman reported) with an old Navy joke: 'There’s always some son-of-a-bitch who didn’t get the word.'"
Even assuming leaders who would rather lose face than commit genocide, their control over events is not perfect. Government bureaucracies, trigger-happy generals, and your generic sons-of-bitches who don’t get the message have a lot of inertia, which any central organising force will struggle to halt. Combined with the hair-trigger launch capability demanded by deterrence through mutually assured destruction, this means nuclear war cannot be removed from the realm of the possible.


’Tis but a scratch

How bad is nuclear war, really?

Ellsberg comes with a prepackaged answer: the nuclear weapons currently deployed by the US and Russia (all other arsenals combined are less than 10% of the total) are equivalent to a doomsday machine which, if activated, would result in a nuclear winter that ends human civilisation.

As far as I can tell, the research is not nearly as clearcut as Ellsberg would like to tell. Ellsberg writes of the “recent scientific confirmation of the thirty-year old nuclear winter “hypothesis””, but I’m not sure what this is meant to refer to.

The Doomsday Machine is about nuclear history and policy, not the effects of nuclear war, so it makes sense for Ellsberg to omit a detailed analysis of what exactly we think might happen to the atmosphere. However, as best as I can tell given other sources, Ellsberg’s claims of a scientific consensus for civilisation-ending nuclear winter following a war waged with post-Cold War nuclear stockpiles is simply too strong given the current evidence. This is a shame. Nuclear winter is a serious threat, and serious threats do not need exaggeration.

So what is our current understanding of nuclear winter? In a word: complicated.

Some older models of nuclear winter were challenged when burning oil wells in Kuwait during the 1991 Gulf War failed to cause global or even continental cooling. Later papers suggest that sufficiently large burning areas, such as entire cities, might lift smoke much higher than isolated burning oil wells and hence cause greater effects. Others argue that modern cities are not very likely to become firestorms. A recent study estimated 3-17% losses in various crops from a limited Indo-Pakistani war alone. Still others claim they’re being stigmatised as “closet Doctor Strangeloves” for their criticism of the nuclear winter hypothesis.

If we assume that the more aggressive nuclear winter models are not totally off the mark, this analysis estimates that billions of people might plausibly starve to death following a modern US-Russia nuclear exchange. However, to arrive at such estimates involves a long chain of assumptions.

I think all we can say for sure are two things: first, that this is not an experiment we ever want to try, and second, that there exists at least one foolproof solution to global warming.

Regardless of a hypothetical nuclear winter, any nuclear war is bad.

Consider the greatest disasters in human history. Events like World War II, the Black Death, the Great Chinese Famine, and the Spanish flu all had a death toll between 10 and 100 million people (though the high-end estimates for the Black Death go to twice that number).

In the early 1960s, the US Joint Chiefs of Staff estimated that a US first strike on the USSR and China would kill 275 million people immediately, and another 50 million within those countries over the next six months due to injuries and fallout. Attacks on Warsaw Pact countries would kill another 100 million. Collateral damage on neutral countries from fallout would depend on which way the wind blows, but likely add at least another 100 million across nearby countries like Finland, Japan, Sweden, and Afghanistan.

Ellsberg recounts his reaction:
"I remember what I thought when I first held the single sheet with the graph on it. I thought, This piece of paper should not exist. It should never have existed. Not in America. Not anywhere, ever. It depicted evil beyond any human project ever. There should be nothing on earth, nothing real, that it referred to."
The scale is an order of magnitude above any other disaster. In terms of human life lost, it is as if all of World War II (from the Holocaust to Hiroshima to Dresden to Leningrad), the Black Death, the Great Chinese Famine, and the Mongol conquests all happened in a day, followed by World War I, the Spanish flu, and every famine the British ever caused in India over the next few months. Finally, add in some risk of a nuclear winter that slowly kills a significant chunk of the rest through starvation. And this is what happens if we assume that the Soviets don’t hit back.

An argument in favour of nuclear weapons is that they help maintain peace between great powers. This is true, but inadequate.

When asked to put a number on the probability of the Cuban missile crisis escalating into a total nuclear war, Kennedy said “between one in three and even”. Assume this is right, and that the alternative to a nuclear standoff was another worldwide military conflict on the scale of World War II (fought with conventional weapons only). The harsh logic of expected value tells us that the crisis was still a bad deal: we shouldn’t gamble 500-1000 million lives on a coin flip to avoid a 50-100 million death conflict.

Thankfully, a nuclear war today might be less damaging.

First, the number of nuclear weapons has gone down. The US arsenal peaked at 30 000 weapons in the 1960s and Soviet/Russian one at 40 000 in the 1980s; both have since fallen to 6000 - 7000. At the same time, the accuracy of missiles has improved, and smaller, more accurate weapons have replaced huge multi-megaton bombs that can wipe out a city even if they miss by a few kilometres.

Second, increased accuracy allows for strategies to change, at least among the most advanced nuclear powers. Countervalue targeting, where the aim is to inflict maximum damage to an enemy by hitting cities, can be swapped for counterforce targeting, in which the enemy’s military is targeted. The US might plausibly carry out a counterforce attack, but the same is not currently true of China, let alone Pakistan or India. Of course, with nuclear weapons it is impossible to avoid collateral damage, and the extent to which countervalue targeting has been swapped out is hard to tell given the secrecy of current nuclear war plans.

By one estimate, even a limited counterforce scenario for a US-Russia nuclear war would lead to 10 million immediate deaths each in the US, Russia, and (if it’s involved) western Europe. Add in countervalue targeting, and that’s another 100 million across the US and Russia (an estimate for western Europe is not given). By weighing the probabilities of each level of countervalue targeting, the author of this estimate came up with a mean death toll of 50 million direct deaths.

So what can we expect for a modern nuclear war? Any nuclear exchange will likely earn a place near the top of Wikipedia’s “list of wars and anthropogenic disasters by death toll” page. A total one between large nuclear powers will instantly shoot to first place from the number of direct deaths alone. It would, entirely literally, be the worst thing ever.

Then there’s the possibility of nuclear winter. It might be uncertain, but its potential scale means its contribution to the expected number of deaths is considerable. Every 1% increase in the chance of half the world’s population starving is, in expectation, another Canada gone.

Understanding exactly how bad nuclear war would be is important, both to guide policy and to judge its importance relative to other causes. Right now there does not yet seem to be a consensus about nuclear winter risks; if we draw a graph of number of deaths versus probability of it happening, the distribution would be very wide, with most of the expected harm coming from the tail end: scenarios of low probability, but involving billions of deaths. Hopefully the immense efforts rightly spent on modelling climate change will have spillover benefits for nuclear winter research, and allow us to be more certain in our predictions.


Institutional insanity, then and now

Ellsberg does not discuss much about US war plans after his time working with them, likely because after he leaked the Pentagon Papers there was no going back to his job at RAND.

(In fact, the Pentagon Papers were just half of the secret material Ellsberg had copied. Ellsberg decided to release the Vietnam papers first, fearing that if he also released the nuclear planning papers, the Vietnam stuff would be forgotten. His plan to release the nuclear papers later was derailed due to a complex chain of events including letting his friend hide them in a dumpster and flooding from a near-hurricane. Much of the material has since been declassified, however.)

Some things have gotten better. The insanity of a single war plan hitting both the USSR and China must have ended as the Sino-Soviet split progressed (or so I assume). Permissive Action Links (PALs) are now often, but not always, used to make unauthorised nuclear weapon use harder. Nuclear brinksmanship is (for now) less common than during the Cold War.

However, as Ellsberg cautions, to think that the modern nuclear situation is much saner than the one that Ellsberg witnessed in the 1950s and 60s would be a mistake.

A key point to understand about nuclear war is that, if it happens, the reason for it will be stupid.
Nuclear weapons are meant to be used. Their intended use is not as explosives, though. As Ellsberg points out:
"[…] they have been used in the precise way that a gun is used when you point it at someone’s head in a direct confrontation, whether or not the trigger is pulled. For a certain type of gun owner, getting their way in such situations without having to pull the trigger is the best use of the gun. It is why they have it, why they keep it loaded and ready to hand."
The world has fallen into a tragedy-of-the-commons -type situation. The commons, in this case, is the absence of nuclear weapons. Such a world is ideal, but the equilibrium is unstable: the first country to get them can threaten others, and so over time things degenerate until most (big) countries develop them. The commons has become exhausted, no nuclear power is better off relative to the others, and they are all paying the cost: upkeep of weapons, delivery systems, and infrastructure, as well as a small but ever-present risk of accidental mass murder.

(This is a simplification, of course. Nuclear weapons allow some countries to better their position relative even to other nuclear powers; for example, both Pakistan and India have nukes, but Pakistan comes off better in the deal since its weapons help offset its disadvantages in population, resources, and territory (as does the asymmetry of their nuclear policies - India has adopted a no-first-use policy, but Pakistan refuses to).)

No sane person starts a nuclear war. If a nuclear weapon detonates, it has failed its purpose. The real risk both during and after the Cold War is that of accidental nuclear war – technical glitches, inadvertent escalation, Kennedy’s “sons-of-bitches who don’t get the word”.

It is possible to imagine a world where the delicate balance of nuclear deterrents can be maintained with the millimetre precision required to ensure that the expected value of harm remains low. In this world, nuclear weapons may even be a net positive, paying back the costs of their upkeep and probability of accident by reducing the likelihood of non-nuclear conflict.

Is this our world?

If all curtains of secrecy were stripped from US nuclear planning, would we see a rational government carefully shouldering its Atlas-like burdens? What guarantee is there that China and Russia, both of which either already have or are soon likely to have dictators for life, will give appropriate weight to impartial concern for human welfare in their nuclear strategies? Didn’t Narendra Modi order air strikes on another nuclear power to increase his reelection odds just last year?

Building institutions that carry out complex tasks reliably is a very difficult problem.

Consider some of the greatest institutions humanity has come up with. Democracy promises that if you let people vote for their leaders, there’s some chance your country won’t slide into authoritarianism or dysfunction. Free markets boil down to the realisation that you can get away with making surprisingly few decisions about the economy. Scientific publishing allows for the mound of human knowledge to continuously expand, except occasionally there’s a replication crisis and the floor falls in for half a field.

Such institutions are among the greatest achievements of human organisation and intelligence. Yet I still wouldn’t bet my life on a breakthrough study replicating, the stock market updating on a building but predictable global pandemic in its early stages, or a European democracy never sliding into dictatorship. Ask me to bet tens of millions of lives on US, Russian, Chinese, French, British, Pakistani, Indian, and Israeli secret military institutions all working reliably over a time horizon of decades, and I start to wonder when the first ship leaves for Mars.

I am not generally fond of arguments about human hubris (too often they boil down to vague complaints that using our ingenuity to improve life would somehow be bad). This time, however, there is truth to them. For the governments of the United States and Russia to believe that they can wield a thousand-weapon arsenal responsibly is pretence. Our current institution-building abilities are not up to the task. (We don’t yet know whether we can even build institutions that guarantee human flourishing in the long run, but unlike the nuclear problem, this problem we have no choice but to attempt.)


What should we do?

What are the most effective things we can do to lower the expected harm of nuclear war – that is, reduce both its probability and the damage it would cause if it happened?

Ellsberg rightly recommends downsizing the US and Russian arsenals (together over 90% of the world total) as a first step, since this would reduce any risk of a catastrophic nuclear winter. For example, the US could start by unilaterally getting rid of its land-based missiles, which would reduce the time pressures on making a launch decision (land-based missiles will likely be the first targets of any attack and hence will be lost unless launched soon after a warning), and deprive Russian missiles of their first targets, making it more justifiable for Russia to cut back on its own arsenal.

Another step is for countries to take weapons off hair-trigger launch alert. Constantly being two mistakes and ten minutes away from nuclear launch is not sustainable in the long run. Deterrence could be maintained through a focus on hardier nuclear forces like submarines, and less reliance on sitting ducks like land-based missile silos.

To make both of these steps more likely, diplomatic efforts into nuclear arms control treaties should be increased. This has not been happening. The US suspended the Intermediate-Range Nuclear Forces Treaty in 2019 due to perceived Russian violations and because it didn’t cover China. The remaining major US-Russia treaty, New START (STrategic Arms Reduction Treaty), will likely expire in February 2021 unless Trump reverses course and negotiations happen very quickly. Treaties for old weapons aren’t enough, either; new technologies, like hypersonic gliders that can keep lower than ballistic missiles and perform evasive manoeuvres, might destabilise nuclear deterrence.

In the long run, the aim should be to either abolish weaponised nukes entirely, or, failing that, at least reach a state where only a few accountable states wield small numbers of weapons. Biological and chemical weapons of mass destruction have been reined in by treaties and mostly abolished. Nuclear weapons should be next.


Surviving by design

During the Cold War, it was easy to construct a compelling narrative around nuclear war: the climactic showdown between the forces of capitalism and communism and between democracy and totalitarianism, to be waged for infinite stakes with the ultimate fruits of modern science. With the end of the Cold War, the narrative was lost. Nuclear war was still possible – the only time a “nuclear briefcase” was ever activated was in 1995 – but it was largely relegated to the realm of technical glitches and accidents; not things that make for a good story.

As Ellsberg points out, the biggest nuclear threat never was, and still isn’t, an intentional conflict (or rogue states or terrorists). It is the potential for failure in the institutions, people, and machines that control the biggest nuclear arsenals.

If nuclear war starts, it won’t be grave geopolitical considerations that trigger it. It will be a country dropping a nuclear weapon on itself, someone accidentally inserting a nuclear war training tape into an operational computer, radar equipment getting confused by the moon, or Russian bureaucracy being Russian bureaucracy. (Each of these happened; see links.)

Stories are nice. It’s tempting to demand a certain narrative coherence from the world; to think that sufficiently bad things, for sufficiently dumb reasons, aren’t allowed to happen.

But we do not live in the world of narrative coherence. We live in the world where civilisation is indefinitely on hold because of bat soup in China. The greatest risks we face aren’t wrapped up in compelling narratives, and they do not come from commensurate causes.

In particular, it is important to be aware that there are no safeguards. A world without us may seem pointless, but the laws of physics will bring it about given the right chain of cause and effect.  If we want protection against catastrophe, we must build it ourselves.

We did not survive the Cold War by design. We survived it by accident: because none of the close calls quite managed to escalate to full-blown war, and because of heroes like Vasili Arkhipov and Stanislav Petrov who decided not to press the button.

We should survive the 21st century by design, not accident. This is not a given; with ever greater technology comes an ever greater number of efficient ways to kill a lot of people.

Making nuclear war less likely and less disastrous is an important part of achieving this. It is not the the only part, nor necessarily the most urgent, but I will say this: if civilisation has to be severely damaged by some apocalyptic scenario, we might as well make sure that it’s something fancy and trendy like unfriendly AI, not something straight from a 1960s comedy film.