Death is bad

 3.5k words (about 12 minutes)

Sometime in the future, we might have the technology to extend lifespans indefinitely and make people effectively immortal. When and how this might happen is a complicated question that I will not go into. Instead, I will take heed of Ian Malcolm in Jurassic Park, who complains that "your scientists were so preoccupied with whether or not they could that they didn't stop to think if they should".

This is (in my opinion rather surprisingly) a controversial question.

The core of it is this: should people die?

Often the best way to approach a general question is to start by thinking about specific cases. Imagine a healthy ten-year old child; should they die? The answer is clearly no. What about yourself, or your friends, or the last person you saw on the street? Wishing for death for yourself or others is almost universally a sign of a serious mental problem; acting on that desire even more so.

There are some exceptions. Death might be the best option for a sick and pained 90-year-old with no hope of future healthy days. It may well be (as I've seen credibly claimed in several places) that the focus on prolonging lifespan even in pained terminally ill people is excessive. "Prolong life, whatever the cost" is a silly point of view; maximising heartbeats isn't what we really care about.

However, now imagine a pained, dying, sick person who has a hope of surviving to live many healthy happy days – say a 40-year-old suffering from cancer. Should they die? No. You would hope that they get treatment, even if it's nauseating fatiguing painful chemotherapy for months on end. If there is no cure, you'd hope that scientists somewhere invent it. Even if it does not happen in time for that particular person, at least it will save others in the future, and eliminate one more horror of the world. It would be a great and celebrated human achievement.

What's the difference between the terminally ill 90-year-old and the 40-year-old with a curable cancer? The difference is technology. We have the technology to cure some cancers, but we don't have the technology to cure the many ageing-related diseases. If we did, then even if the treatment is expensive or difficult, we would hope – and consider it a moral necessity – for both of them to get it, and hope that they both go on living for many more years.

No one dies of time. You are a complex process running on the physical hardware of your brain, which is kept running by the machine that is the rest of your body. You die when that machine breaks. There is no poetic right time when you close your eyes and get claimed by time, there is only falling to one mechanical fault or another.

People (or conscious beings in general) matter, and their preferences should be taken seriously – this is the core of human morality. What is wrong in the world can be fixed – this is the guiding principle of civilisation since the Enlightenment.

So, should people die? Not if they don't want to, which (I assume) for most people means not if they have a remaining hope of happy, productive days.

Counterarguments

The idea that death is something to be defeated, like cancer, poverty, or smallpox, is not a common one. Perhaps there's some piece of the puzzle that is missing from the almost stupidly simple argument above?

One of the most common counterarguments is overpopulation (perhaps surprisingly; environmentalist concerns have clearly penetrated very deep into culture despite not being much of a thing before the 1970s). The argument goes like this: if we solve death, but people keep being born, there will be too many people on Earth, leading to environmental problems, and eventually low quality of life for everyone.

The object-level point (I will return to what I consider more important meta-level points later) is that demographic predictions have a tendency to be wrong, especially about the future (as the Danish (?) saying goes). Malthus figured out pre-industrial demographics just as they came to an end with the industrial revolution. In the 1960s, there were warnings of a population explosion, which fizzled out when it turned out that the demographic transition (falling birth rates as countries develop) is a thing. Right now the world population is expected to stabilise at less than 1.5x the current size, and many developed countries are dealing with problems caused by shrinking populations (which they strangely refuse to fix through immigration).

Another concern are the effects of having a lot of old people around. What about social progress – how would the development of women's rights have been realised if you had a bunch of 19th century misogynists walking around in their top hats? What sort of power imbalances and Gini coefficients would we reach if Franklin Delano Roosevelt could continue cycling through high-power government roles indefinitely, or Elon Musk had time to profit from the colonisation of Mars? What happens to science when it can no longer advance (as Max Planck said) one funeral at at time?

(There is even an argument that life extension technology is problematic because the rich will get it first. This is an entirely general and therefore entirely worthless argument, since it applies to all human progress: the rich got iPhones first – clearly smartphones are a problematic technology, etc., etc. If you're worried about only the rich having access to it for too long, the proper response is to subsidise its development so that the period when not everyone has access to it is as short as possible.)

These are valid concerns that will definitely test the abilities of legislators and voters in the post-death era. However, they can probably be overcome. I think people can be brought around surprisingly far on social and moral attitudes without killing anyone. Consider how pre-2000 almost anyone's opinions would have made them a near-pariah today; many of those people still exist and it would hard to write them off as a total loss. Maybe some minority of immortal old people couldn't cope with all the Pride Parades – or whatever the future equivalent is – marching past their windows and they go off to start some place of their own with sufficient top hat density; then again, most countries have their own conservative backwater region already. If they start going for nukes, that's more of an issue, but not more so than Iran.

As for imbalances of power and wealth, it might require a few more taxes and other policies (the expansion of term limits to more jobs?), but given the strides that equalising policy-making has made it seems hard to argue there is a fundamental impossibility.

And what about all the advantages? A society of the undying might well be far more long-term oriented, mitigating one of the greatest human failures. After all, how often do people bemoan that 70-year-old oil executives just don't care because they won't be around to see the effects of climate change?

What about all the collective knowledge that is lost? Imagine if people in 2050 could hear World War II veterans reminding them of what war really is. Imagine if John von Neumann could have continued casually inventing fields of maths at a rate of about two per week instead of dying at age 53 (while absolutely terrified of his approaching death). Imagine if we could be sure to see George R. R. Martin finish A Song of Ice and Fire.

Also, concerns like overpopulation and Elon Musk's tax plan just seem small in comparison to the literal eradication of death.

Imagine proposing a miracle peace plan to the cabinets of the Allied countries in the midst of World War II. The plan would end the war, install liberal governments in the Axis powers, and no one even has to nuke a Japanese city. (If John von Neumann starts complaining about not getting to test his implosion bomb design, give him a list of unsolved maths problems to shut him up.) Now imagine that the reaction is somewhere between hesitance and resistance, together with comments like "where are we going to put all the soldiers we've trained?", "what about the effects on the public psyche of a random abrupt end without warning?", and "how will we make sure that the rich industrialists don't profit too much from all the suddenly unnecessary loans that they've been given?" At this point you might be justified in shouting: "this war is killing fifteen million people per year, we need to end it now".

The situation with death is similar, except it's over fifty million per year rather than fifteen. (See this chart for breakdown by cause – you'll see that while currently-preventable causes like infectious diseases kill millions, ageing-related ones like heart disease, cancer, and dementia are already the majority.)

Thought experiments

To make the question more concrete, we can try thought experiments. Imagine a world in which people don't die. Imagine visitors from that world coming to us. Would they go "ah yes, inevitable oblivion in less than a century, this is exactly the social policy we need, thanks – let us go run back home and implement it"? Or would they think of our world like we do of a disease-stricken third-world country, in dire need of humanitarian assistance and modern technology?

It's hard to get into the frame of mind of people who live in a society that doesn't hand out automatic death sentences to everyone at birth. Instead, to evaluate whether raising life expectancies to 200 makes sense even given the environmental impacts, we can ask whether a policy of killing people at age 50 to reduce population pressures would be even better than the current status quo – if both an increase and decrease in life expectancies is bad, this is suspicious because it implies we're at the optimum by chance. Or, since the abstract question (death in general) is always harder than more concrete ones, imagine withholding a drug that manages heart problems in the elderly on overpopulation grounds.

You might argue that current life expectancies are optimal. This is a hard position to defend. It seems like a coincidence that the lifespan achievable with modern technology is exactly the "right" one. Also, neither you nor society should not make that choice for other people. Perhaps some people get bored of life and readily step into coffins at age 80; many others want nothing more than to keep living. People should get what they want. Forcing everyone to conform to a certain lifespan is a specific case of forcing everyone to conform to a certain lifestyle; much moral progress in the past century has consisted of realising that this is bad.

I think it's also worth emphasising one common thread in the arguments against solving death: they are all arguments about societal effects. It is absolutely critical to make sure that your actions don't cause massive negative externalities, and that they also don't amount to defecting in prisoner's dilemma or the tragedy of the commons. However, it is also absolutely critical that people are happy and aren't forced to die, because people and their preferences/wellbeing are what matters. Society exists to serve the people who make it up, not the other way around. Some of the worst moral mistakes in history come from emphasising the collective, and identifying good and harm in terms of effects on an abstract collective (e.g. a nation or religion), rather than in terms of effects on the individuals that make it up. Saying that everyone has to die for some vague pro-social reason is the ultimate form of such cart-before-the-horse reasoning.

Why care about the death question?

There are several features that make the case against death, and people's reactions to it, particularly interesting.

Failure of generalisation

First: generalisation. I started this post using specific examples before trying to answer the more general question. I think the popularity of death is a good example of how bad humans are at generalising.

When someone you know dies, it is very clearly and obviously a horrible tragedy. The scariest thing that could happen to you is probably either your own death, the death of people you care about, or something that your brain associates with death (the common fears: heights, snakes, ... clowns?).

And yet, make the question more abstract – think not about a specific case (which you feel in your bones is a horrible tragedy that would never happen in a just world), but about the general question of whether people should die, and it's like a switch flips: a person who would do almost anything to save themselves or those they care about, who cares deeply about suffering and injustice in the world, is suddenly willing to consign five times the death toll of World War I to permanent oblivion every single year.

Stalin reportedly said that a single death is a tragedy, but a million is only a statistic. Stalin is wrong. A single death is a tragedy, and a million deaths is a million tragedies. Tragedies should be stopped.

People These Days

Second: today, we're pretty good at ignoring and hiding death. This wasn't always the case. If you're a medieval peasant, death is never too far away, whether in the form of famine or plague or Genghis Khan. Death was like an obnoxious dinner guest: not fun, but also just kind of present in some form or another whether you invited them or not, so out of necessity involved in life and culture.

Today, unexpected death is much rarer. Child mortality globally has declined from over 40% (i.e. almost every family had lost a child) in 1800 to 4.5% in 2015, and below 0.5% in developed countries. Famines have gone from something everyone lives through to something that the developed world is free from. War and conflict have gone from common to uncommon. Much greater diseases and accidents can be successfully treated. As a result of all these positive trends, death is less present in people's minds.

As I don't have my culture critic license yet, I won't try to make some fancy overarching points about how People These Days Just Don't Understand and how our Materialistic Culture fails to prepare people to deal with the Deep Questions and Confront Their Own Mortality. I will simply note that (a) death is bad, (b) we don't like thinking about bad things, and (c) sometimes not wanting to think about important things causes perverse situations.

Confronting problems

Why do people not want to think that death is bad? I think one central reason is that death seems inevitable. It's tough to accept bad things you can't influence, and much easier to try to ignore them. If at some point you have to confront it anyways, one of the most reassuring stories you can tell is that it has a point. Imagine if over two hundred thousand years, generation after generation of humans, totalling some one hundred billion lives, was born, grew up, developed a rich inner world, and then had that world destroyed forever by random failures, evolution's lack of care for what happens after you reproduce, and the occasional rampaging mammoth. Surely there must be some purpose for it, some reason why all that death is not just a tragedy? Perhaps we aren't "meant" to live long, whatever that means, or perhaps it's all for the common good, or that "death gives meaning to life". Far more comforting to think that then to acknowledge that a hundred billion human lives and counting really are gone forever because they were unlucky enough to be born before we eradicated smallpox, or invented vaccines, or discovered antibiotics, or figured out how to reverse ageing.

Assume death is inevitable. Should you still recognise the wrongness of it?

I think yes, at least if you care about big questions and doing good. I think it's important to be able to look at the world, spot what's wrong about it, and acknowledge that there are huge things that should be done but are very difficult to achieve.

In particular, it's important to avoid the narrative fallacy (Nassim Taleb's term for the human tendency to want to fit the world to a story). In a story, there's a start and an end and a lesson, and the dangers are typically just small enough to be defeated. Our universe has no writer, only physics, and physics doesn't care about hitting you with an unsolvable problem that will kill everyone you love. If you want to increase the justness of the world, recognising this fact is an important starting point.

Taxes

Is death inevitable? In considering this question, it's important once again to remember that death is not a singular magical thing. Your death happens when something breaks badly enough that your consciousness goes permanently offline.

Things, especially complex biological machines produced by evolution, can break in very tricky ways. But what can break can be fixed, and people who declare technological feats impossible have a bad track record. The problem might be very hard: maybe we have to wait until we have precision nano-bots that can individually repair the telomeres on each cell, or maybe there is no effective general solution to ageing and we face an endless grind of solving problem after problem to extend life/health expectancies from 120 to 130 to 140 and so forth. Then again, maybe someone leaves out a petri dish by accident in a lab and comes back the next day to the fountain of youth, or maybe by the end of the century no one is worrying about something as old-fashioned as biology.

There's also the possibility of stopgap solutions, like cryonics (preserving people close to death by vitrifying them and hoping that future technology can revive them). Cryonics is currently in a very primitive state – no large animals successfully having been put through it – but there's a research pathway of testing on increasingly complex organs and then increasingly large animals that might eventually lead to success if someone bothered to pour resources into it.

There is no guarantee when this is happening. If civilisation is destroyed by an engineered pandemic or nuclear war before then, it will never happen.

Of course, in the very long run we face more fundamental problems, like the heat death of the universe. Literally infinite life is probably physically impossible; maybe this is reassuring.

Predictions and poems

I will make three predictions about the eventual abolition of death.

First, many people will resist it. They might see it as conflicting with their religious views or as exacerbating inequality, or just as something too new and weird or unnatural.

Second, when the possibility of extending their lifespan stops being an abstract topic and becomes a concrete option, most people will seize it for themselves and their families.

This is a common path for technologies. Lightning rods and vaccines were first seen by some as affronts to God's will, but eventually it turns out people like not burning to death and not dying of horrible diseases more than they like fancy theological arguments. Most likely future generations will discover that they like not ageing more than they like appreciating the meaning of life by definitely not having one past age 120.

Finally, future people (if they exist) will probably look back with horror on the time when everyone died against their will within about a century.

Edgar Allen Poe wrote a poem called "The Conqueror Worm", about angels crying as they watch a tragic play called "Man", whose (anti-)hero is a monstrous worm that symbolises death. If we completely ignore what Poe intended with this, we can misinterpret one line to come to a nice interpretation of our own. The poem declares that the angels are watching this play in the "lonesome latter years". Clearly this refers to a future post-scarcity, post-death utopia, and the angels are our wise immortal descendants reflecting on the bad old days, when people were "mere puppets [...] who come and go / at the bidding of vast formless things" like famine and war and plague and death. The "circle [of life] ever returneth in / To the self same spot [= the grave]", and so the "Phantom [of wisdom and fulfilled lives] [is] chased for evermore / By a crowd that seize it not".

Death is a very poetic topic, and other poems need less (mis)interpretation. Edna St. Vincent Millay's "Dirge Without Music" is particularly nice, while Dylan Thomas gives away the game in the title: "Do not go gentle into that good night".

EA ideas 4: utilitarianism

4.9k words (≈17 minutes)

 Posts in this series:
FIRST | PREVIOUS | NEXT

Many ideas in effective altruism (EA) do not require a particular moral theory. However, while there is no common EA moral theory, much EA moral thinking leans consequentialist (i.e. morality is fundamentally about consequences), and often specifically utilitarian (i.e. wellbeing and/or preference fulfilment are the consequences we care about).

Utilitarian morality can be thought of as rigorous humanism, where by “humanism” I mean the general post-Enlightenment secular value system that emphasises caring about people, rather than upholding, say, religious rules or the honour of nations. Assume that the welfare of a conscious mind matters. Assume that our moral system should be impartial: that wellbeing/preferences should count the same regardless of who has them, and also in the sense of being indifferent of who’s perspective it is being wielded from (for example, a moral system that says to only value yourself would give you different advice than it gives me). The simplest conclusion you can draw from these assumptions is to consider welfare to be good and seek to increase it.

I will largely ignore differences between the different types of utilitarianism. Examples of divisions within utilitarianism include preference vs hedonic/classical utilitarianism (do we care about the total satisfied preferences, or the total wellbeing; how different are these?) and act vs rule utilitarianism (is the right act the one with the greatest good as its consequence, or the one that conforms to a rule which produces the greatest good as its consequences – and, once again, are they different?).

Utilitarianism is decisive

We want to do things that are “good”, so we have to define what we mean by it. But once we’ve done this, this concept of good is of no help unless it lets us make decisions on how to act. I will refer to the general property of a moral system being capable of making non-paradoxical decisions as decisiveness.

Decisiveness can fail if a moral system leads to contradiction. Imagine a deontological system with the rules “do not lie” and “do not take actions that result in someone dying”. Now consider the classic thought experiment of what such a deontologist would do if the Gestapo knocked on their door and asked if they’re hiding any Jews. A tangle of absolute rules almost ensures the existence of some case where they cannot all be satisfied, or where following them strictly will cause immense harm.

Decisiveness fails if our system allows circular preferences, since then you cannot make a consistent choice. Imagine you follow a moral system that says volunteering at a soup kitchen is better than helping old people across the street, collecting money for charity is better than soup kitchen volunteering, and helping old people across the street is better than collecting money. You arrive at the soup kitchen and decide to immediately walk out to go collect money. You stop collecting money to help an old person across the street. Halfway through, you abandon them and run off back to the soup kitchen.

Decisiveness fails if there are tradeoffs our system cannot make. Imagine highway engineers deciding whether to bulldoze an important forest ecosystem or a historical monument considered sacred. If your moral system cannot weigh environment against historical artefacts (and economic growth, and the time of commuters, and …), it is not decisive.

So for any two choices, a decisive moral system must be able to compare them, and the comparisons it makes cannot be circular preference. This implies a ranking: X is better than Y translates to X is before Y in the ranking list.

(If we allow circular preferences, we obviously can’t make a list, since the graph of “better-than” relations would include cycles. If there are tradeoffs we can’t make – X and Y such that X and Y are neither better than equal or worse than each other – we can generate a ranking list but not a unique one (in set theory terms, we have a partial order rather than a total order).)

Decisiveness also fails if our system can’t handle numbers. It is better to be happy for two minutes than one minute than fifty nine seconds. More generally, to practically any good we can either add or subtract a bit: one more happy thought, one less bit of pain.

Therefore a decisive moral system must rank all possible choices (or actions or world states or whatever), with no circular preferences, and with arbitrarily many notches between each ranking. It sounds like what we need is numbers: if we can assign a number to choices, then there must exist a non-circular ranking (you can always sort numbers), and there’s no problem with handling the quantitativeness of many moral questions.

There can’t be one axis to measure the value of pleasure, one to measure meaning, and another for art. Or there can – but at the most basic level of moral decision-making, we must be able to project everything onto the same scale, or else we’re doomed to have important moral questions where we can only shrug our shoulders. This leads to the idea of all moral questions being decidable by comparing how the alternatives measure up in terms of “utility”, the abstract unit of the basic value axis.

You might say that requiring this extreme level of decisiveness may sometimes be necessary in practice, but it’s not what morality is about; perhaps moral philosophy should concern itself with high-minded philosophical debates over the nature of goodness, not ranking the preferability of everything. Alright, have it your way. But since being able to rank tricky “ought”-questions is still important, we’ll make a new word for this discipline: fnergality. You can replace “morality” or “ethics” with “fnergality” in the previous argument and in the rest of this post, and the points will still stand.

What is utility?

So far, we have argued that a helpful moral system is decisive, and that this implies it needs a single utility scale for weighing all options.

I have not specified what utility is. Without this definition, utilitarianism is not decisive at all.

How you define utility will depend on which version of utilitarianism you endorse. The basic theme across all versions of utilitarianism is that utility is assigned without prejudice against arbitrary factors (like location, appearance, or being someone other than the one who is assigning utilities), and is related to ideas of welfare and preference.

A hedonic utilitarian might define the utility of a state of the world as total wellbeing minus total suffering across all sentient minds. A preference utilitarian might ascribe utility to each instance of a sentient mind having a preference fulfilled or denied, depending on the weight of the preference (not being killed is likely a deeper wish than hearing a funny joke), and the sentience of the preferrer (a human’s preference is generally more important than a cat’s). Both would likely want to maximise the total utility that exists over the entire future.

These definitions leave a lot of questions unanswered. For example, take the hedonic utilitarian definition. What is wellbeing? What is suffering? Exactly how many wellbeing units are being experienced per second by a particular jogger blissfully running through the early morning fog?

The fact that we can’t answer “4.7, ±0.5 depending on how runny their nose is” doesn’t mean utilitarianism is useless. First, we might say that an answer exists in principle, even if we can’t figure it out. For example, a hedonic utilitarian might say that there is some way to calculate the net wellbeing experienced by any sentient mind. Maybe it requires knowing every detail of their brain activity, or a complete theory of what consciousness is. But – critically – these are factual questions, not moral ones. There would be moral judgements involved in specifying exactly how to carry out this calculation, or how to interpret the theory of consciousness. There would also be disagreements, in the same way that preference and hedonic utilitarians disagree today (and it is a bad idea to specify one Ultimate Goodness Function and declare morality solved forever). But in theory and given enough knowledge, a hedonic utilitarian theory could be made precise.

Second, even if we can only approximate utilities, doing so is still an important part of difficult real-world decision-making.

For example, Quality- and Disability-Adjusted Life Years (QALYs and DALYs) try to put a number on the value of a year of life with some disease burden. Obviously it is not an easy judgement to make (usually the judgement is made by having a lot of people answer carefully designed questions on a survey), and the results are far more imprecise than the 3-significant-figure numbers in the table on page 17 here would suggest. However, the principle that we should ask people and do studies to try figure out how much they’re suffering, and then make the decisions that reduce suffering the most across all people, seems like the most fair and just way to make medical decisions.

Using QALYs may seem coldly numerical, but if you care about reducing suffering, not just as a lofty abstract statement but as a practical goal, you will care about every second. It can also be hard to accept QALY-based judgements, especially if they prefer others to people close to you. However, taking an impartial moral view, it is hard not to accept that the greatest good is better than a lesser good that includes you.

(Using opposition to QALYs as an example, Robin Hanson argues with his characteristic bluntness that people favour discretion over mathematical precision in their systems and principles “as a way to promote an informal favoritism from which they expect to benefit”. In addition to the ease of sounding just and wise while repeating vague platitudes, this may be a reason why the decisiveness and precision of utilitarianism become disadvantages on the PR side of things.)

Morality is everywhere

By achieving decisiveness, utilitarianism makes every choice a moral one.

One possible understanding of morality is that it splits actions into three planes. There are rules for what to do (“remember the sabbath day”). There are rules for what not to do (“thou shalt not kill, and if thy doest, thy goeth to hell”). And then there’s the earthly realm, of questions like whether to have sausages for dinner, which – thankfully – morality, god, and your local preacher have nothing to say about.

Utilitarianism says sausages are a moral issue. Not a very important one, true, but the happiness you get from eating them, your preferences one way or the other, and the increased risk of heart attack thirty years from now, can all be weighed under the same principles that determine how much effort we should spend on avoiding nuclear war. This is not an overreach: a moral theory is a way to answer “ought”-questions, and a good one should cover all of them.

This leads to a key strength of utilitarianism: it scales, and this matters, especially when you want to apply ethics to big uncertain things. But first, a slight detour.

Demandingness

A common objection to utilitarianism is that it is too demanding.

First of all, I find this funny. Which principle of meta-ethics is it, exactly, that guarantees your moral obligations won’t take more than the equivalent of a Sunday afternoon each week?

However, I can also see why consequentialist ethics can seem daunting. For someone who is used to thinking of ethics in terms of specific duties that must always be carried out, a theory that paints everything with some amount of moral importance and defines good in terms of maximising something vague and complicated can seem like too much of a burden. (I think this is behind the misinterpretation that utilitarianism says you have a duty to calculate that each action you take is the best one possible, which is neither utilitarian nor an effective way to achieve anything.)

Utilitarianism is a consequentialist moral theory. Demands and duties are not part of it. It settles for simply defining what is good.

(As it should. The definition is logically separate from the implications and the implementation. Good systems, concepts, and theories are generally narrow.)

Scaling ethics to the sea

There are many moral questions that are, in practice, settled. All else being equal, it is good to be kind, have fun, and help the needy.

To make an extended metaphor: we can imagine that there is an island of settled moral questions; ones that no one except psychopaths or philosophy professors would think to question.

This island of settled moral questions provides a useful test for moral systems. A moral system that doesn’t advocate kindness deserves to go in the rubbish. But though there is important intellectual work to be done in figuring out exactly what grounds this island (the geological layers it rests on, if you will), the real problem of morality in our world is how we extrapolate from this island to the surrounding sea.

In the shallows near the island you have all kinds of conventional dilemmas – for example, consider our highway engineers in the previous example weighing nature against art against economy. Go far enough in any direction and you will encounter all sorts of perverse thought experiment monsters dreamt up by philosophers, which try to tear apart your moral intuitions with analytically sharp claws and teeth.

You might think we can keep to the shallows. That is not an option. We increasingly need to make moral decisions about weird things, due to the increasing strangeness of the world: complex institutions, new technologies, and the sheer scale of there being over seven billion people around.

A moral system based on rules for everyday things is like a constant-sized knife: fine for cutting up big fish (should I murder someone?), but clumsy at dealing with very small fish (what to have for dinner?), and often powerless against gargantuan eldritch leviathans from the deep (existential risk? mind uploading? insect welfare?).

Utilitarianism scales both across sizes of questions and across different kinds of situations. This is because it isn’t based on rules, but on a concept (preference/wellbeing) that manages to turn up whenever there are morally important questions. This gives us something to aim for, no matter how big or small. It also makes us value preference/wellbeing wherever it turns up, whether in people we don’t like, the mind of a cow, or in aliens.

Utilitarianism and other kinds of ethics

Utilitarianism, and consequentialist ethics more broadly, lacks one property that is a common social (if not philosophical) use of morality.

Consider confronting a thief robbing a jewellery store. A deontological argument is “stealing is wrong; don’t do it”. A utilitarian argument would need to spell out the harms: “don’t steal, because you will cause suffering to the owner of the shop”. But the thief may well reply: “yes, but the wellbeing I gain from distributing the proceeds to my family is greater, so my act is right”. And now you’d have to point out that the costs to the shop workers who will lose their jobs if the shop goes bankrupt, plus more indirect costs like the effect on people’s trust in others or feelings of safety, outweigh these benefits – if they even do. Meanwhile the thief makes their escape.

By making moral questions depend heavily on facts about the world, utilitarianism does not admit smackdown moral arguments (you can always be wrong about the facts, after all). This is a feature, not a bug. Putting people in their place is sometimes a necessary task (as in the case of law enforcement), but in general it is the province of social status games, not morality.

Of course, nations need laws and people need principles. The insight of utilitarianism is that, important as these things are, their rightness is not axiomatic. There is a notion of good, founded on the reality of minds doing well and fulfilling their wishes, that cuts deeper than any arbitrary rule can. It is an uncomfortable thought that there are cases where you should break any absolute moral rule. But would it be better if there were rules for which we had to sacrifice anything?

Recall the example of the Gestapo asking if you’re hiding Jews in your house. Given an extreme enough case, whether or not a moral rule (e.g. “don’t lie”) should be followed does depend on the effects of an action.

At first glance, while utilitarianism captures the importance of happiness, selflessness, and impartiality, it doesn’t say anything about many other common moral topics. We talk about human rights, but consequentialism admits no rights. We talk about good people and bad people, but utilitarianism judges only consequences, not the people who bring them about. In utilitarian morality, good intentions alone count for nothing.

First, remember that utilitarianism is a set of axioms about the most fundamental definition of good is. Just like simple mathematical axioms can lead to incredible complexity and depth, if you follow utilitarian reasoning down to daily life, you get a lot of subtlety and complexity, including a lot of common-sense ethics.

For example, knowledge has no intrinsic value in utilitarianism. But having an accurate picture of what the world is like is so important for judging what is good that, in practice, you can basically regard accurate knowledge as a moral end in itself. (I think that unless you never intend to be responsible for others or take actions that significantly affect other people, when deciding whether to consider something true you should care only about its literal truth value, and not at all about whether it will make you feel good to believe it.)

To take another example: integrity, in the sense of being honest and keeping commitments, clearly matters. This is not obvious if you look at the core ideas of utilitarianism, in the same way that the Chinese Remainder Theorem is not obvious if you look at the axioms of arithmetic. That doesn’t somehow make it un-utilitarian; for some examples of arguments, see here.

See also this article for ideas on why strictly following rules can make sense even for strict consequentialists, given only the fact that human brains are fallible in predictable ways.

As a metaphor, consider scientists. They are (in some idealised hypothetical world) committed only to the pursuit of truth: they care about nothing except the extent to which their theories precisely explain the world. But the pursuit of this goal in the real world will be complicated, and involve things – say, wild conjectures, or following hunches – that might even seem to go against the end goal. In the same way, real-world utilitarianism is not a cartoon caricature of endlessly calculating consequences and compromising principles for “the greater good”, but instead a reminder of what really matters in the end: the wishes and wellbeing of minds. Rights, duties, justice, fairness, knowledge, and integrity are not the most basic elements of (utilitarian) morality, but that doesn’t make them unimportant.

Utilitarianism is horrible

Utilitarianism may have countless arguments on its side, but one fact remains: it can be pretty horrible.

Many thought experiments show this. The most famous is the trolley problem, where the utilitarian answer requires diverting a trolley from a track containing 5 people to one containing only a single person (an alternative telling is doctors killing a random patient to get the organs to save five others). Another is the mere addition paradox, also known as the repugnant conclusion: we should consider a few people living very good lives as a worse situation than many people living mediocre lives.

Of course, the real world is never as stark as philosophers’ thought experiments. But a moral system should still give an answer – the right one – to every moral dilemma.

Many alternatives to utilitarianism seem to fail at this step; they are not decisive. It is always easier to wallow in platitudes than to make a difficult choice.

If a moral system gives an answer we find intuitively unappealing, we need to either reject the moral system, or reject our intuitions. The latter is obviously dangerous: get carried away by abstract morals, and you might find yourself denying common-sense morals (the island in the previous metaphor). However, particularly when dealing with things that are big or weird, we should expect our moral intuitions to occasionally fail.

As an example, I think the repugnant conclusion is correct: for any quantity of people living extremely happy lives, there is some larger quantity of people living mediocre lives that would be a better state for the world to be in.

First, rejecting the repugnant conclusion means rejecting total utilitarianism: the principle that you sum up individual utilities to get total utility (for example, you might average utilities instead). Rejecting total utilitarianism implies weird things, like the additional moral worth of someone’s life depending on how many people are already in the world. Why should a happy life in a world with ten billion people be worth less than one in a world with a thousand people?

Alternatives also bring up their own issues. To take a simple example, if you value average happiness instead, eliminating everyone who is less happy than the average is a good idea (in the limit, every world of more than one person should be reduced to a world of one person).

Finally, there is a specific bias that explains why the repugnant conclusion seems so repugnant. Humans tend to show scope neglect. If our brains were built differently, and assigned due weight to the greater quantity of life in the “repugnant” choice, I think we’d find it the intuitive one.

However, population ethics is both notoriously tricky and a fairly new discipline, so there is always the chance there exists a better alternative population axiology than totalism.

Is utilitarianism complete and correct?

I’m not sure what evidence or reasoning would let us say that a moral system is complete and correct.

I do think the basic elements of utilitarianism are fairly solid. First, I showed above how requiring decisiveness leads to most of the utilitarian character of the theory (quantitativeness, the idea of utility). The reasons are similar to the ones for using expected value reasoning: if you don’t, you either can’t make some decisions, or introduce cases where you make stupid ones. Second, ideas of impartiality and universality seem like fundamental moral ideas. I’d be surprised if you could build a consistent, decisive, and humane moral theory without the ideas of quantified utility and impartiality.

Though this skeleton may be solid, the real mess lies with defining utility.

Do we care about preferences or wellbeing? It seems that if we define either in a broad enough way to be reasonable, the ideas start to converge. Is this a sign that we’re on the right track because the two main variants of utilitarianism talk about a similar thing, or that we’re on the wrong track and neither concept means much at all?

Wellbeing as pleasure leaves out most of what people actually value. Sometimes people prefer to feel sadness; we have to include this. How? Notice the word I used – “prefer”. It seems like this broad-enough “wellbeing” concept might just mean “what people prefer”. But try defining the idea of preference. Ideal preferences should be sincere and based on perfect information – after all, if you hear information that changes your preference, it’s your estimate of the consequences that changed, not the morally right action. So when we talk about preference, we need complete information, which means trying to answer the question “given perfect information about what you will experience (or even the entire state of the universe, depending on what preferences count) in option A and in option B, which do you prefer?” Now how is this judgement made? Might there be something – wellbeing, call it – which is what a preferrer always prefers?

Capturing any wellbeing/preference concept is difficult. Some things are very simple: a healthy life is preferable to death, for example, and given the remaining horribleness in the real world (e.g. sixty million people dying each year) a lot of our important moral decisions are about the simple cases. Even the problem of assigning QALY values to disease burdens has proven tractable, if not easy or uncontroversial. But solving the biggest problems is only the start.

An important empirical fact about human values is that they’re complex. Any simple utopia is a dystopia. Maybe the simplest way to construct a dystopia is to imagine a utopia and remove one subtle thing we care about (e.g. variety, choice, or challenge).

On one hand, we have strong theoretical reasons why we need to reduce everything to utilities to make moral decisions. On the other, we have the empirical fact that what counts as utility to people is very complex and subtle.

I think the basic framework of utilitarian ideas gives us a method, in the way that the ruler and compass gave the Greeks a method to begin toying with maths. Thinking quantitatively about how all minds everywhere are doing is probably a good way to start our species’ serious exploration of weird and/or big moral questions. However, modern utilitarianism may be an approximation, like Newton’s theory of gravity (except with a lot more ambiguity in its definitions), and the equivalent of general relativity may be centuries away. It also seems certain that most of the richness of the topic still eludes us.

Indirect arguments: what people think, and the history of ethics

In addition to the theoretical arguments above, we can try to weigh utilitarianism indirectly.

First, we can see what people think (we are talking about morality after all – if everyone hates it, that’s cause for concern). On one hand, out of friends with who I’ve talked about these topics with (the median example being an undergraduate STEM student), basically everyone favours some form of utilitarianism. On the other hand, a survey of almost a thousand philosophers found only a quarter accepting or leaning towards consequentialist ethics (slightly lower than the number of deontologists, and less than the largest group of a third of respondents who chose “other”). (However, two thirds endorse the utilitarian choice in the trolley problem, compared to only 8% saying not to switch (the rest were undecided).) My assumption is that a poll of everyone would find a significant majority against utilitarianism, but I think this would be largely because of the negative connotations of the word.

Second, we can look at history. A large part of what we consider moral progress can be summarised as a move to more utilitarian morality.

I am not an expert in the history of ethics (though I’d very much like to hear from one), but the general trend from rule- and duty-based historical morality to welfare-oriented modern morality seems clear. Consider perhaps the standard argument in favour of gay marriage: it’s good for some people and it hurts no one, so why not? Arguments do not get much more utilitarian. (Though of course, other arguments can be made with different starting points, for example a natural right to various freedoms.) In contrast the common counter-argument – that it violates the law of nature or god or at least social convention – is rooted in decidedly non-utilitarian principles. Whereas previously social disapproval was a sufficient reason to deny people happiness, today we assume a heavy, even insurmountable, burden of proof of any custom or rule that increases suffering on net.

A second trend in moral attitudes is often summarised as an “expanding moral circle”: granting moral significance to more and more entities. The view that only particular people of particular races, genders, or nationalities count as moral patients has come to be seen as wrong, and the expansion of moral patienthood to non-humans is already underway.

A concern for anything capable of experiencing welfare is built into utilitarianism. Utilitarianism also ensures that this process will not blow up to absurdities: rather than blindly granting rights to every ant, utilitarianism allows for the fact that the welfare of some entities deserves greater weight, and assures us there’s no need to worry about rocks.

It would be a mistake to say that our moral progress has been driven by explicit utilitarianism. Abolitionists, feminists, and civil rights activists had diverse moral philosophies, and the deontological language of rights and duties has played a big role. But consider carefully why today we value the rights and duties that we do, rather than those of past eras, and I think you’ll find that the most concise way to summarise the difference is that we place more value on welfare and preferences. In short, we are more utilitarian.

Two of the great utilitarian philosophers were Jeremy Bentham and John Stuart Mill, who died in the early and late 1800s respectively (today we have Peter Singer). On the basis of his utilitarian ethics, Bentham advocated for the abolition of slavery and capital punishment, gender equality, decriminalising homosexuality (an essay so radical at its time that it went unpublished for over a hundred years after Bentham’s death), and is especially known as one of the first defenders of animal rights. Mill also argued against slavery, and is especially known as an early advocate of women’s rights. Both were also important all-around liberals.

Nineteenth century utilitarians were good at holding moral views that were ahead of their time. I would not be surprised if the same were true today.

EA ideas 3: uncertainty

2.0k words (7 minutes) Posts in this series:FIRST | PREVIOUS | NEXT

Moral uncertainty is uncertainty over the definition of good. For example, you might broadly accept utilitarianism, but still have some credence in deontological principles occasionally being more right.

Moral uncertainty is different from epistemic uncertainty (uncertainty about our knowledge, its sources, and uncertainty over our degree of uncertainty about these things). In practice these often mix – uncertainty over an action can easily involve both moral and epistemic uncertainty – but since is-ought confusions are a common trap in any discussion, it is good to keep these ideas firmly separate.

Dealing with moral uncertainty

Thinking about moral uncertainty quickly gets us into deep philosophical waters.

How do we decide which action to take? One approach is called “My Favourite Theory” (MFT), which is to act entirely in accordance to the moral theory you think is most likely to be correct. There are a number of counterarguments, many of which involve around problems of how we draw boundaries between theories: if you have 0.1 credence in each of 8 consequentialist theories and 0.2 credence in a deontological theory, should you really be a strict deontologist? (More fundamentally: say we have some credence in a family of moral systems with a continuous range of variants – say, differing by arbitrarily small differences in the weights assigned to various forms of happiness – does MFT require we reject this family of theories in favour of ones that vary only discretely, since in the former case the probability of a particular variant being correct is infinitesimal?). For a defence of MFT, see this paper.

If we reject MFT, when making decisions we have to somehow make comparisons between the recommendations of different moral systems. Some regard this as non-sensical; others write theses on how to do it (some of the same ground is covered in a much shorter space here; this paper also discusses the same concerns with MFT that I mentioned in the last paragraph, and problems with switching to “My Favourite Option” – acting according to the option that is most likely to be correct, summed over all moral theories you have credence in).

Another less specific idea is the parliamentary model. Imagine that all moral theories you have some credence in send delegates to a parliament, who can then negotiate, bargain, and vote their way to a conclusion. We can imagine delegates for a low-credence theory generally being overruled, but, on the issues most important to that theory, being able to bargain their way to changing the result.

(In a nice touch of subtlety, the authors take care to specify that though the parliament acts according to a typical 50%-to-pass principle, the delegates act as if they believe that the percent of votes for an action is the probability that it will happen, removing the perverse incentives generated by an arbitrary threshold.)

As an example of other sorts of meta-ethical considerations, Robin Hanson compares the process of fitting a moral theory to our moral intuitions to fitting a curve (the theory) to a set of data points (our moral intuitions). He argues that there’s enough uncertainty over these intuitions that we should take heed of a basic principle of curve-fitting: keep it simple, or otherwise you will overfit, and your curve will veer off in one direction or another when you try to extrapolate.

Mixed moral and epistemic uncertainty

Cause X

We are probably committing a moral atrocity without being aware of it.

This is argued here. The first argument is that past societies have been unaware of serious moral problems and we don’t have strong enough reasons to believe ourselves exempt from this rule. The second is that there are many sources of potential moral catastrophe – there are very many ways of being wrong about ethics or being wrong about key facts – so though we can’t point to any specific likely failure mode with huge consequences, the probability that at least one exists isn’t low.

In addition to an ongoing moral catastrophe, it could be that we are overlooking an opportunity to achieve a lot of good for cheap. In either case there would be a cause, dubbed Cause X, which would be a completely unknown but extremely important way of improving the world.

(In either case, the cause would likely involve both moral and epistemic failure: we’ve both failed to think carefully enough about ethics to see what it implies, and failed to spot important facts about the world.)

“Overlooked moral problem” immediately invites everyone to imagine their pet cause. That is not what Cause X is about. Imagine a world where every cause you support triumphed. What would still be wrong about this world? Some starting points for answering this are presented here.

If you say “nothing”, consider MacAskill’s anecdote in the previous link: Aristotle was smart and spent his life thinking about ethics, but still thought slavery made sense.

Types of epistemic uncertainty

I use the term "epistemic uncertainty" because the concept is broader than just uncertainty over facts. For example, our brains are flawed in predictable ways, and dealing with this is different from dealing with being wrong or having incomplete information about a specific fact.

Flawed brains

A basic cause for uncertainty is that human brains make mistakes. Especially important are biases, which consistently make our thinking wrong in the same way. This is a big and important topic; the classic book is Kahneman’s Thinking, Fast and Slow, but if you prefer sprawling and arcane chains of blog posts, you’ll find plenty here. I will only briefly mention some examples.

The most important bias to avoid when thinking about EA may be scope neglect. In short, people don’t automatically multiply. It is the image of a starving child that counts in your brain, and your brain gives this image the same weight whether the number you see on the page has three zeros or six after it. Trying to reason about any big problem without being very mindful of scope neglect is like trying to captain a ship that has no bottom: you will sink before you move anywhere.

Many biases are difficult to counter, but occasionally someone thinks of a clever trick. Status quo bias is a preference for keeping things as they are. It can often be spotted through the reversal test. For example, say you argue that we shouldn’t lengthen human lifespans further. Ask yourself: should we then decrease life expectancy? If you think that we should have neither more nor less of something, you should also have a good reason for why it just so happens that we have an optimum amount already. What are the chances that the best possible lifespan for humans also happens to be the highest one that present technology can achieve?

Crucial considerations

A crucial consideration is something that flips (or otherwise radically changes) the value of achieving a general goal.

For example, imagine your goal is to end raising cows for meat, because you want to prevent suffering. Now say there’s a fancy new brain-scanner that lets you determine that even though the cow ends up getting chucked into a meat grinder, on average the cow’s happiness is above the threshold for when non-existence is preferable to existence (assume this is a well-defined concept in your moral system). Your morals are the same as before, but now they’re telling you to raise more cows for meat.

An example of a chain of crucial considerations is whether or not we should develop some breakthrough but potentially dangerous technology, like AI or synthetic biology. We might think that the economic and personal benefits make it worth the expense, but a potential crucial consideration is the danger of accidents or misuse. There might be another crucial consideration that it’s better to have the technology developed internationally and in the open, rather than have advances made by rogue states.

There are probably many crucial considerations that are either unknown or unacknowledged, especially in areas that we haven’t thought about for very long.

Cluelessness

The idea of cluelessness is that we are extremely uncertain about the impact of every action. For example, making a car stop as you cross the street might affect a conception later that day, and might make the difference between the birth of a future Gandhi or Hitler later on. (Note that many non-consequentialist moral systems seem even more prone to cluelessness worries – William MacAskill points this out in this paper, and argues for it more informally here.)

I’m not sure I fully understand the concerns. I’m especially confused about what the practical consequences of cluelessness should be on our decision-making. Even if we’re mostly clueless about the consequences of our actions, we should base them on the small amount of information we do have. However, at the very least it’s worth keeping in mind just how big uncertainty over consequences can be, and there are a bunch of philosophy paper topics here.

For more on cluelessness, see for example:

• Simplifying Cluelessness (an argument that cluelessness is an important and real consideration)
• an in-depth look at different forms of cluelessness
• the author of the previous paper discussing related ideas in a podcast (a transcript is available).

Reality is underpowered

Imagine we resolve all of our uncertainties over moral philosophy, iron out the philosophical questions posed by cluelessness, confidently identify Cause X, avoid biases, find all crucial considerations, and all that remains is the relatively down-to-earth work of figuring out which interventions are most effective. You might think this is simple: run a bunch of randomised controlled trials (RCTs) on different interventions, publish the papers, and maybe wait for a meta-analysis to combine the results of all relevant papers before concluding that the matter is solved.

Unfortunately, it’s often the case that reality is underpowered (in the statistical sense): we can’t run the experiments or collect the data that we’d need to answer our questions.

To take an extreme example, there are many different factors that affect a country’s development. To really settle the issue, we might make groups of, say, a dozen countries each, give them different amounts of the development factors (holding everything else fairly constant), watch them develop over 100 years, run a statistical analysis of the outcomes, and then draw conclusions about how much the factors matter. But try finding hundreds of identical countries with persuadable national leaders (and at least one country must have a science ethics board that lets this study go forwards).

To make a metaphor with a different sort of power: the answers to our questions (on what effects are the most important in driving some phenomenon, or which intervention is the most effective) exist, sharp and clear, but the telescopes with which we try to see them aren’t good enough. The best we can do is interpret the smudges we do see, inferring as much as we can without the brute force of an RCT.

This is an obvious point, but an important one to keep in mind to temper the rush to say we can answer everything if only we run the right study.

Conclusions?

All this uncertainty might seem to imply two conclusions. I support one of them but not the other.

The first conclusion is that the goal of doing good is complicated and difficult (as is the subgoal of having accurate beliefs about the world). This is true, and important to remember. It is tempting to forget analysis and fall back on feelings of righteousness, or to switch to easier questions like “what feels right?” or “what does society say is right?”

The second conclusion is that this uncertainty means we should try less. This is wrong. Uncertainties may rightly redirect efforts towards more research, and reducing key uncertainties is probably one of the best things we can do, but there’s no reason why they should make us reduce our efforts.

Uncertainty and confusion are properties of minds, not reality; they exist on the map, not the territory. To every well-formed question there is an answer. We need only find it.

 

Next post

EA ideas 2: expected value and risk neutrality

2.6k words (9 minutes) Posts in this series:
PREVIOUS | NEXT

The expected value (EV) of an event / choice / random variable is the sum, over all possible outcomes, of {value of outcome} times {probability of that outcome} (if all outcomes are equally likely, it is the average; if they’re not, it’s the probability-weighted average).

In general, a rational agent makes decisions that maximise the expected value of the things they care about. However, EV reasoning involves more subtleties than its mathematical simplicity suggests, in both the real world and in thought experiments.

Is a 50% chance of 1000€ exactly as good as a certain gain of 500€ (that is, are we risk-neutral?), or a 50% chance of 2000€ with a 50% chance of a 1000€ loss instead?

Not necessarily. A bunch of research (and common sense) says people put decreasing value on an additional unit of money: the thousandth euro is worth more than the ten-thousandth. For example, average happiness scales roughly logarithmically with per-capita GDP. The thing to maximise in a monetary tradeoff is not the money, but the value you place on money; with a logarithmic relationship, the diminishing returns mean that more certain bets are better than naive EV-of-money reasoning implies. A related reason is that people weight losses more than gains, which makes the third case look worse than the first even if you don’t assume a logarithmic money->value function.

However, a (selfish) rational agent will still maximise EV in such decisions – not of money, but of what they get from it.

(If you’re not selfish and live in a world where money can be transferred easily, the marginal benefit curve of efficiently targeted donations is essentially flat for a very long time – a single person will hit quickly diminishing returns after getting some amount of money, but there are enough poor people in the world that enormous resources are needed before you need to worry about everyone reaching the point of very low marginal benefit from more money. To fix the old saying, albeit with some hit to its catchiness: “money can buy happiness only (roughly) logarithmically for yourself, but (almost) linearly in the world at large, given efficient targeting”.)

In some cases, we don’t need to worry about wonky thing->value functions. Imagine the three scenarios above, but instead of euros we have lives. Each life has the same value; there’s no reasonable argument for the thousandth life being worth less than the first. Simple EV reasoning is the right tool.

Why expected value?

This conclusion easily invites a certain hesitation. Any decision involving hundreds of lives is a momentous one; how can we be sure of exactly the right way to value these decisions, even in simplified thought experiments? What’s so great about EV?

A strong argument is that maximising EV is the strategy that leads to the greatest good over many decisions. In a single decision, a risky but EV-maximising choice can backfire – you might take a 50-50 bet of saving 1000 lives and lose, in which case you’ll have done much worse than picking an option of certainly saving 400. However, it’s a mathematical fact that given enough such choices, the actual average value will tend towards the EV. So maximising EV is what results in the most value in the long run.

You might argue that we’re not often met with dozens of similar momentous decisions. Say that we’re reasonably confident the same choice will never pop up again, and certainly not many times; doesn’t the above argument no longer apply? Take a slightly broader view though, and consider which strategy gets you the most value across all decisions you make (of which there will realistically be many, even if no single decision occurs twice): the answer is still EV maximisation. We could go on to construct crazier thought experiments – toy universes in which only one decision ever occurs, for example – and then the argument really begins to break down (though you might try to save it by some wild scheme of imagining many hypothetical agents faced with the same choice and consider a Kantian / rule-utilitarian principle of deciding by answering the question of which strategy would be right if it were the one adopted across all countless hypothetical instances of this decision).

There are other arguments too. Imagine 1000 people are about to die of a disease, and you have to decide between a cure that will certainly cure 400 versus an experimental one that will either cure everyone or save no-one. Imagine you are one of these people. In the first scenario, you have a 40% chance of living; in the second, a 50% chance. Which would you prefer?

On a more mathematical level, von Neumann (an all-around polymath) and Morgenstern (co-founder of game theory with von Neumann) have proved that under fairly basic assumptions of what is rational behaviour, a rational agent acts as if they’re maximising the EV of some preference function.

Problems with EV

Diabolical philosophers have managed to dream up many challenges for EV reasoning. For example, imagine there’s two dollars on the table. You toss a coin; if it’s heads you take the money on the table, if it’s tails the money on the table doubles and you toss again. You have a 1/2 chance of winning 2 dollars, 1/4 chance of winning 4, 1/8 chance of winning 8, and so on, for a total EV of 1/2 x 2 + 1/4 x 4 + … = 1 + 1 + … . The sequence diverges to infinity.

Imagine a choice: one game of the “St. Petersburg lottery” described above, or a million dollars. You’d be crazy not to pick the latter.

Is this a challenge to the principle of maximising EV? Not in our universe. We know that whatever casino we’re playing at can’t have an infinite amount of money, so we’re wise to intuitively reject the St. Petersburg lottery. (This section on Wikipedia has a very nice demonstration of why, even if the casino is backed by Bill Gates’s net worth, the EV of the St. Petersburg game is less than $40.)

The St. Petersburg lottery isn’t the weirdest EV paradox by half, though. In the Pasadena game, the EV is undefined (see the link for a definition, analysis, and an argument that such scenarios are points against EV-only decision-making). Nick Bostrom writes about the problems of consequentialist ethics in an infinite universe (or a universe that has a finite probability of being infinite) here.

There’s also the classic: Pascal’s wager, the idea that even if the probability of god existing is extremely low, the benefits (an eternity in heaven) are great enough that you should seek to believe in god and live a life of Christian virtue.

Unlike even Bostrom’s infinite ethics, Pascal’s wager is straightforwardly silly. We have no reason to privilege the hypothesis of a Christian god over the hypothesis – equally probable given the evidence we have – that there’s a god who punishes us exactly for what the Christian god rewards us for, or that god is a chicken and condemns all chicken-eaters to an eternity of hell. So even if you accept the mathematically dubious multiplication of infinities, Pascal’s wager doesn’t let you make an informed decision one way or another.

However, the general format of Pascal’s wager – big values multiplied by small probabilities – is the cause of much of EV-related craziness, and dealing with such situations is a good example of how naive EV reasoning can go wrong. The more general case is often referred to as Pascal’s mugging, and exemplified by the scenario (see link) where a mugger threatens to torture an astronomical amount of people unless you give them a small amount of money.

Tempering EV extremeness with Bayesian updating

Something similar to Pascal’s mugging easily happens if you calculate EVs by multiplying together very rough guesses involving small probabilities and huge outcomes.

The best and most general approach to these sorts of issues is laid out here.

The key insight is to remember two things. First, every estimate is a probability distribution: if you measure a nail or estimate the effectiveness of a charity, the result isn’t just your best-guess value, but also the uncertainty surrounding it. Second, Bayesian updating is how you change your estimates when given new evidence (and hence you should pay attention to your prior: the estimate you have before getting the new information).

Using some maths detailed here, it can be shown that if your prior and measurement both follow normal distributions, then your new (Bayesian) estimate will be another normal distribution, with a mean (=expected value) that is an average of the prior and measurement means, weighted by the inverse variance of the two distributions. (Note that the link does it with log-normal distributions, but the result is the same; just switch between variables and their logarithms.)

Here’s an interactive graph that lets you visualise this.

The results are pretty intuitive. Let’s say our prior for the effectiveness of some intervention has a mean of zero. If we take a measurement with low variance, our updated estimate probability distribution will shift most of the way towards our new measurement, and its variance will decrease (it will become narrower):

Red is the probability distribution of our prior estimate. Green is our measurement. Black is our new belief, after a Bayesian update of our prior with the measurement. Dotted lines show the EV (=average, since the distributions are symmetrical) for each probability distribution. You can imagine the x-axis as either a linear or log scale.

If the same measurement has greater variance, our estimates shift less:

And if we have a very imprecise measurement – for example, we’ve multiplied a bunch of rough guesses together – the estimate barely shifts even if the estimate is high:

Of course, we can argue about what our priors should be – perhaps, for many of the hypothetical scenarios with potentially massive benefits (for instance concerning potential space colonisation in the future), the variance of our prior should be very large, in which case even highly uncertain guesses will shift our best-guess EV a lot. But the overall point still stands: if you go to your calculator, punch in some numbers, and conclude you’ve discovered something massively more important than anything else, it’s time to think very carefully about how much you can really conclude.

Overall, I think this is a good example of how a bit of maths can knock off quite a few teeth from a philosophical problem.

(Here’s a link to a wider look at pitfalls of overly simple EV reasoning with a different framing, by the same author as this earlier link. And here is another exploration of the special considerations involved with low-probability, high-stakes risks.)

Risk neutrality

An implication of EV maximisation as a decision framework is risk neutrality: when you’ve measured things in units of what you actually care about (e.g. converting money to the value it has for you as discussed above), you should be neutral about the choice between 10% chance of 10 value units and 100% chance of 1, and you really should prefer a 10% chance of 11 “value units” over a 100% chance of 1 “value unit”, or a 50-50 bet between losing 10 and gaining 20 over a certain gain of 14.

This is not an intuitive conclusion, but I think we can be fairly confident in its correctness. Not only do we have robust theoretical reasons for using EV, but we can point to specific bugs in our brains that makes us balk at risk-neutrality: biases like scope neglect, which makes humans underestimate the difference between big and small effects, or loss aversion, which makes losses more salient than gains, or a preference for certainty.

$$$%%IF YOU SEE DOLLAR SIGNS IN THE NEXT SECTION, EQUATION RENDERING VIA MATHJAX IS NOT WORKING IN YOUR BROWSER$$$

Stochastic dominance (an aside)

Risk neutrality is not necessarily specific to EV maximisation. There’s a far more lenient, though also far more incomplete, principle of rational decision making that goes under the clumsy name of “stochastic dominance”: given options $$A$$ and $$B$$, if the probability of a payoff of $$X$$ or greater is more under option $$A$$ than option $$B$$ for all values of $$X$$, then $$A$$ “stochastically dominates” option B and should be preferred. It’s very hard to argue against stochastic dominance.

Consider a risky and a safe bet; to be precise, call them option $$A$$, with a small probability $$p$$ of a large payoff $$L$$, and option $$B$$, with a certain small payoff $$S$$. Assume that $$pL > S$$, so EV maximising says to take option $$A$$. However, we don’t have stochastic dominance: the probability of getting a small amount of value $$v$$ ($$v < S$$) is greater with $$B$$ than $$A$$, whereas the probability of getting a large amount of value ($$S < v < L$$) is greater with option $$A$$.

The insight of this paper (summarised here) is that if we care about the total amount of value in the universe, are sufficiently uncertain about this total amount, and make some assumptions about its distribution, then stochastic dominance alone implies a high level of risk neutrality.

The argument goes as follows: we have some estimate of the probability distribution $$U$$ of value that might exist in the universe. We care about the entire universe, not just the local effects of our decision, so what we consider is $$A + U$$ and $$B + U$$ rather than $$A$$ and $$B$$. Now consider an amount of value $$v$$. The probability that $$A + U$$ exceeds $$v$$ is the probability that $$U > v$$, plus the probability that $$(v - L) < U < v$$ and $$A$$ pays off $$L$$ (we called this probability $$p$$ earlier). The probability that $$B + U$$ exceeds $$v$$ is the probability that $$U > v - S$$.

Is the first probability greater? This depends on the shape of the distribution of $$U$$ (to be precise, we’re asking whether $$P(U > v) + p P(v - L < U < v) > P(U > v - S)$$, which clearly depends on $$U$$). If you do a bunch of maths (which is present in the paper linked above; I haven’t looked through it), it turns out that this is true for all $$v$$ – and hence we have stochastic dominance of $$A$$ over $$B$$ – if the distribution of $$U$$ is wide enough and has a fat tail (i.e. trails off slowly as $$v$$ increases).

What’s especially neat is that this automatically excludes Pascal’s mugging. The smaller the probability $$p$$ of our payoff is, the more stringent the criteria get: we need a wider and wider distribution of $$U$$ before $$A$$ stochastically dominates $$B$$, and at some point even the most stringent Pascalian must admit $$U$$ can’t plausibly have that wide of a distribution.

It’s far from clear what $$U$$’s shape is, and hence how strong this reasoning is (see the links above for that). However, it is a good example of how easily benign background assumptions introduce risk neutrality into the problem of rational choice.

Implications of risk neutrality: hits-based giving

What does risk neutrality imply about real-world altruism? In short, that we should be willing to take risks.

A good overview of these considerations is given in this article. The key point:

[W]e suspect that, in fact, much of the best philanthropy is likely to fail.

For example, GiveWell thinks that Deworm the World Initiative probably has low impact, but still recommends them as one of their top charities because there’s a chance of massive impacts.

Hits-based giving comes with its own share of problems. As the article linked above notes, it can provide a cover for arrogance and make it harder to be open about decision-making. However, just as high-risk high-reward projects make up a disproportionate share of successes in scientific research and entrepreneurship, we shouldn’t be surprised if the bulk of returns on charity comes from a small number of risky bets.

 

Next post

Review: Permutation City (Greg Egan)

Book: Permutation City, by Greg Egan (1979).
2.2k words (≈ 8 minutes)

One way to classify science fiction works is by the scope of the speculative concepts in the work.

For example, the first tier could contain works in which the only speculative elements are things with non-Earth-shattering consequences. Maybe dinosaur DNA could somehow remain intact for over sixty million years. Maybe an evil organization is plotting to create a pandemic.

Then there would be works in which the speculative element is something paradigm-shifting. What if humans made contact with aliens? What would an artificial intelligence do? What if genetic engineering were cheap and widespread?

The level after that would be works that ask similar questions, but go deeper into their consequences, especially by exploring what they say about human nature. How do you tell whether reality is simulated? Can humans even understand sufficiently advanced aliens? What does the possibility of artificial intelligence say about consciousness?

And the last tier is works in which the whole point is speculating about the ultimate nature of the universe itself. Isaac Asimov’s short story "The Last Question" is a classic example. Greg Egan’s 1994 novel Permutation City is another.

(It is hard to limit spoilers in this review, since the plot of the book is very tightly wound to the questions it explores. You have been warned.)


Copies everywhere

The first paragraphs are as pedestrian as it gets: our protagonist, Paul Durham, wakes up in a room and looks around.

Oh, and Durham is inside a computer (literally, though not too literally). He is a “copy”; the “original” had a brain scan made of himself, and started running that in a computer. The simulated reality isn’t an exact copy: only brains are simulated in any detail, while the rest of the environment is an approximation, though a photorealistic one, to save on running costs. Even on economy mode, though, copies in the 2050 world of Permutation City run at best at less than one tenth the speed of the real world.

In the novel’s world, many rich clients have their brains scanned before their biological death, and the copies started after they die. However, they cannot retreat into their virtual worlds, since copies can be affected by real-world events, particularly because legally, they are software, not people.

The wealthiest copies run on private computers managed and paid for by a trust fund. Less wealthy ones run much slower, and their running speed depends on the price of computing power changes, which is traded on a global market (note that the book was published in the 1990s; cloud-based computing power as a service was probably not a very common idea).

What this means for the world of Permutation City is that in addition to private copies running relatively fast, there are also virtual slums of slow-running copies that can afford computing power only when it’s cheapest, and cannot generate new income because their slow running speed (tens or hundreds of times less than the real-world) makes them useless for most jobs. A subculture of poorer copies, calling themselves the "Solipsist Nation", tries to reject external reality completely.

Egan’s bleak vision of copy inequality is not one I have encountered before, and one that seems a bit too credible for comfort.

All the standard brain emulation -related questions are also given some space. To what extent is a copy, based on a brain scan done some time before a person’s death, really a continuation of the life of that person? What about the legal and moral status of a copy of a copy? In general, Egan keeps the sledgehammer on the wall when exploring moral questions, leaving readers to draw their own conclusions and instead focusing on characters’ reactions and attitudes to these questions.


Assemble from the dust?

Egan does, however, present a sustained speculative argument about the nature of copies and therefore consciousness. He presents a thought experiment, which, within the novel, is a literal experiment on thought.

Durham’s copy is conscious. That much can be granted; if consciousness is a byproduct of brain activity, there is no reason for it to not be present in a simulation of the brain.

(Egan is fairly conservative with the technical specifications of the copies; it is mentioned that the brain simulation is not accurate down to the quantum level, and that time for a copy proceeds in discrete time intervals of one millisecond of subjective time. Some might argue that continuous time and/or quantum-level simulation might be necessary for consciousness, but this is a topic that Egan wisely avoids.)

Next, the speed of copy-Durham’s subjective time can be slowed down and sped up by will, simply by changing the rate at which the simulation computes itself. This can be taken even further - if Durham’s copy were simulated only in bursts once every day, or at random intervals, or one frame today and the next frame in a thousand years, copy-Durham would feel no difference.

We can easily stretch the relative times of copies and the real world, but can we break any connection whatsoever between the two experiences of time? Permutation City assumes yes: the frames of the copy’s simulation can be sliced up and rearranged, and the thread of subjective experience will still continue from the perspective of the copy as if nothing were happening even when the simulation is hopping back and forth from one time to another when viewed from the external world.

This is the critical step, and, I think, the weak link. To be able to slice up the copy’s time, the simulation must be able to set itself to, say, the frame at time t=2, and then later to the frame at t=1.

But how can the simulation know the contents of the frame at t=2? It must first compute all preceding frames. After it has done so, of course, there is no obstacle to the simulation loading one frame into memory as the current state, then another and another, all out of order. But is this arbitrary procedure of loading frames into the simulation’s memory what causes the experience of consciousness for the person being simulated? Or would the initial computation of the states be the key?

At one point copy-Durham wonders what his subjective experience of consciousness really is: the current time slice loaded in the simulation, the computation of those time slices, or something else? The novel’s answer seems to the first option.

This raises some very interesting questions. If we reject this premise, the other alternative seems to be that it is the process of computation of frames that causes them to “occur” in the simulation, at least from the perspective of conscious beings in the simulation. This is an interesting topic, and raises many questions. Might the real world, then, be thought of as a computational procedure, in the sense that it is the “computation” of the next moment that makes it happen?

If that sounds like too much to accept, consider what it would mean for the view in the book to be correct: consciousness can thread its way through the disconnected slices of the simulation and therefore the subjective time within the simulation is entirely independent of real-world time. Egan then adds a dose of solipsism: the thread of consciousness of the simulation is real, for the experiencer, despite the fact that it has nothing to do with the “real” time of the outside universe. It has, in other words, somehow “assembled itself from dust”, as Egan likes to repeat, including the italics.

If copy-Durham can assemble itself from the dust despite the time slices of his simulation being scrambled, then why couldn’t other things assemble themselves from the dust?

Extrapolate this further. Imagine a universe of nothing more than an arbitrarily large space of random fluctuations; some of them would, by chance, form sequential, coherent timelines containing conscious entities, which would then be experienced by those entities. Our reality, in the solipsist universe of Permutation City, would just be one of these sequences.

And that’s only the first fourth of the book.


Immortality = cellular automata + solipsist cosmology

What applications could the idea of assembling from the dust ever have? Eternal life and near-omnipotence, apparently.

The other main character, Maria Deluca, is a software engineer who spends a lot of the time she should be working on playing around with the “Autoverse”, a massively complex cellular automata with its own system of chemistry that mirrors real-world chemistry, except without quantum effects (I get the feeling Egan is not fond of quantum physics). Egan spends a lot of pages on the Autoverse, but it is worth it; I found myself wishing for a real one.

After Maria gains some success with getting Autoverse bacteria to mutate, Durham enlists her to design a program to produce an entire planet, complete with primitive bacteria, in the Autoverse. There is not nearly enough computing power, even in Egan’s world of 2050, to run an entire Autoverse planet, but that’s not the point.

Durham’s idea is to simulate the first few minutes of a self-replicating cellular automata computer on a computer, and then stop. The continuation of the self-replicating computer represents a coherent timeline and by the logic of the novel’s solipsist universe, it will simply assemble itself from the dust and continue to exist from the perspective of the wealthy clients who paid to have their copies put on the thing. Thus Durham, and his clientele of billionaires, escape our reality into an alternative universe consisting of an ever-expanding computer that simulates their copies in addition to the Autoverse planet.


Solipsism!

If this weren’t speculative enough, Egan turns the solipsism up to eleven in the second part of the book.

Without revealing too much, the basic idea is that the Autoverse planet has developed intelligent life, which has its own theories for the origin of their universe that do not include being a simulation inside a simulation that was launched by a simulation made by a crackpot theorist and a dozen billionaires hoping for eternal life.

Reality in the universe (or should I say, space of random states) of Permutation City is a subjective thing, and so the logically coherent theories of the simulated lifeforms eventually become more real than the version of reality Durham and the other copies believe in, with destabilizing effects on their apparently-not-quite-eternal universe.

Brain emulation is already a topic with plenty of philosophical questions to explore. Egan, though, is not content with remaining in that territory, and instead takes the reader on a philosophical roller coaster through the consequences of ever wilder and wilder solipsism.


I was told you have to mention literary features when discussing literature …

… but Permutation City was written more for its concepts than its literary merit.

Egan does portray a reasonably diverse cast of characters. We have an eccentric and determined theorist, a software engineer with a terminally ill mother and time-consuming hobbies she cannot bring herself to quit, a remorseful billionaire struggling with past crimes, and a survival-oriented virtual slum -dweller. Many of them struggle in a genuine way with questions of identity and morality in the copy-filled world of Permutation City, and some scenes were touching, but none of the characters were particularly memorable.

Many of Egan’s chapters (not all are named) have names that are anagrams of "Permutation City", an allusion to the slicing of copy-Durham’s simulation. “Remit not paucity” is the most common chapter name. As far as I can tell, it seems to be a warning against trying to eliminate all scarcity from life, as Durham’s flawed universe does. There is also a disconcerting heavily anagrammatic poem at the front of the book, indirectly attributed to the main character Paul Durham. If it has meaning besides building atmosphere, I can’t figure it out.


Meaningful answers?

Permutation City is far from the only work of science fiction to explore esoteric philosophical themes. Peter Watts’ Blindsight (main point: how useful is consciousness; what if a space-faring civilization did not have it?) and Neal Stephenson’s Anathem (main point: uhhh …) also deal with the philosophical questions surrounding consciousness.

However, Permutation City is exceptional in the extent and scope of its speculation. It is also structured well in this regard; Egan gradually ramps up the level of speculation throughout the work, allowing the reader to update their knowledge of how the novel’s world works after the introduction of each speculative leap, and helping to maintain immersion by showing the internal consistency. It also, probably not coincidentally, lays bare Egan’s chain of reasoning, exposing it to readers for easy analysis. The book definitely succeeds in provoking questions.

As to whether the book’s big ideas are anywhere close to being correct, I think Isaac Asimov’s fictional computer in The Last Question put it best:

"THERE IS AS YET INSUFFICIENT DATA FOR A MEANINGFUL ANSWER"