---
date: Friday, May 23^rd^, 2024
title: Ethics and the Future of Intelligence
---

[COMP90087 The Ethics of Artificial
Intelligence](https://handbook.unimelb.edu.au/2024/subjects/comp90087)
is a Master's subject at the University of Melbourne designed as an
introduction to moral philosophy and applied ethics for students with a
technical background. It offers students who will go on to contribute to
building AI systems of the future a framework for evaluating,
critiquing, and designing AI technology in a socially responsible
manner.

While there are many important social and ethical issues raised by
today's AI technologies (comprising much of the content of this
subject), it is *also* true that as we project developments in AI
technology into the future, we can foresee new and different ethical
issues that might arise.

Accordingly, I was invited to give a guest lecture on ethical questions
raised by potential future advancements in AI for the final week of
COMP90087, 2024.

## Lecture recording

Here's a public version of the lecture recording. Download the lecture
slides ([PDF](future-ethics-lecture.pdf)).

```{=html}
<iframe src="https://www.youtube.com/embed/-DvCQAiX2QA" width="100%" height="340" frameborder="0" allowfullscreen>
```
```{=html}
</iframe>
```
## Learning outcomes

**At the end of this module, you should be able to:**

1.  Understand and describe the concepts of narrow AI, general AI, and
    superintelligence.

2.  Understand and describe potential positive and negative consequences
    of building a superintelligent AI system and "the alignment
    problem."

3.  Apply established ethical theories to reason about the question:
    "should society build a superintelligent AI system?"

## Readings

Note: Reading (3) is just four pages long!

With the exception of readings (2) and (7), all of these readings are
available from the publisher's website, though you may require an
institutional affiliation to access them.

**On the possibility of human-level AI (and beyond):**

1.  Alan M. Turing, 1950, "Computing machinery and intelligence".
    *Mind,* LIX(236): 433--460. DOI:
    [10.1093/mind/LIX.236.433](https://doi.org/10.1093/mind/LIX.236.433).

    The classic paper from Turing that proposes the "imitation game"
    (known today as the "Turing test"), a benchmark for AI systems to
    communicate in a convincingly human manner, and confronts several
    objections to the possibility of this test being passed by a future
    AI system.

2.  Nick Bostrom, 2014, *Superintelligence: Paths, Dangers, Strategies.*
    Oxford University Press.

    A book-length treatment from Bostrom, a philosopher and futurist, on
    the possibility of implementing a superintelligent AI system (and
    some possible consequences).

**On risks arising from advanced AI systems:**

3.  Norbert Wiener, 1960, "Some moral and technical consequences of
    automation". *Science* 131, 1355--1358. DOI:
    [10.1126/science.131.3410.1355](https://doi.org/10.1126/science.131.3410.1355).

    A short and sweet essay from Norbert Wiener, the pioneer of
    cybernetics and a deep thinker about the relationship between
    technology and society, in which Wiener points out that future
    advanced AI systems may become difficult to control.

4.  Eliezer Yudkowsky, 2008, "Artificial intelligence as a positive and
    negative factor in global risk". *Global Catastrophic Risks*,
    308--345. DOI:
    [10.1093/oso/9780198570509.003.0021](https://doi.org/10.1093/oso/9780198570509.003.0021).

    Decades later, Eliezer Yudkowsky looked around at modern AI systems
    and thought, it's time to take this control problem seriously, the
    consequences could be severe. This chapter contains some of
    Yudkowsky's extensive writings on this topic.

5.  Andrew Critch and David Krueger, 2020, "AI research considerations
    for existential safety". Preprint
    [arXiv:2006.04948](https://arxiv.org/abs/2006.04948).

    More recently, Critch and Krueger put together this extensive
    research agenda that lays out a more robust case for risks from
    advanced AI systems.

**On ethical questions raised by the advent of advanced AI systems:**

6.  Nick Bostrom and Eliezer Yudkowsky, 2014, "The ethics of artificial
    intelligence". *The Cambridge Handbook of Artificial Intelligence*
    316--334. Cambridge University Press. DOI:
    [10.1017/CBO9781139046855.020](https://doi.org/10.1017/CBO9781139046855.020).

    A complement to the lecture, some additional ethical questions that
    arise in a world with AI systems with human-level or super-human
    intelligence.

**On the broader topic of ethics and existential risk (not just from AI
systems):**

7.  Toby Ord, 2020, *The Precipice: Existential Risk and the Future of
    Humanity*. Bloomsbury.

    Toby Ord was once a student in computer science and philosophy at
    the University of Melbourne. Ord since studied philosophy at the
    University of Oxford and has contributed to analysing ethical
    questions about the long-term future of humanity. This book gives
    Ord's analysis of the science of extinction risk from various
    sources.