---
date: Thursday, October 15^th^, 2026
subtitle: On the role of scientists in the future of AI technology
title: Science and Safety
---

History shows that achieving a high degree of control over the behaviour
of our inventions requires *deep scientific understanding* of the
systems involved. The long march of scientific progress in mathematics,
physics, materials science, chemistry, biology, computer science,
neuropsychology, and economics has repeatedly contributed foundational
understanding of this kind. In turn, this foundational understanding
enables much of the robustness currently exhibited by the physical,
medical, and digital infrastructure upon which today's civilisation
relies. There are, of course, many grand scientific challenges remaining
to be solved. And so, we march on.

Humanity's understanding of modern AI systems could not be described as
deep. The designers of frontier AI systems openly disclaim their lack of
ability to understand how their systems function, let alone
intentionally design the behaviours of these systems with any degree of
robustness. It is therefore no surprise that these systems lack robust
safety properties leading to unintended and dangerous behaviours from
sycophancy and social media meltdowns to swarming behaviours and
industrial reward hacking.

Today's AI systems have already begun to reach human expert or
superhuman levels of capability within technical domains such as
software development, machine learning research, cybersecurity, and
mathematics. It seems that AI capabilities will continue to advance in
these and other domains. With each such advance, the scale of potential
harm associated with system misbehaviour or misuse becomes more severe.
Many AI experts believe future possible harms include global catastrophe
or even human extinction.

Understanding systems of this complexity and scale is certainly a grand
scientific challenge. Nevertheless, there are hints of structure
underlying AI learning and behaviour, suggesting that deeper
understanding is possible. Consider the striking results of mechanistic
interpretability, or the scaling laws that hold from small-scale models
all the way to the frontier. One can imagine understandable scientific
laws connecting structured learning signals derived from real-world data
and environments to predictable structure in whatever internal cognitive
architecture generates the eventual system's behaviour.

In normal times, this research agenda would be enough to occupy a whole
field for a generation or more. However, we do not live in normal times.
We live in a time where there is an urgent and desperate need to solve
this grand challenge as soon as possible while we still have the chance.
We also live in a time where radical ambition is rewarded, since a
suitably ambitious community of researchers can accelerate their
progress by thoughtfully leveraging the latest advances in AI itself.

Armed with such laws, humanity would have a fighting chance of safely
navigating the future of AI. Our insights could help us to foresee and
forestall the development of dangerous capabilities or adversarial
objectives in our systems. Humanity could then carefully and
deliberately engineer a controlled disruption of the current
civilisational equilibrium. We could allow advanced AI to contribute to
breakthroughs in other grand scientific and social challenges facing
humanity. But the *most urgent,* and, partly for that reason, the
*grandest* scientific challenge we face is to understand these AI
systems.