History shows that achieving a high degree of control over the behaviour of our inventions requires deep scientific understanding of the systems involved. The long march of scientific progress in mathematics, physics, materials science, chemistry, biology, computer science, neuropsychology, and economics has repeatedly contributed foundational understanding of this kind. In turn, this foundational understanding enables much of the robustness currently exhibited by the physical, medical, and digital infrastructure upon which today’s civilisation relies. There are, of course, many grand scientific challenges remaining to be solved. And so, we march on.

Humanity’s understanding of modern AI systems could not be described as deep. The designers of frontier AI systems openly disclaim their lack of ability to understand how their systems function, let alone intentionally design the behaviours of these systems with any degree of robustness. It is therefore no surprise that these systems lack robust safety properties leading to unintended and dangerous behaviours from sycophancy and social media meltdowns to swarming behaviours and industrial reward hacking.

Today’s AI systems have already begun to reach human expert or superhuman levels of capability within technical domains such as software development, machine learning research, cybersecurity, and mathematics. It seems that AI capabilities will continue to advance in these and other domains. With each such advance, the scale of potential harm associated with system misbehaviour or misuse becomes more severe. Many AI experts believe future possible harms include global catastrophe or even human extinction.

Understanding systems of this complexity and scale is certainly a grand scientific challenge. Nevertheless, there are hints of structure underlying AI learning and behaviour, suggesting that deeper understanding is possible. Consider the striking results of mechanistic interpretability, or the scaling laws that hold from small-scale models all the way to the frontier. One can imagine understandable scientific laws connecting structured learning signals derived from real-world data and environments to predictable structure in whatever internal cognitive architecture generates the eventual system’s behaviour.

In normal times, this research agenda would be enough to occupy a whole field for a generation or more. However, we do not live in normal times. We live in a time where there is an urgent and desperate need to solve this grand challenge as soon as possible while we still have the chance. We also live in a time where radical ambition is rewarded, since a suitably ambitious community of researchers can accelerate their progress by thoughtfully leveraging the latest advances in AI itself.

Armed with such laws, humanity would have a fighting chance of safely navigating the future of AI. Our insights could help us to foresee and forestall the development of dangerous capabilities or adversarial objectives in our systems. Humanity could then carefully and deliberately engineer a controlled disruption of the current civilisational equilibrium. We could allow advanced AI to contribute to breakthroughs in other grand scientific and social challenges facing humanity. But the most urgent, and, partly for that reason, the grandest scientific challenge we face is to understand these AI systems.