This reflection note started life on June 1st, 2019, which marked ten years since I joined the AI safety movement, just six months before the now infamous 2009 AI Safety Summit in Puerto Rico. Then it reflected on a decade, and now, five years further on, it reflects on the last 15 years and revises and builds on some of the lessons I have taken from that period.
Back in 2009, I was an AI safety newbie. What I knew about was political communications. I came from the private sector, the tech industry primarily, and I had never worked in civil society, for a philanthropy, or on AI safety issues. The only thing I had a clear sense of was that I was walking into a massive learning experience, though even knowing that, I couldn’t have predicted quite how much the next 15 years would teach me, including how many of my priors about alignment would get relitigated by the 2023 releases.
I wanted to take a moment to reflect a little on some of the key lessons I have learned. They are not comprehensive, and they are not absolute, but I hope they may be of some use to the community as we enter what truly is now the last few years in which we can shape the trajectory of advanced AI before the window closes on the most catastrophic outcomes. They are:
- We will not win because we are right, we will win because we are organised.
- Good policy doesn’t adopt itself, it’s about politics.
- Don’t mistake access for influence.
- Don’t see progress as linear, no governance fight ever ends.
- We often behave like we don’t have an opposition.
- There is no “correct” next step or “right” intervention
- We have a matter of years to prevent catastrophe. Let’s act like it.
- The field is not a series of projects. It must be an ecosystem of change.
- Money talks
- Never allow an institution’s existence to become its primary objective.
- Decisiveness in times of uncertainty is vital.
- We have more friends than we have enemies – but that’s not enough
- Scale.
- Advanced AI presents an unprecedented opportunity to bring everybody with us.
1. We will not win because we are right, we will win because we are organised.
At their core, many of the theories of change adopted by our community are still based on the misconception that we will win because we are right. This view is deeply embedded: How often have you been in a meeting where someone suggests we need to deepen the evidence base, or communicate the results of a study more effectively, or try and get people to understand the trade-offs inherent in a given forecast of the path to safe advanced AI? These approaches suppose that we will win because we are right, and the more right we become, the more winning we will do.
Now, it is completely understandable to believe that we are right, and that if we explain to everyone else in enough detail, and in language they can relate to, why, they will understand and agree – not only with our analysis of the problem but also with our proposed solution. Anyone who has spent a weekend on LessWrong has felt that pull.
But it doesn’t work. It has never worked, and it never will. We absolutely need detailed analysis to understand what it is that we are trying to do, and to inform our strategy. But, when it comes to the debate and the politics, a facts-first approach will not prevail. It’s more important to know your opposition, identify your allies, understand their incentives, work out what will open the window for the policy you need, and go at it.
I learned to understand the difference between changing perceptions and winning arguments. We don’t need to be proved right; we need to make other people believe they are right by wanting the same thing we do. It’s different, and no amount of well-calibrated graphs will close that gap.
I have seen that we win when we appreciate the need for a coordinated inside and outside game, that different actors have different roles and that we never know which approach will be decisive. We win when we support and adopt a multiplicity of approaches. In practical terms, this means we need to organise so that policy, politics, strategic communications and campaigning work together to deliver the result. No approach ever works in isolation, and a Berkeley group house full of brilliant researchers is not a substitute for a coalition.
2. Good policy doesn’t adopt itself, it’s about politics.
Although much of the disagreement about AI risk claims to be about facts, the fight is inherently political. There is next to no actual debate over whether capabilities are advancing quickly, and the case that frontier systems need serious oversight was largely settled by the time of the 2023 extinction-risk statement. What remains is mostly a fight over who bears the cost of acting. Though there are exceptions, it is rarely a question of whether we need one mechanism or another – we most often need them all: evals, licensing, incident reporting, compute thresholds, and the rest. We have very little time to solve what may be the highest-stakes coordination problem in human history.
The blockers to policy are not a lack of understanding (see my first point), they are a lack of power. Frontier labs, their investors, and accelerationists who treat any constraint as an existential insult have used that power to defeat or dilute safety legislation, sometimes within a single session. We must use power to enable the adoption of effective and ambitious governance. The calculation and understanding of political power, and the ability to exert it on multiple fronts, is what will decide the outcome. If we insist on remaining “above politics”, we will surely remain above winning. Politics is the system by which human society organises itself, and we need to reorganise it – we should not avoid that conclusion for fear of being ‘partisan’. We are engaged in a political endeavour whether we like it or not, and we need to embrace that.
3. Don’t mistake access for influence.
This happens a lot. The example I come across most often is when advocates working an inside-game, government-facing strategy tell comms people that they shouldn’t publish anything criticising the AI Safety Institute official they are working with, even if that official is in the process of watering a compute-reporting proposal down to nothing, because it’s the only shot they have.
An effective inside game only ever works with an effective outside game. The outside game creates leverage, the inside game provides a graceful landing zone, and the wheel turns towards progress. Neither approach works alone, and we should do better in recognising that they are peers in the process of change. Policy folks shouldn’t discount the role of communications and campaigns, and nor should comms and campaigns folks discount the policy wonks. They are two critical pieces of the same system. This also needs to extend to our approach. Access is important, but it mustn’t prevent others from being critical, and naming names when necessary. Likewise, when a policy-maker does do something right, we must sing it from the tree-tops. We need to apply pressure where it’s needed and celebrate success where it occurs, including when a lab publishes a responsible scaling policy that actually has teeth.
4. Don’t see progress as linear, no safety fight ever ends.
We still have a tendency, particularly funders and strategists, to constantly ask what we can stop doing, as if we were marshalling a set of grants, rather than trying to shape the trajectory of advanced AI. I don’t think that’s the right way to look at the field. No battle is ever definitively won. If we scrape a bill through by one vote, it only takes one vote to gut it in committee. Our opposition will always be resourced to make the same tired old arguments about the costs of regulation and the dangers of 'stifling innovation'. It’s whack-a-mole and it always will be. What we need to do is learn from the opposition, which has learned this lesson.
While they keep us busy with benchmark disputes and pointless timeline arguments, our opponents also invest in people, lobbying infrastructure and long-term capacity, not just outcomes.
Building expertise and capacity and investing for the long term creates infrastructure we know we will always need – infrastructure that can adapt as capabilities shift and the compute landscape changes, and keep working through a decade of model releases.
It’s this kind of infrastructure that can go beyond the reversible ‘winning moment’ to embed a safety culture as part of the outcome of a policy win. By building long-term attention, expertise and engagement among legislators, lab staff and the wider public, we can change the frame of the debate and change the way key communities think about frontier risk. This is change that is resilient, locks in forward motion and wins for the long term.
5. We often behave like we don’t have an opposition.
There are moments where our advocacy decisions take for granted that the policy process functions, that the public interest is its primary driver, that labs will honour their own commitments, and that commentators are objective. This is, in itself, a form of denial. Take, for example, the US Congress. Political science has shown us empirically that it does not reliably favour the public interest, and the fate of SB 1047 showed how quickly a well-supported bill can be reshaped by a concentrated industry. The same will be true in many other places – perhaps all of them.
Our opposition is a carefully constructed web of equity stakes, ideology, political access, lobbying expertise and vast compute budgets. There is not a single central driver of the campaign against safety rules, but a set of overlapping interests and beliefs and a willingness by many actors to ignore what they disagree with in order to get part of what they want. It is implicitly co-ordinated and deeply powerful.
The question: “What will the opposition do next?” should be near the top of the list in all our planning and strategy meetings, alongside “How will the opposition respond to our latest strategic decision?”. But we rarely discuss these questions, and we often end up doing so only after a lab has already reframed our proposal as an innovation tax. We need to build proactive opposition management into the core of the way the community operates.
6. There is no “correct” next step or “right” intervention
Theory of change is not the right functional alignment; alignment happens around objectives. I believe there should be a set of principles that you sign up to when you join the AI safety movement. We have one in the Frontier Safety Coordination Circle and it has proved very useful, and I am taking the model into new programs at Thresholds Philanthropy.
One of those community principles should be that we don’t debate each other’s “theory of change”, we debate objectives. Let’s avoid endless circular conversations about whether evals or interpretability or control is the real path to safety, and focus discussion on what it is that we want to change and why. Everyone will come with their own view, based on their own experience and their own capacities and skills, and a lot of those debates are just people re-running their priors about alignment tractability at higher volume.
We should align around strategic objectives and let people get on with it. We should invest in people whose judgement we trust and vest decision-making in them, precisely because their approach may well be different to ours. By aiming for the same targets (and keeping them high level) we create space for diverse approaches, innovation and creativity, and still enable ourselves to scale by creating an effective peer network. We recognise our limits and know when there is a gap that someone else needs to fill.
This requires that the community incentivises collaboration and undertakes gap identification in a straightforward and low-drama way. But if we get it right, we relieve the pressure on everyone in the community to constantly pivot, chameleon-like, to fit the latest theory of change, and nobody has to re-litigate their life's work every time a new model drops.
7. We have a matter of years to prevent catastrophe. Let’s act like it.
We now face the final few years of the critical window for setting the trajectory of advanced AI. Over the next few years, we must make AI safety about governance and technical mitigation at a scale and pace that can prevent a loss of control over frontier systems, and also prepare for the many incidents we now know for certain are coming. Under those conditions, the calculation of p(doom) becomes a very different proposition, and anyone still running a fifteen-year research timeline is quietly assuming a world that may not exist.
Some will courageously go and sit outside the gates of gigawatt frontier training datacentres, lie down in front of new compute buildouts, chain their bikes to lab reception desks and otherwise escalate radical tactics. In the more “mainstream” work we do as a movement – investor pressure on lab valuations, creating narratives for the centre, empowering the voice of safety-minded engineers, backing new and different actors – we also need to embrace radical strategy. That won’t always mean radical tactics, but it must mean setting our priorities and allocating our resources against what we know needs to happen, rather than what we think can happen, and resisting the temptation to call a responsible scaling policy “a plan” when it is a set of thresholds we have not yet agreed to enforce.
We have left the world of the politics of the possible and entered the world of the politics of the necessary. Things will get broken, including some relationships with lab staff and the odd funder, and we need to be ready for that. Radical tactics are not necessarily a sign of radical strategy, and a protest outside an office is not a theory of change. We need to be clear eyed on what radical strategy is, how it is working, and the reality of our situation, which is that the labs are moving faster than our institutions.
8. The movement is not a series of projects. It must be an ecosystem of change.
The movement, and the organisations within it, have a tendency to silo very quickly. Almost for our own sanity, we lean into the instinct to put barriers and boundaries around the work we do. But, as we all know, everything is connected – particularly when it comes to governing a general-purpose technology that touches compute, evals, labour markets, biosecurity and export law all at once. The challenge for us all is that building connective tissue across our movement is hard and made even harder when advocates are measured by their individual impacts rather than their ability to collaborate and put impact above brand. Meaningful policy change almost always happens through collective effort, not via the individual heroics of any one person, lab or Alignment Forum thread.
Not everyone needs to work on everything and not everyone needs to be part of every decision – indeed that would paralyse us – but we do need to think smartly about how to structure around and otherwise deconstruct silos. One example that addresses this need directly is the Frontier Safety Coordination Circle, where a few people with a helicopter view, whose job is not to explain to everyone what they should be doing (nobody is that clever), spot where gaps are being left or a single issue such as eval funding is taking up too much attention, and collaboratively plan around that.
Leaders of organisations and those with influence inside the community need to be conscious of their power, and not try to get everyone to focus on whatever is their latest source of ire, grievance or the latest “silver bullet” theory, as that approach risks a form of whiplash of attention and resource, with half the field retraining on interpretability in March and on AI control by June. We must be aware our community is at its best when it functions as an ecosystem – including with a degree of overlap and redundancy that drives resilience and creativity, and enabled capacity that can help the community think forward – and embrace the many different strategies and diversity in people and approach.
9. Money talks
Funders, to a great extent, dictate what gets worked on. That’s OK, but only if they effectively coordinate, consult, listen and resist the tendency to chase whichever threat model is trending on LessWrong this quarter.
Nothing gets in the way of clear strategic thought like money. Not that funders don’t have very important insights to contribute, they do. But no funder has a monopoly on truth and righteousness, and in a resource-constrained field where the prospect of more resources has a massive gravitational pull, the reality is that you usually get most money out of most funders by agreeing with them. Hence funders tend to distort the strategy of the field, and it is a classic case of the epistemic trap where a grantee’s model of the world converges on the funder’s model of the world, which then looks like independent confirmation. There may be no way to avoid this, but what we can do is be aware.
Every funder has to appreciate and be conscious of the fact that they have “funder wings” – they get treated differently and protected from criticism, and a critique of an Open Philanthropy-funded org at an EA Global afterparty carries social costs that a critique of a random startup never would. We would serve ourselves better as a community if every grantee was enabled to go into those funder conversations with a peer-to-peer approach.
The field often looks like a buyer’s market where information only flows in one direction, with grantees peddling unique nuggets of strategic insight in a zero-sum competition for resources, and funders being cagey about their strategy and intentions. But everyone, funders and grantees alike, brings something unique and important to the table. The best response to the “funder-wings” issue is to approach all field interactions assuming that each one of those contributions is truly equally important, including the contributions of the 22-year-old MATS scholar who just noticed something about the eval suite that everybody else missed.
My father, who was a grant-maker in the arts for much of his professional life, gave me a piece of advice when I joined the European AI Safety Foundation many years ago. He said: “Your grantees will never love you, and they may not even like you, but that’s ok. They just need to trust you. Always tell everyone everything that’s going on, don’t hide the ball, be clear and transparent in your decision making, and they will trust you. Then you can work together.” It was sound advice.
10. Never allow an institutions’ approach or ongoing existence to become its primary objective.
In the business of making AI go well, beliefs and vision and commitment matter a lot. Institutions matter less. There is a very common urge that we have partially caught from the commercial world and, partially, is just a feature of the human desire for belonging: We invest heavily in symbols of who we are as people, and that includes for many of us the lab or nonprofit for which we work. We, understandably, want to feel our values align with others and are represented by our brand, something I experienced very directly when I worked for OpenAI, a company with a hugely powerful brand image and the loyalty in its workforce that can inspire
The question should not be whether X or Y organisation will exist next year, but is its existence important to strategy and to achieving our collective aims? Institutions are not what deliver safety, people are what deliver safety, and we need to be ready to deconstruct and rebuild the architecture and infrastructure of the movement as strategy requires. Building power involves building structures and betting on people.
This is challenging. Even in an unbranded network like the Frontier Safety Coordination Circle, which was in some sense founded on the principle that it was not and should never become an institution, we encounter it. This is another place where we need to invest in people, to provide reassurance that people can be brave, think beyond the limitations of their current positions and institutional boundaries and think beyond the limits of their carefully thought-through processes and procedures. If it’s not serving the objective, it needs to go. That’s where real creativity and impact lie, where ambition is de-risked and not limited by what we did last year, or by what our lab’s safety framework says we are permitted to say.
11. Decisiveness in times of uncertainty is vital.
Decisions are hard, and they are increasingly hard when the sheer enormity of the problem rises out of the mist in front of you and you don’t know where to start. But the decision to start is actually the most important one, and in these times, it often has to be made without a clear picture of where this will all end, or even whether timelines are five years or twenty. We have to get comfortable with the fact that the things we have taken for granted for so long are now under threat. We have to understand that the societal impacts of advanced AI are all around us and will play out in ways the forecasts won’t be able to predict. Humans don’t conform to the rules of scaling laws.
In the face of all of that anxiety and uncertainty, we must act. We won’t always make the right calls, and developments will happen that we did not foresee, but if we are real with ourselves about the stakes, and invest in people with clear vision and a capacity for calm reflection and decisiveness, we can and will move forward. The skill of decisiveness, balanced by humility and never straying into hubris or galaxy-brained reasoning about why the obvious move is wrong, should be the core of leadership recruitment.
12. We have more allies than we have opponents – but that’s not enough
We run a huge amount of public opinion polling (give me a shout if you want access to the brilliant AI Public Opinion Hub where it all lives and is made available to the community). We poll “the public”, we poll investors, we poll young people, parents, policymakers, nurses, and on and on. They are all relevant to the work we do in so many countries around the world, and we need to know what they think in order to change it. And what we see is that people get it, people are scared, people worry about the future, people read the writing on the wall, even if the wall is a frontier model card nobody has read past the first page of.
The vast majority of populations, segments and communities across the world are worried about advanced AI risk and want safety action. But we don’t see enough action! Why not? There is a gap between what “people” want and what happens inside the frontier labs and in the legislatures, and that gap is where power lies. Political power, the power to shape public opinion, the power to represent interests, economic and cultural power, and the power of a few lab valuations large enough to make a board nervous.
That space is where we can and must make a difference. It’s not the only piece, as I said earlier: policy, mass public campaigns, outside-game protests outside lab offices and so on are all important. But where I will continue to focus my efforts in the coming years is that gap. We have to build power, it doesn’t happen by itself, and we can’t bank on the wisdom of the crowd, or on a sufficiently well-calibrated forecast, to see us through.
13. Scale
It’s a big job. We need a lot of people and resources. But more than anything else we need to think big. If we can’t have sensible debates as societies because our information systems are full of lies and hate and owned by people who profit from polarisation, we need to change those information systems. If the media is owned by interests that vilify parts of society to undermine the structures that can deliver good outcomes, we need to change media ownership. If other issue communities, fighting for human rights or freedoms or equity or any of the other connected issues that will decide the future of our societies, need our help, we should help them. We can’t do everything, but everything does need to get done, so we need to think laterally and strategically about resource allocation, about strategy and about sequencing: compute thresholds first, evals second, and so on. But most of all, we need to think and act at scale. There is no right answer or perfect plan, but these few years are all we have, and we should try with everything we have.
14. The risks from advanced AI present an unprecedented opportunity to bring everybody with us.
Let’s be clear that we are not trying to save the world from a model. We are trying to create a fairly specific view of an ideal transition to advanced AI, one which, in reality, has never before existed.
Paramount to our thinking, in both communications and politics, must be this: a society that has the best hope of surviving a sharp takeoff will be one that has brought everybody with it. While the elite decisionmakers at the labs and in government hold the cards on policy to mitigate the risks, empowering an inclusive, just and equitable society to cope with them when they arrive will be every bit as important if we want human civilisation not just to survive, but to flourish.
In summary, it seems to me that we have to take a new approach to the next, last few years of the fight for a safe transition to advanced AI. We need to think bigger, focus on power and building it, bring people along, and not be afraid to bet on multiple approaches. If we build a community or ecosystem of capable people who trust each other and can move fast without worrying about whether others have their back, we have a chance of winning. We need to embrace the fact that radical is the new normal, we need to look ahead and plan for the next fight, not the last benchmark. And we need to take care of people and realise that the psychological effects of this period and its implications are profound, for all of us. We know that if we lose our best strategic talent to constantly having to sell and resell themselves just to keep the lights on, or to the labs with the equity packages, we won’t have a shot at winning. We will have a very dark, if mercifully short, future.
If we can keep our heads while all around us are losing theirs, we have a shot at changing the course of human society in the most wonderful way.
Thanks for reading. Tom