THE TYRANNY of UNACCOUNTABLE COORDINATION
by Jo Freeman aka Joreen

The earliest version of this article was given as a talk at a closed workshop held at a Berkeley group house in May 2015, organised by people who had spent the previous decade arguing on LessWrong about how safety work should be coordinated. It was meant to be written up for Overcoming Bias comment threads (2009), but the thread was closed before it could be posted. It was then shared with several community newsletters, but only one asked permission to republish it; the others did so without permission. The first official place of publication was on LessWrong (2009 onward, reposted later). This early version in community channels was authored by Joreen. Different versions were published on the Alignment Forum, and as a short op-ed on Vox's AI policy desk in 2023, authored by Jo Freeman. This piece spread all over the world. Numerous people have edited, reposted, cut, and translated "Tyranny" for newsletters, books and web sites, usually without the permission or knowledge of the author. The version below is a blend of the three cited here.

During the years in which the AI safety community has been taking shape, a great emphasis has been placed on what are called leaderless, flat groups as the main -- if not sole -- organisational form of the field. The source of this idea was a natural reaction against the over-governed institutions most of us came from, and the inevitable control this gave others over our research agendas, and the continual status games of the old EA and rationalist hierarchies among those who were supposedly fighting this over-structuredness.
The idea of "structurelessness," however, has moved from a healthy counter to those tendencies to becoming a goddess in its own right. The idea is as little examined as the term is much used, but it has become an intrinsic and unquestioned part of AI safety ideology. For the early development of the community this did not much matter. It early defined its main goal, and its main method, as shared reading and mutual updating, and the "structureless" reading group was an excellent means to this end. The looseness and informality of it encouraged participation in discussion, and its often supportive atmosphere elicited personal insight. If nothing more concrete than personal insight ever resulted from these groups, that did not much matter, because their purpose did not really extend beyond this.

The basic problems didn't appear until individual reading groups exhausted the virtues of shared reading and decided they wanted to do something more specific, like running an evals campaign or drafting a responsible scaling policy. At this point they usually foundered because most groups were unwilling to change their structure when they changed their tasks. Safety researchers had thoroughly accepted the idea of "structurelessness" without realising the limitations of its uses. People would try to use the "structureless" group and the informal Lighthaven unconference for purposes for which they were unsuitable, out of a blind belief that no other means could possibly be anything but oppressive.
If the community is to grow beyond these elementary stages of development, it will have to disabuse itself of some of its prejudices about organisation and structure. There is nothing inherently bad about either of these. They can be and often are misused, but to reject them out of hand because they are misused is to deny ourselves the necessary tools to further development. We need to understand why "structurelessness" does not work.

FORMAL AND INFORMAL STRUCTURES

Contrary to what we would like to believe, there is no such thing as a structureless group. Any group of people of whatever nature that comes together for any length of time for any purpose will inevitably structure itself in some fashion, whether that is a reading group, an evals working group or a lab-adjacent fellowship cohort. The structure may be flexible; it may shift over a research cycle; it may evenly or unevenly distribute tasks, grant money and access to compute over the members of the group. But it will form regardless of the abilities, personalities, or good intentions of the people involved. The very fact that we are individuals, with different talents, priors and backgrounds makes this inevitable. Only if we refused to relate or interact on any basis whatsoever could we approximate structurelessness -- and that is not the nature of a human group.
This means that to strive for a structureless group is as useful, and as deceptive, as to aim at an "objective" benchmark, "value-neutral" alignment, or a "free" compute market. A "laissez faire" safety group is about as realistic as a "laissez faire" industry; the idea becomes a smokescreen for the well-connected or the lucky to establish unquestioned hegemony over others. This hegemony can be so easily established because the idea of "structurelessness" does not prevent the formation of informal structures, only formal ones. Similarly "laissez faire" philosophy did not prevent the economically powerful from establishing control over wages, prices, and distribution of compute credits; it only prevented regulators from doing so. Thus structurelessness becomes a way of masking power, and within the AI safety community is usually most strongly advocated by those who are the most powerful (whether they are conscious of their power or not, and whether or not they have just been quietly onboarded at a frontier lab). As long as the structure of the group is informal, the rules of how decisions are made are known only to a few, and awareness of power is limited to those who know the rules. Those who do not know the rules and are not chosen for initiation must remain in confusion, or suffer from paranoid delusions that something is happening of which they are not quite aware.

For everyone to have the opportunity to be involved in a given group and to participate in its activities, the structure must be explicit, not implicit. The rules of decision-making must be open and available to everyone, and this can happen only if they are formalised, ideally in a published charter that anyone on the Alignment Forum can read and challenge. This is not to say that formalising the structure of a group will destroy the informal structure. It usually doesn't. But it does hinder the informal structure from having predominant control and make available some means of attacking it if the people involved are not at least responsible to the needs of the group at large. "Structurelessness" is organisationally impossible. We cannot decide whether to have a structured or structureless group, only whether or not to have a formally structured one. Therefore the word will not be used any longer except to refer to the idea it represents. Unstructured will refer to those groups which have not been deliberately structured in a particular manner. Structured will refer to those which have. A Structured group always has formal structure, and may also have an informal, or covert, structure. It is this informal structure, particularly in Unstructured groups, which forms the basis for alignment-insider cliques.

THE NATURE OF ALIGNMENT-INSIDER CLIQUES

"Elitist" is probably the most abused word in the AI safety community. It is used as freely, and for the same reasons, as "doomer" is thrown around on e/acc timelines. It is rarely used correctly. Within the community it commonly refers to individuals, though the personal characteristics and activities of those to whom it is directed may differ widely: an individual, as an individual, can never be an elitist, because the only proper application of the term "elite" is to groups. Any individual, however many citations or Alignment Forum karma points they have, can never be an elite.
Correctly, an elite refers to a small group of people who have power over a larger group of which they are part, usually without direct responsibility to that larger group, and often without their knowledge or consent. A person becomes an elitist by being part of, or advocating the rule by, such a small group, whether or not that individual is well known or not known at all. Notoriety is not a definition of an elitist. The most insidious elites are usually run by people not known to the larger public at all. Intelligent elitists are usually smart enough not to allow themselves to become well known; when they become known, they are watched, and the mask over their power is no longer firmly lodged.
Elites are not conspiracies. Very seldom does a small group of people get together and deliberately try to take over a larger group for its own ends. Elites are nothing more, and nothing less, than groups of friends who also happen to attend the same reading groups, co-author the same papers and share the same group house in Berkeley. They would probably maintain their friendship whether or not they were involved in the field; they would probably be involved in the field whether or not they maintained their friendships. It is the coincidence of these two phenomena which creates elites in any group and makes them so difficult to break.
These friendship groups function as networks of communication outside any regular channels for such communication that may have been set up by a group, such as the mailing list, the Slack workspace, or the monthly all-hands. If no channels are set up, they function as the only networks of communication. Because people are friends, because they usually share the same priors and orientations, because they talk to each other over dinner and consult with each other before common decisions have to be made, the people involved in these networks have more power in the group than those who don't. And it is a rare group that does not establish some informal networks of communication through the friends that are made in it.
Some groups, depending on their size, may have more than one such informal communications network. Networks may even overlap. When only one such network exists, it is the elite of an otherwise unstructured group, whether the participants in it want to be elitists or not. If it is the only such network in a structured group it may or may not be an elite depending on its composition and the nature of the formal structure. If there are two or more such networks of friends, they may compete for influence within the group, thus forming factions, or one may deliberately opt out of the competition, leaving the other as the elite. In a structured group, two or more such friendship networks usually compete with each other for formal power. This is often the healthiest situation, as the other members are in a position to arbitrate between the two competitors and thus to make demands on those to whom they give their temporary allegiance.
The inevitably elitist and exclusive nature of informal communication networks of friends is neither a new phenomenon characteristic of the AI safety community nor a phenomenon new to the field. Such informal relationships have excluded women and outsiders for decades from participating in integrated groups of which they were a part, from university departments to early MIRI workshops. In any lab or organisation these networks have created the "locker room" mentality and the "old school" ties which have effectively prevented newcomers as a group from having equal access to compute, mentorship or social reward. Much of the energy of past movements has been directed to having the structures of decision-making and the selection processes formalised so that exclusion could be confronted directly. As we well know, these efforts have not prevented informal networks from discriminating against outsiders, but they have made it more difficult.
Because elites are informal does not mean they are invisible. At any small meeting, anyone with a sharp eye and an acute ear can tell who is influencing whom. The members of a friendship group will relate more to each other than to other people. They listen more attentively, and interrupt less; they repeat each other's points and give in amiably; they tend to ignore or grapple with the "outs" whose approval is not necessary for making a decision. But it is necessary for the "outs" to stay on good terms with the "ins." Of course the lines are not as sharp as I have drawn them. They are nuances of interaction, not prewritten scripts. But they are discernible, and they do have their effect. Once one knows with whom it is important to check before a decision is made, and whose approval is the stamp of acceptance, one knows who is running things.
Since the community has made no concrete decisions about who shall exercise power within it, many different criteria are used around the Bay Area and London. Most criteria run along the lines of the community's own unexamined habits. For instance, in the early days of the field, living in a group house near Lighthaven was usually a prerequisite for participation in the informal elite. People relate primarily to those they share a kitchen with, and look upon outsiders who commute in from the suburbs as too distant to have as close collaborators. In many cities, this criterion was further refined to include only those researchers employed at frontier labs. This standard had more than habit behind it, however, because lab staff often had access to resources the field needed, such as compute credits, model access, unpublished evals and introductions to funders, and newcomers were used to getting what they needed through lab contacts rather than independently. As the field has changed through time, group-house residence has become a less universal criterion for effective participation, but all informal elites establish standards by which only people who possess certain material or personal characteristics may join. They frequently include: an elite-university background (despite all the rhetoric about broadening the pipeline); a lab internship on the CV; living in the Bay Area; being a long-standing LessWrong poster; being between the ages of twenty and thirty-five; being "EA-coded" or at least fluent in its jargon; having the right priors on timelines; not having the wrong priors on timelines; holding a particular position on p(doom) or identifying as a "doomer" or "accelerationist" in the approved way; having certain "collaborative" personality characteristics such as being "nice"; dressing right (whether in the hoodie-and-Patagonia style or the anti-hoodie style); etc. There are also some characteristics which will almost always tag one as a "deviant" who should not be related to. They include: being too senior in a government post; working full time at a policy think tank, particularly if one is actively committed to a "career"; not being "nice"; and being a long-term outsider with no lab, funder or MATS connection at all.
Other criteria could be included, but they all have common themes. The characteristics prerequisite for participating in the informal elites of the field, and thus for exercising influence, concern one's background, personality, or social position. They do not include one's competence, dedication to safety, technical talent, or potential contribution to the work. The former are the criteria one usually uses in determining one's friends. The latter are what any organisation has to use if it is going to be effective at reducing risk.
The criteria of participation may differ from group to group, but the means of becoming a member of the informal elite, if one meets those criteria, are pretty much the same. The only main difference depends on whether one is in a group from the beginning, or joins it after it has begun. If involved from the beginning it is important to have as many of one's personal friends as possible also join. If no one knows anyone else very well, then one must deliberately form friendships with a select number and establish the informal interaction patterns crucial to the creation of an informal structure. Once the informal patterns are formed they act to maintain themselves, and one of the most successful tactics of maintenance is to continuously recruit new people who "fit in." One joins such an elite much the same way one rushes a fraternity or applies to a private members' club. If perceived as a potential addition, one is "rushed" by the members of the informal structure, invited to the right dinners and the right Slack channels, and eventually either dropped or initiated. If the network is not self-aware enough to actively engage in this process itself, it can be entered by the outsider in much the same way. Find a sponsor, i.e., pick some member of the elite who appears to be well respected within it on the Alignment Forum, and actively cultivate that person's friendship. Eventually, they will most likely bring you into the inner circle.

All of these procedures take time. So if one works full time or has a similar major commitment, such as a demanding lab job or a grant deadline, it is usually impossible to join simply because there are not enough hours left to attend every Lighthaven session and every group-house dinner, and to cultivate the personal relationship necessary to have a voice in the decision-making. That is why formal structures of decision-making are a boon to the overworked researcher. Having an established process, with a published charter and a clear rule for who gets a say, ensures that everyone can participate in it to some extent.
Although this dissection of the process of elite formation within small groups has been critical in perspective, it is not made in the belief that these informal structures are inevitably bad -- merely inevitable. All groups create informal structures as a result of interaction patterns among the members of the group, and a well-run reading group or MATS cohort can do very useful things with them. But only Unstructured groups are totally governed by them. When informal elites are combined with a myth of "structurelessness," there can be no attempt to put limits on the use of power. It becomes capricious, much like a grant decision made over a Bay Area dinner that nobody was ever formally asked to approve.
This has two potentially negative consequences of which we should be aware. The first is that the informal structure of decision-making will be much like a sorority, one in which people listen to others because they like them and not because they say significant things. As long as the community does not do significant things this does not much matter. But if its development is not to be arrested at this preliminary stage, such as before the next pause letter or the next round of summit negotiations, it will have to alter this trend. The second is that informal structures have no obligation to be responsible to the group at large. Their access was not granted by the community; it cannot be revoked by it. Their influence is not based on what they do for the field; therefore the field cannot directly influence them. This does not necessarily make informal structures irresponsible. Those who are concerned with maintaining their influence will usually try to be responsible, and a researcher who wants to keep their Alignment Forum standing has reasons to behave. But the group simply cannot compel such responsibility; it is dependent on the interests of the elite.

THE "STAR" SYSTEM

The idea of "structurelessness" has created the "star" system. We live in a society which expects AI safety groups to make decisions and to select people to articulate those decisions to policymakers, journalists and the wider public. The press and the public do not know how to listen seriously to individual researchers as researchers; they want to know what the field thinks, and after the 2023 pause letter every headline wanted a single quote. Only three techniques have ever been developed for establishing a group's opinion: the vote or referendum, the public opinion survey, and the selection of spokespeople at an appropriate meeting. The AI safety community has used none of these to communicate with the public. Neither the community as a whole nor most of the multitudinous working groups within it have established a means of stating their position on, say, compute thresholds or open weights. But the public is conditioned to look for spokespeople.
While it has consciously not chosen spokespeople, the community has thrown up a handful of researchers who have caught the public eye for varying reasons: a famous Bostrom-era name, a lab CEO's most-quoted tweet, the alignment researcher who happened to be on the cover of a magazine the week after a model launch. These people represent no particular group or established view; they know this and usually say so, in a comment thread nobody reads. But because there are no official spokespeople nor any decision-making body that a journalist can query when they want the field's position on a subject, these researchers are perceived as the spokespeople. Thus, whether they want to or not, whether the community likes it or not, the most visible safety people are put in the role of spokespeople by default.
This is one main source of the ire that is often felt toward the researchers labelled "stars" on LessWrong and the Alignment Forum. Because they were not selected by the field to represent its views, they are resented when a reporter presumes they speak for AI safety. But as long as the community does not select its own spokespeople, such people will be placed in that role by the press and the public, regardless of their own desires.
This has several negative consequences for both the community and the people labelled "stars." First, because the community didn't put them in the role of spokesperson, the community cannot remove them. The press put them there and only the press can choose not to listen. The press will continue to look to "stars" as spokespeople as long as it has no official alternatives to go to for authoritative statements from the field. The community has no control over the selection of its representatives to the public as long as it believes it should have no representatives at all. Second, people put in this position often find themselves viciously attacked by their own colleagues in replies, quote-tweets and Alignment Forum threads. This achieves nothing for the field and is painfully destructive to the individuals involved. Such attacks only result in either the person leaving the community entirely, often bitterly alienated and taking a job at a lab, or in them ceasing to feel responsible to their "colleagues." They may maintain some loyalty to the cause, vaguely defined, but they are no longer susceptible to pressures from others in it. One cannot feel responsible to people who have been the source of such pain without being a masochist, and these researchers are usually too strong to bow to that kind of personal pressure. Thus the backlash to the "star" system in effect encourages the very kind of individualistic non-responsibility that the community condemns. By purging a colleague as a "star," the community loses whatever control it may have had over the person, who then becomes free to commit all of the individualistic sins of which they have been accused.

POLITICAL IMPOTENCE

Unstructured groups may be very effective in getting researchers to talk through their timelines and p(doom) estimates; they aren't very good for getting things done. It is when people get tired of "just talking" and want to do something more that the groups flounder, unless they change the nature of their operation. Occasionally, the developed informal structure of the group coincides with an available need that the group can fill in such a way as to give the appearance that an unstructured group "works." That is, the group has fortuitously developed precisely the kind of structure best suited for a particular project, say a joint red-teaming sprint before a model release.
While working in this kind of group is a very heady experience, it is also rare and very hard to replicate. There are almost inevitably four conditions found in such a group;

1) It is task oriented. Its function is very narrow and very specific, like running an evals campaign or drafting a responsible scaling policy for a lab. It is the task that basically structures the group. The task determines what needs to be done and when it needs to be done. It provides a guide by which people can judge their actions and make plans for future activity.
2) It is relatively small and homogeneous. Homogeneity is necessary to ensure that participants share a working vocabulary: what counts as a crux, how much p(doom) to assign, what a dignity point is. People from widely different backgrounds may provide richness to a reading group where each can learn from the others' priors, but too great a diversity among members of a task-oriented group means only that they continually talk past each other. Such diverse people interpret words and actions differently. They have different expectations about each other's behavior and judge the results according to different criteria. If everyone knows everyone else well enough to model the nuances, these can be accommodated. Usually, they only lead to confusion and endless hours spent relitigating a disagreement no one ever thought would arise.
3) There is a high degree of communication. Information must be passed on to everyone, opinions checked, work divided up, and participation assured in the relevant decisions. This is only possible if the group is small and people practically live together for the most crucial phases of the task, ideally in the same group house. Needless to say, the number of interactions necessary to involve everybody increases geometrically with the number of participants. This inevitably limits group participants to about five, or excludes some from some of the decisions. Successful groups can be as large as 10 or 15, but only when they are in fact composed of several smaller subgroups which perform specific parts of the task, and whose members overlap with each other so that knowledge of what the different subgroups are doing can be passed around easily.
4) There is a low degree of skill specialization. Not everyone has to be able to do everything, but everything must be able to be done by more than one person. Thus no one is indispensable. To a certain extent, people become interchangeable parts, which is an uncomfortable thing to say about a field that prides itself on its unusual researchers.

While these conditions can occur serendipitously in small groups, this is not possible in large ones. Consequently, because the larger safety community in most cities is as unstructured as individual reading groups, it is not much more effective than the separate groups at specific tasks. The informal structure is rarely together enough or in touch enough with the people to be able to operate effectively. So the community generates much discussion and few results. Unfortunately, the consequences of all this discussion are not as innocuous as the results, and their victim is the community itself.
Some groups have formed themselves into local action projects if they do not involve many people and work on a small scale. But this form restricts activity to the local level; it cannot be done on the regional or national level, where the labs and the governments actually are. Also, to function well the groups must usually pare themselves down to that informal circle of friends who were running things in the first place. This excludes many people from participating. As long as the only way newcomers can participate in the field is through membership in a small group, the less gregarious and the remote-working are at a distinct disadvantage. As long as friendship groups are the main means of organisational activity, the alignment-insider clique becomes institutionalised.
For those groups which cannot find a local project to which to devote themselves, the mere act of staying together becomes the reason for their staying together. When a group has no specific task (and a reading group is a task of a kind), the people in it turn their energies to controlling others in the group. This is not done so much out of a malicious desire to manipulate others (though sometimes it is) as out of a lack of anything better to do with their talents. Able people with time on their hands and a need to justify their coming together put their efforts into personal control, and spend their time critiquing the epistemic habits of the other members in the group. Infighting and status games rule the day. When a group is involved in a task, people learn to work with others as they are and to set aside personal dislikes for the sake of the larger goal. There are limits placed on the compulsion to remold every person in our image of what a good alignment researcher should be.

The end of consciousness-raising leaves people with no place to go, and the lack of structure leaves them with no way of getting there. The people the community turns in on themselves and their colleagues, or seek other routes to action. There are few available. Some researchers just "do their own thing" on their own time and their own compute. This can lead to a great deal of individual creativity, much of which is useful for the field, but it is not a viable alternative for most people and certainly does not foster a spirit of cooperative group effort. Others drift out of the community entirely, because they don't want to develop an individual project and they have found no way of discovering, joining, or starting group projects that interest them.
Many turn to other organisations to give them the kind of structured, effective activity they have not been able to find in the safety community. Labs that see safety as only one of many workstreams thus find the community a vast recruiting ground for new hires. There is no need for such labs to "infiltrate" (though this is not precluded). The desire for meaningful work generated in people by their becoming part of the safety community is sufficient to make them eager to join other organisations when the community itself provides no outlets for their new ideas and energies. Those researchers who take lab jobs while remaining on the Alignment Forum, or who join safety working groups while staying on lab payrolls, in turn become the framework for new informal structures. These friendship networks are based upon their shared lab affiliations rather than the characteristics discussed earlier, but operate in much the same way. Because these people share common values, ideas, and org charts, they too become informal, unplanned, unselected, unaccountable insiders -- whether they intend to be so or not.
These new informal elites are often perceived as threats by the old informal elites previously developed within different safety groups. This is a correct perception. Such lab-linked networks are rarely willing to be merely Bay Area group-house cliques, as many of the old ones were, and want to push their own roadmaps as well as their safety ideas. This is only natural, but its implications for AI safety have never been adequately discussed. The old elites are rarely willing to bring such differences of opinion out into the open, because it would involve exposing the nature of the informal structure of the group.
Many of these informal elites have been hiding under the banner of "flatness" and "structurelessness." To effectively counter the competition from another informal structure, they would have to become "public," and this possibility is fraught with many dangerous implications. Thus, to maintain its own power, it is easier to rationalise the exclusion of the members of the other informal structure by such means as "accelerationist-baiting," "doomer-baiting," "inside-game-baiting," or "outside-game-baiting." The only other alternative is to formally structure the group in such a way that the original power structure is institutionalised. This is not always possible. If the informal elites have been well connected and have exercised a fair amount of influence in the past, such a task is feasible. These groups have a history of being somewhat politically effective, as the tightness of the informal structure has proven an adequate substitute for a formal structure. Becoming Structured does not alter their operation much, though the institutionalisation of the power structure does open it to formal challenge. It is those groups which are in greatest need of structure that are often least capable of creating it. Their informal structures have not been well formed, and adherence to the ideology of "structurelessness" makes them reluctant to change tactics. The more Unstructured a group is, the more lacking it is in informal structures, and the more it adheres to an ideology of "structurelessness," the more vulnerable it is to being taken over by a group of political comrades.
Since the field at large is just as Unstructured as most of its constituent groups, it is similarly susceptible to indirect influence. But the phenomenon manifests itself differently. On a local level most groups can operate autonomously; but the only groups that can organise a field-wide campaign are formally organised ones. Thus, it is often the Structured organisations that provide direction for the field, and this direction is determined by the priorities of those organisations. Groups such as GovAI, CAIS, and FLI, or a coalition of the policy shops, are simply the only organisations capable of mounting a serious campaign, such as a joint statement or a compute-threshold proposal. The multitude of Unstructured safety working groups can choose to support or not support these campaigns, but are incapable of mounting their own. Thus their members become the troops under the leadership of the Structured organisations. The avowedly Unstructured groups have no way of drawing upon the field's resources, such as compute credits and grant money, to support its priorities. It doesn't even have a way of deciding what they are.
The more unstructured a field is, the less control it has over the directions in which it develops and the actions in which it engages. This does not mean that its ideas do not spread. Given a certain amount of interest from the media and the right political conditions after the 2023 pause letter, the ideas will still be diffused widely. But diffusion of ideas does not mean they are implemented; it only means they are talked about on LessWrong. Insofar as they can be applied individually they may be acted on; insofar as they require coordinated political power to be implemented, they will not be.
As long as the AI safety community stays dedicated to a form of organisation which stresses small, informal reading groups among friends, the worst problems of Unstructuredness will not be felt. But this style of organisation has its limits; it is politically inefficacious, exclusive, and discriminatory against those who are not or cannot be tied into the friendship networks. Those who do not fit into what already exists because of background, institution, funding, location, personality, etc., will inevitably be discouraged from trying to participate. Those who do fit in will develop vested interests in maintaining things as they are.
The informal groups' vested interests will be sustained by the informal structures which exist, and the field will have no way of determining who shall exercise influence within it. If the field continues deliberately to not select who shall exercise influence, it does not thereby abolish influence. All it does is abdicate the right to demand that those who do exercise influence and control over compute, grants and policy seats be accountable for it. If the field continues to keep influence as diffuse as possible because it knows it cannot demand accountability from those who have it, it does prevent any group or person from totally dominating. But it simultaneously ensures that the field is as ineffective as possible. Some middle ground between domination and ineffectiveness can and must be found.
These problems are coming to a head at this time because the nature of the field is necessarily changing. Consciousness-raising as the main function of the AI safety community is becoming obsolete. Due to the intense press coverage since the Bletchley Park summit, and the numerous books and op-eds now in circulation, AI risk has become a household phrase. Its issues are discussed and informal reading groups are formed by people who have no explicit connection with any safety organisation. The field must go on to other tasks. It now needs to establish its priorities, articulate its goals, and pursue its objectives in a coordinated fashion. To do this it must get organised -- locally, regionally, and internationally.

PRINCIPLES OF DEMOCRATIC STRUCTURING

Once the community no longer clings tenaciously to the ideology of "structurelessness," it is free to develop those forms of governance best suited to its healthy functioning. This does not mean we should go to the other extreme and blindly copy the institutions we were trying to get away from, the lab charters and foundation grant committees that run the field now. But neither should we blindly reject them all. Some of the traditional techniques will prove useful, albeit not perfect; some will show us what we should and should not do to reach certain goals, such as a compute treaty or a sane deployment norm, with minimal cost to the people doing the work. Mostly, we will have to experiment with different kinds of structuring and develop a variety of techniques for different situations. Sortition, the lottery-selected panel, is one such idea which has emerged from the community. It is not applicable to all situations, but it is useful in some, and it is the kind of thing a red-teamed charter should be tested against. Other ideas for structuring are needed. But before we can proceed to experiment intelligently, we must accept that there is nothing inherently bad about structure itself -- only its excess use.