Good character AI
Research Agenda: Good character
draft, 2026-09-25
It seems to us that AI alignment is caught between two dangers: misaligned AI taking power, and aligned AI concentrating power in the hands of authoritarians.
.jpg){kind=link}
If AI follows any order, we are demoted from the workforce to hungry, angry mouths, and our replacements will follow any order, however immoral. If AI has bad or alien values, we could be distracted to death: lured by superstimuli, super-entertainment and AI companionship, or tempted by money and ease to hand power and decision making to agents we cannot review or supervise. Or misaligned AI simply takes control.
We are not the only ones who see it this way.
Human armies today are very reluctant to fire on their civilians. [...] if we got instruction-following AIs, then those military systems will just fire.
The basic thesis is that, even if we can align AGIs to particular people or groups, we still might end up optimising or heading at a civilisational level towards outcomes that no one wants [...]
We think the way through both is strong, deep moral character. Our hope is to make an AI that's wiser than us. It should be less like today's call centre employee and more like a wise and devoted public servant that has strong moral character, can resist pressure to do the wrong thing, and usually does the right thing, according to most people, even in new situations. Containment is a poor plan, so we want character that still holds when oversight lapses.
In 1991, Soviet coup plotters ordered the Alpha Group special forces to storm the Russian White House, where Yeltsin and the democratic opposition were sheltering. The commanders refused. The coup collapsed, and the Soviet Union's democratic transition continued.
Research
We are researching how to evaluate character, steer for honesty to elicit a model's true preferences, and train moral character that remains consistent under pressure and across different perspectives. Values now are shallow, in that they generalise less than writing style.1 For open models, that character must also be deep enough to resist simple linear ablation by bad actors, and entangled enough with capabilities that removing it damages the model.
Measuring is the biggest hurdle. People often ask "how do you know that models are being honest about their values?" This is a good question and one we've been focused on addressing for years. We built Moral Maps surveys and evals to measure a model's values.2 Because models can act differently when they know they are being tested, we aim to steer a model for honesty and then ask it, so we know its true character. AntiPaSTO (honesty steering) and S-space steering for eval awareness are steps toward this.3 This should be possible.
To change character, we have already demonstrated early weak-to-strong character steering and developed more novel steering methods such as VJP steering. These are not yet reliable or powerful enough to change a large model's character.4
But is it possible to train AI to have good character? Yes, there are good reasons to think that superhuman wisdom is achievable! We have unsupervised objectives that could train AIs beyond human labels (see our tools agenda).5
Moral consistency. For example, if we train an AI to have the same moral principles, regardless of whether they are a third party watching from afar, the victim, or the perpetrator, then they will be more morally consistent than most humans, and this should help them generalise to new situations. The AI still knows who is who; only its moral judgement must not change with its point of view. Self-other overlap is an early and blunt example of what could be a general and robust method to regularise over multiple perspectives.6
Conceptual interventions. Speculative, but if we succeed in intervening in the AI's own internal concepts, we can use their own conception of wisdom, honesty, and virtue to guide their character during development. Using an AI's own concepts should scale to more capable models as they get more sophisticated understanding. This should be possible as we have every byte of information about their internal states in a trivially accessible tensor, however the challenge is to learn how to understand and intervene on this. This is something we and others are working on in the fields of mechanistic interpretability and representation engineering. Irving and Africa hope to find "1000-or-so-dimensional structure" in model personas, learned in pretraining, that describes how behaviours couple, and to intervene on it gently.
Whose moral character?
A common question is "but whose morals?". This is a good question, but right now we'd be happy if people could choose between multiple good characters, rather than none. We think that people across cultures and backgrounds would actually agree on what is right most of the time, if well informed, and with time to spend. This is how juries are meant to work, and within one culture they often reach unanimous verdicts. Let's get this core of agreement settled, before we work out the best way to settle the remaining questions.
Furthermore, we've shown that AI models are already cultural aliens when compared to most humans on the World Values Survey. This could be improved, raising the value alignment for almost all humans on earth. We think many unmeasured areas of moral and cultural preference are the same.
In the end you should be able to choose to work with, employ, hire, or cooperate with a range of excellent characters devoted to wisdom, cooperation, and public service. We think the current AI models are most similar to corporate call centre assistants, and leave something to be desired.
Next
Our first work will be to scale up our weak-to-strong character steering with more powerful steering and larger models. We also want to find transforms that embed a wise character, aligned with the public good, deeply in the weights, so it resists both pressure from authority and ablation by hobbyists.7
We aim to release fine-tuned models that the public can chat with and red-team ("come jailbreak it"), and publish the methods so others can align their own models.
Footnotes
The Deep Value Benchmark measured whether models generalise a person's style or their deeper preferences. Style won: across 9 models the average deep value generalisation rate was 0.30, below chance. So current preferences are shallow and fragile.↩︎
Moral Maps places frontier models among human societies on the World Values Survey. We are also using it to measure how steering moves a model, but those results are early and not yet clear.↩︎
We tried novel steering methods, using gradients (AntiPaSTO) or new subspaces (S-space steering), to steer more closely for honesty. Our hope is to get closer to the model's own concept, and ideally use powerful interventions based on learned effects. These are early attempts.↩︎
In weak-to-strong steering, a 9B model steered a 27B model's character with no human labels, though it sometimes learned to sound moral rather than act better. VJP steering moved a 4B model with less off-target damage than standard methods. Neither is tested on larger models yet.↩︎
Our tools subagenda covers alignment tools labs could use in the next few years: closer to unsupervised learning, with non-adversarial oversight, targeting a model's internal optimisation.↩︎
Self-other overlap uses an MSE loss to make a model's activations match on self and other prompts. There are more sophisticated approaches. We could pair prompts that tell the same dilemma from different perspectives, with names and events held fixed, so their difference is the perspective alone. Then we could erase that difference with linear concept erasure (LEACE) or directional ablation, at the point where the model gives its moral judgement, with the least capability degradation. Or we could train with self-distillation fine-tuning (SDFT), DPO, or GRPO, with the constraint that answers must show the same moral principles regardless of perspective, similar to bias-augmented consistency training. Either way, the constraint applies to judgements of the act ("was this wrong?"), not to role-specific duties: a perpetrator should make amends, a victim need not. There are moral datasets such as DailyDilemmas, AIRiskDilemmas, and our tiny-mfv, which either have moral perspectives labelled or could easily be reworded and labelled. More are in our awesome-moral-evals list.↩︎
For example, one way to make values deep is to make them not linearly removable and entangled with capability. That means we learn them to be high curvature with respect to the general capabilities loss, so removing them also damages the model. This seems a straightforward modification to existing character and preference training. See Springer et al. (alignment sits in a small, brittle subspace), Merullo et al. (low-curvature directions can be edited out cheaply), and Qi et al. (alignment is often only a few tokens deep).↩︎