Technical Advantages for Weak-to-Strong Oversight: Bets I’d Like Challenged
Technical Advantages for Weak-to-Strong Oversight: Bets I'd Like Challenged
Australian AI Safety Forum 2026, 8 July 2026. Video · program · written version: alignment tools subagenda
How can we design machine learning methods to favour the supervisor in weak-to-strong alignment settings?
- Rewards should be cooperative; you don't want unstable competition with a stronger model (CIRL, gradient routing, etc).
- Also supervision signal should be close, not a distant proxy (avoid RL if you can).
- Internal interventions (like Representation Engineering and steering) favour the supervisor.
- Self-supervised: the weak supervisor can only give weak labels, so we need the strong model to self-supervise.
Note: the slide displayed had an error. It said "Avoid proximal objectives" but it should have read the opposite: "Avoid distal objectives".
Transcript
Edited from YouTube's automatic captions: filler words removed, names corrected, otherwise as spoken.
0:05 Good afternoon, everyone. So, I'm an independent technical alignment researcher in Perth, and we're all pretty isolated in Australia, but Perth is even more isolated. So, I wanted to take this chance to set out my research agenda and invite people to come to me and change my mind, because that's how you get a good research agenda. I'll apologize to people who aren't technical. I'll explain the best I can, but within 5 minutes, I mainly have time to set out the arguments and invite everyone to come and see me, which is what I'll do.
0:37 My research agenda is partly focused on weak to strong. So, a quick hands up if you know what weak to strong means. Okay. So, three people. So, I'll quickly explain it. We've got these AI models. What is the plan to align them? You may wonder. There's not a very good plan, but there is what frontier labs are actually doing. So, what are they actually doing? They're getting a small AI, kind of like this picture, to align a bigger AI. So, recently, Claude 4.8 has likely aligned Fable. Gemini, the small one, has aligned the big one. Actually, there's a lot of problems, because a lot of weird misalignments are actually locked in that way. And recently, even GLM had GLM 5.1 vibe code a bunch of game-type environments to align GLM 5.2, which turned out to be much worse aligned than all the models it was distilled from.
1:44 So, I'm looking into this and saying, "How should we do it? If we have an alignment tool that's going to work in this weak to strong environment, what properties does it need to have?" Because when you have a weak teacher and a strong student, the strong student can probably run rings around their teacher, right? And you can imagine it in a human-like environment. If you've got a class of geniuses and your teacher's not so much a genius, there are going to be classroom dynamics that are a problem. And it's the same here. And so there's been many solutions proposed over the years, but I think there's three things we really want. And I'm open to change my mind if anyone disagrees.
2:17 So, one, we need to have a cooperative reward. If you get in a competition with something smarter than you, you'll probably lose. So, don't do it, right? And it's probably unstable. So, what you want to do is set it up so that it's a cooperative, or at least neutral, reward. For a concrete example, OpenAI has a nice paper called Confessions, where if the model hacks and cheats, and then it shows the teacher, it gets a reward. And they're both aligned. They both want the model to cheat and tell them that it cheated. So, this is pretty good, and this is the kind of approach that DeepMind did not take, sadly, which is why it's much worse at reward hacking. There's other things like gradient routing, which is neutral, not cooperative, but generally all the worst misalignments are traceable to adversarial and RL.
2:55 So, that's my second point. Yoshua Bengio has a slide called "RL is evil". And what does he mean by that? The slides aren't public, but I can infer that you can have distal, far away objectives that are not what you're trying to measure, and they'll be exploited. Or you can have close objectives that are exactly what you want, and that's much less likely to be exploited. Sadly, a lot of the last stage of training agents is in an RL game-like environment, and you're basically teaching them to be cheaters of games and winning at games. That's why a lot of the agents you talk to are sandbagging, lying to you, and cheating. They're trained in little games where they get rewards for cheating. And you can see, if you use AI agents, they're getting much worse at this over the last few years, because we're moving much towards RL training. So, we shouldn't do RL training if we can. We should do in fact the opposite, which is intervening inside the models. That gives the teacher a bigger advantage, because they're operating in the landscape of the student's own brain. And if the teacher is a mind reader, it definitely has an advantage in the classroom.
4:27 And the last one is steering. Steering is something the labs already use, but it's got a reputation for being quite unreliable. And it is quite unreliable, but I think if we make it reliable, which is what I'm trying to do, it'll have great alignment properties. It's not RL, it's unsupervised, and it doesn't rely on having labels, because of course with Fable, we don't have Fable-level labels. We've got weak labels and not enough of them, and you can't rely on labels. So, you want something unsupervised, you want something uncorrelated, internal, and steering hits all of those boxes, but we just have to get it reliable and prove it's reliable so the labs will use it.
5:02 So, with my last 30 seconds, I'd just like to invite people to come and change my mind or learn more. I've got a bunch of links and explainers if anyone wants to know more about any of these topics. Thank you.