@burny_tech - https://www.youtube.com/watch?v=c3b-JASoPi0

Andrej Karpathy: Current AI systems are imitation learners, but for superhuman AIs we will need better reinforcement learning like in AlphaGo. The model should selfplay, be in a the loop with itself and its own psychology, to achieve superhuman levels of intelligence. "We've got these next word prediction things. Do you think there's a path towards building a physicist or a Von Neumann type model that has a mental model of physics that's self-consistent and can generate new ideas for how do you actually do Fusion? How do you get faster than light if it's even possible? Is there any path towards that or is it a fundamentally different Vector in terms of these AI model developments?" "I think it's fundamentally different in one aspect. I guess what you're talking about maybe is just capability question because the current models are just not good enough and I think there are big rocks to be turned here and I think people still haven't really seen what's possible in the space at all and roughly speaking I think we've done step one of AlphaGo. We've done imitation learning part, there's step two of AlphaGo which is the RL and people haven't done that yet and I think it's going to fundamentally be the part that is actually going to make it work for something superhuman. I think there's big rocks in capability to still be turned over here and the details of that are kind of tricky but I think this is it, we just haven't done step two of AlphaGo. Long story short we've just done imitation. I don't think that people appreciate for example number one how terrible the data collection is for things like ChatGPT. Say you have a problem some prompt is some kind of mathematical problem a human comes in and gives the ideal solution right to that problem. The problem is that the human psychology is different from the model psychology. What's easy or hard for the human is different to what's easy or hard for the model. And so human kind of fills out some kind of a trace that comes to the solution but some parts of that are trivial to the model and some parts of that are massive leap that the model doesn't understand and so you're kind of just losing it and then everything else is polluted by that later. So fundamentally what you need is the model needs to practice itself how to solve these problems. It needs to figure out what works for it or does not work for it. Maybe it's not very good at four-digit addition so it's going to fall back and use a calculator, but it needs to learn that for itself based on its own capability and its own knowledge. So that's number one that's totally broken I think bur it's a good initializer though for something agent like. And then the other thing is we're doing reinforcement learning from human feedback but that's a super weak form of reinforcement learning, it doesn't even count as reinforcement learning. I think what is the equivalent in AlphaGo for RLHF is what I call it's a vibe check. Imagine if you wanted to train an AlphaGo RLHF. It would be giving two people two boards and said which one do you prefer and then you would take those labels and you would train model and then you would RL against that. What are the issues with that? Number one is that's it's just vibes of the board, that's what you're training against. Number two if it's a reward model that's a neural net then it's very easy to overfit to that reward model for the model you're optimizing over and it's going to find all these spurious ways of hacking that massive model, that's the problem. AlphaGo gets around these problems because they have a very clear objective function you can ARL against it. RLHF is nowhere near RL, it's silly. And the other thing is, imitation learning is super silly. RLHF is nice improvement, but it's still silly. I think people need to look for better ways of training these models, so that it's in the loop with itself and its own psychology, and I think we're there will probably be unlocks in that direction."

Follow me for more AI https://x.com/burny_tech/status/1732567307908477093