How to explain why RL is difficult to someone who knows nothing about it?
I’ve been working on an RL project at work. The person who assigned it to me is a computer scientist who is not an expert on RL, but understands it’s a difficult problem. (My boss is on equal footing with the person who assigned the project to me. My boss is not a computer scientist and doesn’t know anything about RL.) This guys boss is a business manager who doesn’t know anything about RL and knows very little about ML. The business manager wants a report on how the project is going from me and I’m getting the sense that he doesn’t really understand why this is taking so long.
For context, I’ve been working on this project for about 4 months for 15 hours per week. In that time, I’ve built an entire code base for the problem from scratch and programmed up several models. I have one that mostly works at the moment, but I need to make some changes to the reward functions to get it performing well consistently. I’m the only one working on this project, so I’ve done all of this myself. I also had only done vanilla RL prior to this, so I’ve had to learn a ton about deep RL to make this work. Luckily I know someone who’s an expert in deep RL (outside work) and has been able to give me pointers. I’m feeling like I’ve made a ton of progress and am nearing the home stretch in terms of having a completely polished model. However I’m getting the sense that this guy is not super thrilled with me. This guy doesn’t have any official authority over me, so this is mainly about trying to explain how much work RL is in addition to mg normal slides about the project and where I’m at.
Model based RL is looking a little more stable in the last year. Dreamerv3 and TD-MPC2 claim to be able to train on hundreds of tasks with no per-task hyperparameter tuning, and report smooth loss curves that scale predictably.
Have to wait and see if it pans out though.
I think this is overstating the contribution of these kinds of works. They still learn a Q-function via Mean-Squared Bellman Error, which means they’re subject to the same kind of instability in the value function as DDPG. They use a maximum entropy exploration method on the policy, which doesn’t come with exploration efficiency guarantees (at least not ones that are anywhere near optimal). The issue is that RL is extremely implementation-dependent. You can correctly implement an algorithm that got great results in a paper and have it still crash and burn.
At a basic level, the issue is that we just don’t have sound theory for extending RL to continuous non-linear MDPs. You can try stuff, but it’s all engineers’ algorithms, not mathematicians’ algorithms – you have no idea if or when it’ll all break down, and if it does all break down, they’re not gonna tell you that in the paper. Fundamentally we need theoretical work showing how to correctly solve these kinds of problems, and that’s something a problem that these experimentally-focused papers are not attempting to address.
Progress requires directly addressing these issues. In my opinion, that’s most likely to come though theoretically-driven work. For the value-divergence problem, that means Gradient Temporal Difference algorithms and their practical extensions (such as TD with Regularized Corrections). For exploration, that means using insights from online learning, like best-of-both-worlds algorithms that give a clear “exploration objective” that policies can optimize.