Offline Deep Q* Estimation with Diffusion Models

Summary

In offline reinforcement learning, estimating the optimal action-value function Q* can be formulated as solving the optimal Bellman equation based solely on offline observations. A fundamental challenge is that the reward function and transition kernel are unknown, so the optimal Bellman operator is not directly observable. To address this, we propose a novel framework that decouples operator estimation from value function learning. We first formulate conditional diffusion models to estimate the reward law and transition kernel, inducing a data-driven approximation of the optimal Bellman operator. We then plug these estimators into the Bellman equation and obtain a deep estimator of Q* by minimizing the empirical Bellman residual over a neural network function class. Theoretically, we establish sharp nonasymptotic convergence rates for learning the optimal Bellman operator via end-to-end analysis of conditional diffusion estimation in total variation distance, and then derive oracle value-stage rates for excess Bellman residual risk and L2 convergence rates under a concentrability condition. Our analysis does not rely on completeness assumptions. Extensive numerical experiments demonstrate effectiveness.