TY - GEN
T1 - Towards General Function Approximation in Nonstationary Reinforcement Learning
AU - Feng, Songtao
AU - Yin, Ming
AU - Huang, Ruiquan
AU - Wang, Yu Xiang
AU - Yang, Jing
AU - Liang, Yingbin
N1 - Publisher Copyright:
© 2024 IEEE.
PY - 2024
Y1 - 2024
N2 - Function approximation has experienced significant success in the field of reinforcement learning (RL). Despite a handful of progress on developing theory for Nonstationary RL with function approximation under structural assumptions, existing work for nonstationary RL with general function approximation is still limited. In this work, we propose a UCB-type of algorithm LSVI-Nonstationary following the popular least-square-value-iteration (LSVI) framework. LSVI-Nonstationary features the restart mechanism and a new design of bonus term to handle nonstationarity, and performs no worse than the existing confidence-set based algorithm SW-OPEA in [1], which has been shown to outperform the existing algorithms for nonstationary linear and tabular MDPs in the small variation budget setting.
AB - Function approximation has experienced significant success in the field of reinforcement learning (RL). Despite a handful of progress on developing theory for Nonstationary RL with function approximation under structural assumptions, existing work for nonstationary RL with general function approximation is still limited. In this work, we propose a UCB-type of algorithm LSVI-Nonstationary following the popular least-square-value-iteration (LSVI) framework. LSVI-Nonstationary features the restart mechanism and a new design of bonus term to handle nonstationarity, and performs no worse than the existing confidence-set based algorithm SW-OPEA in [1], which has been shown to outperform the existing algorithms for nonstationary linear and tabular MDPs in the small variation budget setting.
UR - http://www.scopus.com/inward/record.url?scp=85202813405&partnerID=8YFLogxK
UR - http://www.scopus.com/inward/citedby.url?scp=85202813405&partnerID=8YFLogxK
U2 - 10.1109/ISIT57864.2024.10619458
DO - 10.1109/ISIT57864.2024.10619458
M3 - Conference contribution
AN - SCOPUS:85202813405
T3 - IEEE International Symposium on Information Theory - Proceedings
SP - 1
EP - 6
BT - 2024 IEEE International Symposium on Information Theory, ISIT 2024 - Proceedings
PB - Institute of Electrical and Electronics Engineers Inc.
T2 - 2024 IEEE International Symposium on Information Theory, ISIT 2024
Y2 - 7 July 2024 through 12 July 2024
ER -