基于一维卷积循环神经网络的深度强化学习算法

Reinforcement Learning Algorithm Based on One-dimensional Convolutional Recurrent Network

摘要:

针对现有深度强化学习算法在状态空间维度大的环境中难以收敛的问题,提出了在时间维度上提取特征的基于一维卷积循环网络的强化学习算法；首先在深度Q网络(deep Q network, DQN)的基础上构建一个深度强化学习系统；然后在深度循环Q网络(deep recurrent Q network, DRQN)的神经网络结构基础上加入了一层一维卷积层,用于在长短时记忆(long short-term memory, LSTM)层之前提取时间维度上的特征；最后在与时序相关的环境下对该新型强化学习算法进行训练和测试；实验结果表明这一改动可以提高智能体的决策水平,并使得深度强化学习算法在非图像输入的时序相关环境中有更好的表现。

Abstract:

Existing deep reinforcement learning algorithms have difficulty converging in environments with large state space dimensions. So a reinforcement learning algorithm based on one-dimensional convolutional recurrent networks that extracts features in the time dimension is proposed. Firstly, a deep reinforcement learning system based on DQN is built. Then a one-dimensional convolutional layer is added into the neural network architecture of DRQN for extracting the features in the time dimension before the LSTM layer. Finally, the new reinforcement learning algorithm is trained and tested in a timing-related environment. The experimental results show that this change can improve the decision-making level of the agent, making deep reinforcement learning algorithms have better performance in non-image input and timing-related environment