Accelerating Multi-step Sparse Reward Reinforcement Learning

After the great successes of deep reinforcement learning (DRL) in recent years, developing methods to speed up DRL algorithms for more complex tasks closer to those in the real world has become increasingly important. In particular, there is a lack of research on long-horizon tasks that contain multiple subtasks or intermediate steps and can only provide sparse rewards at task completion point. This paper suggests to 1) use human priors to decompose a task and provide abstract demonstrations – the correct sequences of steps to guide exploration and learning, and 2) adjust the exploration parameters adaptively according to the online performances of the policy. The proposed ideas are implemented on three popular DRL algorithms, and experimental results on gridworld and manipulation tasks prove the concept and effectiveness of the proposed techniques.