Learning Long-Term Reward Redistribution via Randomized Return Decomposition

11/26/2021
by   Zhizhou Ren, et al.
6

Many practical applications of reinforcement learning require agents to learn from sparse and delayed rewards. It challenges the ability of agents to attribute their actions to future outcomes. In this paper, we consider the problem formulation of episodic reinforcement learning with trajectory feedback. It refers to an extreme delay of reward signals, in which the agent can only obtain one reward signal at the end of each trajectory. A popular paradigm for this problem setting is learning with a designed auxiliary dense reward function, namely proxy reward, instead of sparse environmental signals. Based on this framework, this paper proposes a novel reward redistribution algorithm, randomized return decomposition (RRD), to learn a proxy reward function for episodic reinforcement learning. We establish a surrogate problem by Monte-Carlo sampling that scales up least-squares-based reward redistribution to long-horizon problems. We analyze our surrogate loss function by connection with existing methods in the literature, which illustrates the algorithmic properties of our approach. In experiments, we extensively evaluate our proposed method on a variety of benchmark tasks with episodic rewards and demonstrate substantial improvement over baseline algorithms.

READ FULL TEXT

page 1

page 2

page 3

page 4

research
02/25/2020

Human Apprenticeship Learning via Kernel-based Inverse Reinforcement Learning

This paper considers if a reward function learned via inverse reinforcem...
research
02/09/2020

Maximizing the Total Reward via Reward Tweaking

In reinforcement learning, the discount factor γ controls the agent's ef...
research
05/16/2022

Qualitative Differences Between Evolutionary Strategies and Reinforcement Learning Methods for Control of Autonomous Agents

In this paper we analyze the qualitative differences between evolutionar...
research
04/06/2023

Robust Decision-Focused Learning for Reward Transfer

Decision-focused (DF) model-based reinforcement learning has recently be...
research
05/17/2021

Learning to Win, Lose and Cooperate through Reward Signal Evolution

Solving a reinforcement learning problem typically involves correctly pr...
research
02/21/2021

Delayed Rewards Calibration via Reward Empirical Sufficiency

Appropriate credit assignment for delay rewards is a fundamental challen...
research
11/14/2022

Interactively Learning to Summarise Timelines by Reinforcement Learning

Timeline summarisation (TLS) aims to create a time-ordered summary list ...

Please sign up or login with your details

Forgot password? Click here to reset