Model-based adaptation for sample efficient transfer in reinforcement learning control of parameter-varying systems

by   Ibrahim Ahmed, et al.

In this paper, we leverage ideas from model-based control to address the sample efficiency problem of reinforcement learning (RL) algorithms. Accelerating learning is an active field of RL highly relevant in the context of time-varying systems. Traditional transfer learning methods propose to use prior knowledge of the system behavior to devise a gradual or immediate data-driven transformation of the control policy obtained through RL. Such transformation is usually computed by estimating the performance of previous control policies based on measurements recently collected from the system. However, such retrospective measures have debatable utility with no guarantees of positive transfer in most cases. Instead, we propose a model-based transformation, such that when actions from a control policy are applied to the target system, a positive transfer is achieved. The transformation can be used as an initialization for the reinforcement learning process to converge to a new optimum. We validate the performance of our approach through four benchmark examples. We demonstrate that our approach is more sample-efficient than fine-tuning with reinforcement learning alone and achieves comparable performance to linear-quadratic-regulators and model-predictive control when an accurate linear model is known in the three cases. If an accurate model is not known, we empirically show that the proposed approach still guarantees positive transfer with jump-start improvement.


page 1

page 2

page 3

page 4


Fractional Transfer Learning for Deep Model-Based Reinforcement Learning

Reinforcement learning (RL) is well known for requiring large amounts of...

Reinforcement Learning with Partial Parametric Model Knowledge

We adapt reinforcement learning (RL) methods for continuous control to b...

Pre-trained Word Embeddings for Goal-conditional Transfer Learning in Reinforcement Learning

Reinforcement learning (RL) algorithms typically start tabula rasa, with...

On the Feasibility of Cross-Task Transfer with Model-Based Reinforcement Learning

Reinforcement Learning (RL) algorithms can solve challenging control pro...

Towards Evaluating Adaptivity of Model-Based Reinforcement Learning Methods

In recent years, a growing number of deep model-based reinforcement lear...

Model-Based Reinforcement Learning for Physical Systems Without Velocity and Acceleration Measurements

In this paper, we propose a derivative-free model learning framework for...

Adaptive Energy Management for Real Driving Conditions via Transfer Reinforcement Learning

This article proposes a transfer reinforcement learning (RL) based adapt...

Please sign up or login with your details

Forgot password? Click here to reset