optimal-control-and-reinforcement-learning

1.book

1.1 Reinforcement learning and optimal control

Dimitri P. Bertsekas

The purpose of the book is to consider large and challenging multistage decision problems, which can be solved in principle by dynamic programming and optimal control, but their exact solution is computationally intractable. We discuss solution methods that rely on approximations to produce suboptimal policies with adequate performance. These methods are collectively referred to as reinforcement learning, and also by alternative names such as approximate dynamic programming, and neuro-dynamic programming.

Our subject has benefited enormously from the interplay of ideas from optimal control and from artificial intelligence. One of the aims of this monograph is to explore the common boundary between these two fields and to form a bridge that is accessible by workers with background in either field.

The mathematical style of the book is somewhat different from the author's dynamic programming books, and the neuro-dynamic programming monograph, written jointly with John Tsitsiklis. We rely more on intuitive explanations and less on proof-based insights. Still we provide a rigorous short account of the theory of finite and infinite horizon dynamic programming, and some basic approximation methods, in an appendix. For this we require a modest mathematical background: calculus, elementary probability, and a minimal use of matrix-vector algebra.

The methods of this book have been successful in practice, and often spectacularly so, as evidenced by recent amazing accomplishments in the games of chess and Go. However, across a wide range of problems, their performance properties may be less than solid. This is a reflection of the state of the art in the field: there are no methods that are guaranteed to work for all or even most problems, but there are enough methods to try on a given challenging problem with a reasonable chance that one or more of them will be successful in the end. Accordingly, we have aimed to present a broad range of methods that are based on sound principles, and to provide intuition into their properties, even when these properties do not include a solid performance guarantee. Hopefully, with enough exploration with some of these methods and their variations, the reader will be able to address adequately his/her own problem.

2.courses

2.1 AA 203: Optimal and Learning-Based Control

Optimal control solution techniques for systems with known and unknown dynamics. Dynamic programming, Hamilton-Jacobi reachability, and direct and indirect methods for trajectory optimization. Introduction to model predictive control. Adaptive control, model-based and model-free reinforcement learning, and connections between modern reinforcement learning and fundamental optimal control ideas.

2.2 Reinforcement learning and optimal control for robotics (ROB-GY 6323)

What kind of movements should a robot perform in order to walk, jump or manipulate objects? Can it compute optimal behaviors online? Can it learn this directly from trial and error? This course will introduce modern methods for robotics movement generation based on numerical optimal control and reinforcement learning. It will cover fundamental topics in numerical optimal control (Bellman equations, differential dynamic programming, model predictive control) and reinforcement learning (actor-critic algorithms, model-based reinforcement learning, deep reinforcement learning) applied to robotics. It will also contain hands-on exercises for real robotic applications such as walking and jumping, object manipulation or acrobatic drones. Recommended background in at least one of the following: linear systems; robotics; machine learning; convex optimization; programming (python).

2.3 Optimal Control and Reinforcement Learning @CMU

This course surveys the use of optimization to design behavior. We will explore ways to represent policies including hand-designed parametric functions, basis functions, tables, and trajectory libraries. We will also explore algorithms to create policies including parameter optimization and trajectory optimization (first and second order gradient methods, sequential quadratic programming, random search methods, evolutionary algorithms, etc.). We will discuss how to handle the discrepancy between models used to create policies and the actual system being controlled (evaluation and robustness issues). The course will combine lectures, student-presented material, and projects. The goal of this course will be to help participants find the most effective methods for their problems.

2.4 Model Predictive Control and Reinforcement Learning

This block course of 8 days duration is intended for master and PhD students from engineering, computer science, mathematics, physics, and other mathematical sciences. The aim is that participants understand the main concepts of model predictive control (MPC) and reinforcement learning (RL) as well the similarities and differences between the two approaches. In hands-on exercises and project work they learn to apply the methods to practical optimal control problems from science and engineering.

3.researcher

Sergey Levine @ UC Berkeley

Zac Manchester @ CMU

4.organization

4.1 Robotic Artificial Intelligence and Learning Lab(RAIL)

Welcome to the RAIL lab website! Our research focus is to enable machines to exhibit flexible and adaptable behavior, acquired autonomously through learning. To that end, we work on learning algorithms, robotics, and computer vision.

4.2 Stanford Autonomous Systems Lab

The Autonomous Systems Lab (ASL) develops methodologies for the analysis, design, and control of autonomous systems, with a particular emphasis on large-scale robotic networks and autonomous aerospace vehicles. The lab, comprised of 18 researchers, combines expertise from control theory, robotics, optimization, and operations research to develop the theoretical foundations for networked autonomous systems operating in uncertain, rapidly-changing, and potentially adversarial environments. Our work has been recognized with several awards, including a Presidential Early Career Award for Scientists and Engineers.

https://github.com/StanfordASL

4.3 System control and optimization laboratory

The research focus of the laboratory is on optimization and control with a current application focus on robotic, mechatronic and renewable energy systems. The core area of expertise are embedded optimization algorithms, i.e. methods for real-time optimization on embedded platforms, with a focus on nonlinear systems. The work of the group spans from dynamic system modelling and optimal control problem formulations to open-source software development and real-world control implementations. The group’s interdisciplinary work is located between numerical mathematics, computer science, and control engineering.

4.4 The Robotic Exploration Lab @ Carnegie Mellon University

The Robotic Exploration Lab in The Robotics Institute at Carnegie Mellon University conducts research in control, motion planning, and navigation for robotic systems that explore our planet and our universe.