Will Kearns, Aakash Sur, BHI PhD Students and Trevor Cohen, BHI Faculty

Tuesdays, Autumn Quarter: 11:30 am-12:20 pm, Health Sciences Building, T478

Through this course, students will be introduced to reinforcement learning methods and how to apply them to train health dialog systems to address specific problems in healthcare. We will cover a range of machine learning methods including tree search, tree pruning, Markov decision processes, and Q-learning. We will explore both classical methods and recent advances in the development of dialog system components including natural language understanding, dialog management, and natural language generation. The course structure will be a mixture of lectures and interactive coding sessions culminating in the deployment of a health dialog system.

We welcome questions during the class as others might share the same questions. If you need individual help, please see one of the instructors after class or send a question to the group on slack.

Neural Approaches to Conversational AI Question Answering, Task-Oriented Dialogues and Social Chatbots

Designing Voice User Interfaces: Principles of Conversational Experiences

Introduction to Conversational Agents and Reinforcement Learning

We will introduce conversational agents operating within a natural language environment, an ideal context for employing reinforcement learning (RL). We will survey the methods of RL and its applications of NLP. Finally, we will cover the software requirements for the class, and ensure students can interactively follow along in coding exercises.

Week 1 Slides

Write an NLU model

None

Natural Language Understanding

We will explore how we can train agents to understand their conversational environments.

Week 2 Slides

ONENET:Joint Domain, Intent, Slot Prediction for Spoken Language Understanding

Dynamic Integration of Background Knowledge in Neural NLU Systems

Train NLU model: with a Pipeline in Rasa

Tree Search

We will model decisions as trees and learn to efficiently search them using classic algorithms such as breadth-first search and depth-first search. In addition, we will introduce heuristic based searches, including A* search.

Week 3 Slides

Joint A* CCG Parsing and Semantic Role Labelling

Week 3 Coding

Advanced Tree Searches

We will cover how to model two player games as trees, and how the optimal strategy can be recovered from these trees. In addition, we will cover how to prune these trees to limit the total search space using alpha-beta pruning, and heuristic pruning.

Week 4 Slides

None

In Class Exercise with Grundy's Game of Nim

Dialog Management

Health dialog systems for patients and consumers

An overview of end-to-end language understanding and dialog management for personal digital assistants

Train Rasa DM with Interactive Learning

Markov Decision Processes

In this class, we will extend our tree based decision models to graphs with Markov models. We will learn how to calculate the best route through a Markov decision process (MDPs) using the Bellman equations. Finally, we will extend these ideas to conversational agents using partially observable Markov decision processes (POMDPs).

Week 6

POMDP-Based StatisticalSpoken Dialog Systems:A Review

Training a real-world POMDP-based Dialogue System

None

Q-Learning

Here we will introduce one of the key concepts in RL, Q-learning. This approach overcomes the limitations of MDPs and allows us to conduct on-line or off-line learning without complete information.

Week 7

TBD

TBD

Deep Q-Networks

Moving past basic tabular Q-learning, we will cover current approaches which revolve around Deep Q-Networks (DQN). We will cover popular examples of DQNs used to master video games, and conversations. Finally, we will cover how to efficiently train these models using experience replay.

Week 8

Playing Atari with Deep Reinforcement Learning

Agenda-based user simulation for bootstrapping a POMDP dialogue system

A User Simulator for Task-Completion Dialogues

Week 8: Deep Learning

Week 8: Cart Pole

Train dialog policy w/ episodic replay

Train a DQN using an agenda based User Simulator

Advanced Neural Methods for Dialog Systems

We will cover advanced neural architectures for training dialog systems, e.g. A2C and MemNN models.

ConveRT: Efficient and Accurate Conversational Representations from Transformers

The Illustrated Transformer (Transformer Tutorial)

TBD

Ethics and NLG

We will finish off the course with a discussion of ethics in the development of dialog systems using two case studies and then focus the discussion on health dialog systems in particular.

The Design and Implementation of XiaoIce, an Empathetic Social Chatbot

Twitter taught Microsoft’s AI chatbot to be a racist asshole in less than a day

TBD