The authors group tools that support planning in sections: maps, communication, traffic rules, middleware, simulators and benchmarks. Source.

About simulators and dataset. And how to couple between tools, either with co-simulation software or open interfaces. Source.

About maps. ''The planning tasks with different targets entail map models with different level of details. HD map provides the most sufficient information and can be generally categorized into three layers: road model, lane model and localization model''. Source.

Authors: Tong, K., Ajanovic, Z., & Stettinger, G.

• Motivations:

  • 1- Help researchers to make full use of open-source resources and reduce effort of setting up a software platform that suites their needs.

    • [example] "It is a good option to choose open-source Autoware as software stack along with ROS middleware, as Autoware can be further transferred to a real vehicle. During the development, he or she can use Carla as a simulator, to get its benefits of graphic rendering and sensor simulation. To make the simulation more realistic, he or she might adopt commercial software CarMaker for sophisticated vehicle dynamics and open-source SUMO for large-scale traffic flow simulation. OpenDRIVE map can be used as a source and converted into the map format of Autoware, Carla and SUMO. Finally, CommonRoad can be used to evaluate the developed algorithm and benchmark it against other approaches."

  • 2- Avoid reinventing the wheel.
    • Algorithms are available/adapted from robotics.
    • Simulators are available/adapted from gaming.

• Mentioned software libraries for motion planning:

  • ROS (from robotics):
    • Open Motion Planning Library (OMPL)
    • MoveIt
    • navigation package
    • teb local planner.
  • Python: PythonRobotics
  • CPP: CppRobotics

• How to express traffic rules in a form understandable by an algorithm?

  • 1- Traffic rules can be formalized in higher order logic (e.g. using the Isabelle theorem prover) to check the compliance of traffic rules unambiguously and formally for trajectory validation.
  • 2- Another approach is to represent traffic rules geometrically as obstacles in a configuration space of motion planning problem.

  • "In some occasions, it is necessary to violate some rules during driving for achieving higher goals (i.e. avoiding collision) [... e.g. with] a rule book with hierarchical arrangement of different rules."

• About data visualization?

  • RViz is a popular tool in ROS for visualizing data flow, as I also realized at IV19.
  • Apart from that, it seems each team is having its own specific visualization tool, sometimes released, as AVS from UBER and GM Cruise.

• What is missing for the research community?

  • Evaluation tools for quantitative comparison.
  • Evaluation tools incorporating human judgment, not only from the vehicle occupants but also from other road users.
  • A standard format for motion datasets.

• I am surprised INTERACTION dataset was not mentioned.

Both urban and highway behaviour options are combined using a cost-based arbitrator. Together with Parking and AvoidCollisionInLastResort, these four arbitrators and the SafeStop fallback are composed together to the top-most priority-based AutomatedDriving arbitrator. Source.

Top-right: two possible options. The arbitrator generally prefers the follow lane behaviour as long as it matches the route. Here, a lane change is necessary and selected by the cost-based arbitration: ChangeLaneRight has lower cost than FollowEgoLane, mainly due to the routing term in the cost expression. Bottom: the resulting behaviour selection over time. Source.

Authors: Orzechowski, P. F., Burger, C., & Lauer, M.

• Motivation:

  • Propose an alternative to FSMs (finite state machines) and behaviour-based systems (e.g. voting systems) in hierarchical architectures.
  • In particular, FSMs can suffer from:
    • poor interpretability: why is one behaviour executed?
    • maintainability: effort to refine existing behaviour.
    • scalability: effort to achieve a high number of behaviours and to combine a large variety of scenarios.
    • options selection: "multiple behaviour options are applicable but have no clear and consistent priority against each other."

      • "How and when should an automated vehicle switch from a regular ACC controller to a lane change, cooperative zip merge or parking planner?"

    • multiple planners: Each behaviour component can compute its manoeuvre command with any preferred state-of-the-art method.

      • "How can we support POMDPs, hybrid A* and any other planning method in our behaviour generation?".

• Main idea:

  • cost-based arbitration between so-called "behaviour components".
  • The modularity of these components brings several advantages:
    • Various scenarios can be handled within a single framework: four-way intersections, T-junctions, roundabout, multilane bypass roads, parking, etc.
    • Hierarchically combining behaviours, complex behaviour emerges from simple components.
    • Good efficiency: the atomic structure allows to evaluate behaviour options in parallel.

• About arbitration:

  • "An arbitrator contains a list of behavior options to choose from. A specific selection logic determines which option is chosen based on abstract information, e.g., expected utility or priority."

  • [about cost] "The cost-based arbitrator selects the behavior option with the lowest expected cost."

  • Each behaviour option is evaluated based on its expected average travel velocity, incorporating routing costs and penalizing lane changes.
    • The resulting behaviour can thus be well explained:

    • "The selection logic of arbitrators is comprehensive."

  • About hierarchy:

    • "To generate even more complex behaviours, an arbitrator can also be a behaviour option of a hierarchically higher arbitrator."

• About behaviour components.

  • There are the smallest building blocks, representing basic tactical driving manoeuvres.
  • Example of atomic behaviour components for simple tasks in urban scenarios:
    • FollowLead
    • CrossIntersection
    • ChangeLane
  • They can be specialized:
    • Dense scenarios behaviours: ApproachGap, IndicateIntention and MergeIntoGap to refine ChangeLane (multi-phase behaviour).
      • Note: an alternative could be to use one single integrated interaction-aware behaviour such as POMDP.
    • Highway behaviours (structured but high speed): MergeOntoHighway, FollowHighwayLane, ChangeHighwayLane, ExitFromHighway.
    • Parking behaviours: LeaveGarage, ParkNearGoal.
    • Fail-safe emergency behaviours: EmergenyStop, EvadeObject, SafeStop.
  • For a behaviour to be selected, it should be applicable. Hence a behaviour is defined together with:
    • invocation condition: when does it become applicable.

      • "[example:] The invocation condition of CrossIntersection is true as long as the current ego lane intersects other lanes within its planning horizon."

    • commitment condition: when does it stay applicable.
  • This reminds me the concept of macro actions, sometimes defined by a tuple <applicability condition, termination condition, primitive policy>.
  • It also makes me think of MODIA framework and other scene-decomposition approaches.

• A mid-to-mid approach:

  • "[input] The input is an abstract environment model that contains a fused, tracked and filtered representation of the world."

  • [output] The selected high-level decision is passed to a trajectory planner or controller.
  • What does the "decision" look like?
    • One-size-fits-all is not an option.
    • It is distinguished between maneuvers in a structured or unstructured environment:
    • 1- unstructured: a trajectory, directly passed to a trajectory planner.
    • 2- structured: a corridor-based driving command, i.e. a tuple <maneuver corridor, reference line, predicted objects, maneuver variant>. It requires both a trajectory planner and a controller.

• One distinction:

  • 1- top-down knowledge-based systems.

    • "The action selection in a centralized, in a top-down manner using a knowledge database."

    • "The engineer designing the action selection module (in FSMs the state transitions) has to be aware of the conditions, effects and possible interactions of all behaviors at hand."

  • 2- bottom-up behaviour-based systems.

    • "Decouple actions into atomic simple behaviour components that should be aware of their conditions and effects."

    • E.g. voting systems.
  • Here the authors combine atomic behaviour components (bottom/down) with more complex behaviours using generic arbitrators (top/up).

The review divides motion-planning into five parts. The decision-making part contains risk evaluation, criteria minimization, and constraint submission. In the last part, a low-level reactive planner deforms the generated motion from the high-level planner. Source.

The review offers two detailed tools for comparing methods for motion planning for highway scenarios. Criteria for the generated motion include: feasible, safe, optimal, usable, adaptive, efficient, progressive and interactive. The authors stressed the importance of spatiotemporal consideration and emphasize that highway motion-planning is highly structured. Source.

Contrary to solve-algorithms methods, set-algorithm methods require a complementary algorithm should be added to find the feasible motion. Depending on the importance of the generation (iv) and deformation (v) part, approaches are more or less reactive or predictive. Finally, based on their work on AI-based algorithms, the authors define four subfamilies to compare to human: logic, heuristic, approximate reasoning, and human-like. Source.

The review also offers overviews for possible space configurations, i.e. the choices for decomposition of the evolution space (sampling-based decomposition, connected cells decomposition and lattice representation) as well as path-finding algorithms (e.g. Dijkstra, A*, and RRT). attractive and repulsive forces, parametric and semi-parametric curves, numerical optimization and artificial intelligence are also developed. Source.

Authors: Claussmann, L., Revilloud, M., Gruyer, D., & Glaser, S.

Challenges for learning-based control methods. Source.

Authors: Kuutti, S., Bowden, R., Jin, Y., Barber, P., & Fallah, S.

• Three categories are examined:
  • lateral control alone.
  • longitudinal control alone.
  • longitudinal and lateral control combined.
• Two quotes:

  • "While lateral control is typically achieved through vision, the longitudinal control relies on measurements of relative velocity and distance to the preceding/following vehicles. This means that ranging sensors such as RADAR or LIDAR are more commonly used in longitudinal control systems.".

  • "While lateral control techniques favour supervised learning techniques trained on labelled datasets, longitudinal control techniques favour reinforcement learning methods which learn through interaction with the environment."

mMP refers to machine learning methods for longitudinal motion planning. Source.

Authors: Zhou, H., & Laval, J.

• This review has been completed at a school of "civil and environmental engineering".
  • It does not have any scientific contribution, but offers a quick overview about some current trends in decision-making.
  • The authors try to look at industrial applications (e.g. Waymo, Uber, Tesla), i.e. not just focussing on theoretical research. Since companies do no communicate explicitly about their approaches, most of their publications should be considered as research side-projects, rather than "actual state" of the industry.
• One focus of the review: the machine learning approaches for decision-making for longitudinal motion.
  • About the architecture and representation models. They mention the works of DeepDriving and (H. Xu, Gao, Yu, & Darrell, 2016).
    • Mediated perception approaches parse an entire scene to make a driving decision.
    • Direct perception approaches first extract affordance indicators (i.e. only the information that are important for driving in a particular situation) and then map them to actions.

      • "Only a small portion of detected objects are indeed related to the real driving reactions so that it would be meaningful to reduce the number of key perception indicators known as learning affordances".

    • Behavioural reflex approaches directly map an input image to a driving action by a regressor.
      • This end-to-end paradigm can be extended with auxiliary tasks such as learning semantic segmentation (this "side task" should further improves the model), leading to Privileged training.
  • About the learning methods:
    • BC, RL, IRL and GAIL are considered.
    • The authors argue that their memory and prediction abilities should make them stand out from the rule-based approaches.

    • "Both BC and IRL algorithms implicitly assume that the demonstrations are complete, meaning that the action for each demonstrated state is fully observable and available."

    • "We argue that adopting RL transforms the problem of learnt longitudinal motion planning from imitating human demonstrations to searching for a policy complying a hand-crafted reward rule [...] No studies have shown that a genuine reward function for human driving really exists."

• About congestion:

  • "The AV industry has been mostly focusing on the long tail problem caused by corner errors related to safety, while the impact of AVs on traffic efficiency is almost ignored."

  • It reminds me the finding of (Kellett, J., Barreto, R., Van Den Hengel, A. & Vogiatzis. N., 2019) in "How Might Autonomous Vehicles Impact the City? The Case of Commuting to Central Adelaide": driverless cars could lead to more traffic congestion.

Some figures:

The focus is on the BP module, together with its predecessor (environment) and its successor (LP) in a modular architecture. Source.

Classification for Question 1 - environment representation. A combination is possible. In black, my notes giving examples. Source.

Classification for Question 2 - on the architecture. Source.

Classification for Question 3 - on the decision logic representation. Source.

Authors: Ilievski, M., Sedwards, S., Gaurav, A., Balakrishnan, A., Sarkar, A., Lee, J., Bouchard, F., De Iaco, R., & Czarnecki K.

The authors divide their review into three sections:

• Question 1: How to represent the environment? (relation with predecessor of BP)
  • Four representations are compared: raw data, feature-based, grid-based and latent representation.
• Question 2: How to communicate with other modules, especially the local planner (LP)? (relation with successor (LP) of BP)
  • A first sub-question is the relevance of separation BP / LP.
    • A complete separation (top-down) can lead to computational redundancy (both have a collision checker).
    • One idea, close to sampling techniques, could be to invert the traditional architecture for planning, i.e. generate multiple possible local paths (~LP) then selects the best manoeuvre according to a given cost function (~BP). But this exasperates the previous point.
  • A second sub-question concerns prediction: Should the BP module have its own dedicated prediction module?
    • First, three kind of prediction are listed, depending on what should be predicted (marked with ->):
      • Physics-based (-> trajectory).
      • Manoeuvre-based (-> low-level motion primitives).
      • Interaction-aware (-> intent).
    • Then, the authors distinguish between explicitly-defined and implicitly-defined prediction models:
      • Explicitly-defined can be:
        • Integrated with the motion planning process (called Internal prediction models) such as belief-based decision making (e.g. POMDP). Ideal for planning under uncertainty.
        • Decoupled from the planning process (called External prediction models). There is a clear interface between prediction and planning, which aids modularity.
      • Implicitly-defined, such as RL techniques.
• Question 3: How to make BP decisions? (BP itself)
  • A first distinction in representation of decision logic is made depending based on non-learnt / learnt:
    • Using a set of explicitly programmed production rules can be divided into:
      • Imperative approaches, e.g. state machines.
      • Declarative approaches often based on some probabilistic system.
        • The decision-tree structure and the (PO)MDP formulation makes it more robust to uncertainty.
        • Examples include MCTS and online POMDP solvers.
    • Logic representation can also rely on mathematical models with parameters learned a priori.
      • A distinction is made depending on "where does the training data come from and when is it created?".
      • In other words, one could think of supervised learning (learning from example) versus reinforcement learning (learning from interaction).
      • The combination of both seems beneficial:
        • An initial behaviour is obtained through imitation learning (learning from example). Also possible with IRL.
        • But improvements are made through interaction with a simulated environment (learning from interaction).
          • By the way, the learning from interaction techniques raise the question of the origin of the experience (e.g. realism of the simulator) and its sampling efficiency.
      • Another promising direction is hierarchical RL where the MDP is divided into sub-problems (the lower for LP and the higher for BP)
        • The lowest level implementation of the hierarchy approximates a solution to the control and LP problem ...
        • ... while the higher level selects a manoeuvre to be executed by the lower level implementations.
  • As mentioned in my the section on Scenarios and Datasets, the authors mention the lack of benchmark to compare and evaluate the performance of BP technologies.

One quote about the representation of decision logic:

• As identified in my notes about IV19, the combination of learnt and non-learnt approaches looks the most promising.

• "Without learning, traditional robotic solutions cannot adequately handle complex, dynamic human environments, but ensuring the safety of learned systems remains a significant challenge."

• "Hence, we speculate that future high performance and safe behaviour planning solutions will be hybrid and heterogeneous, incorporating modules consisting of learned systems supervised by programmed logic."

Some figures:

Comparison and fusion of the hierarchical and parallel architectures. Source.

The PCB algorithm implemented in the BP module. Source.

Related work by (Xu, Pan, Wei, & Dolan, 2014) - Grey ellipses indicate the magnitude of the uncertainty of state. Source.

Authors: Wei, J., Snider, J. M., & Dolan, J. M.

Note: I find very valuable to get insights from the CMU (Carnegie Mellon University) Team, based on their experience of the DARPA Urban Challenges.

• Related works:
  • A prediction- and cost function-based algorithm for robust autonomous freeway driving. 2010 by (Wei, Dolan, & Litkouhi, 2010).
    • They introduced the "Prediction- and Cost-function Based (PCB) algorithm" used.
    • The idea is to generate-forward_simulate-evaluate a set of manoeuvres.
    • The planner can therefore take surrounding vehicles’ reactions into account in the cost function when it searches for the best strategy.
    • At the time, the authors rejected the option of a POMDP formulation (computing the control policy over the space of the belief state, which is a probability distribution over all the possible states) deemed as computationally expensive. Improvements in hardware and algorithmic have been made since 2014.
  • Motion planning under uncertainty for on-road autonomous driving. 2014 by (Xu, Pan, Wei, & Dolan, 2014).
    • An extension of the framework to consider uncertainty (both for environment and the others participants) in the decision-making.
    • The prediction module is using a Kalman Filter (assuming constant velocity).
    • For each candidate trajectory, the uncertainty can be estimated using a Linear-Quadratic Gaussian (LQG) framework (based on the noise characteristics of the localization and control).
    • Their Gaussian-based method gives some probabilistic safety guaranty (e.g. likelihood 2% of collision to occur).
• Proposed architecture for decision-making:
  • First ingredient: Hierarchical architecture.
    • The hierarchy mission -> manoeuvre -> motion 3M concept makes it very modular but can raise limitations:

    • "the higher-level decision making module usually does not have enough detailed information, and the lower-level layer does not have authority to re-evaluate the decision."

  • Second ingredient: Parallel architecture.
    • This is inspired from ADAS engineering.
    • The control modules (ACC, Merge Assist, Lane Centreing) are relatively independent and work in parallel.
    • In some complicated cases needing cooperation, this framework may not perform well.
      • This probably shows that just extending the common ADAS architectures cannot be enough to reach the level-5 of autonomy.
  • Idea of the proposed framework: combine the strengths of the hierarchical and parallel architectures.
    • This relieves the path planner and the control module (the search space is reduced).
    • Hence the computational cost shrinks (by over 90% compared to a sample-based planner in the spatio-temporal space).
• One module worth mentioning: Traffic-free Reference Planner.
  • Its input: lane-level sub-missions from the Mission Planning.
  • Its output: kinematically and dynamically feasible paths and a speed profile for the Behavioural Planner (BP).
    • It assumes there is no traffic on the road, i.e. ignores dynamic obstacles.
    • It also applies traffic rules such as speed limits.
  • This guides the BP layer which considers both static and dynamic obstacles to generate so-called "controller directives" such as:
    • The lateral driving bias.
    • The desired leading vehicle to follow.
    • The aggressiveness of distance keeping.
    • The maximum speed.

Left: example of end-to-end architecture with key terms. Right: difference open-loop / close-loop evaluation. Source.

Source.

Source.

Authors: Tampuu, A., Semikin, M., Muhammad, N., Fishman, D., & Matiisen, T.

• A rich literature overview and some useful reminders about general IL and RL concepts with focus to AD applications.

  • It constitutes a good complement to the "Related trends in research" part of my video "From RL to Inverse Reinforcement Learning: Intuitions, Concepts + Applications to Autonomous Driving".

• I especially like the structure of the document: It shows what one should consider when starting an end-to-end / IL project for AD:

  • I have just noted here some ideas I find interesting. In no way an exhaustive summary!

• 1- Learning methods: working with rewards (RL) or with losses (behavioural cloning).

  • About distribution shift problem in behavioural cloning:

    • "If the driving decisions lead to unseen situations (not present in the training set), the model might no longer know how to behave".

    • Most solutions try to diversify the training data in some way - either by collecting or generating additional data:
      • data augmentation: e.g. one can place two additional cameras pointing forward-left and forward-right and associate the images with commands to turn right and turn left respectively.
      • data diversification: addition of temporally correlated noise and synthetic trajectory perturbations. Easier on "semantic" inputs than on camera inputs.
      • on-policy learning: recovery annotation and DAgger. The expert provides examples how to solve situations the model-driving leads to. Also "Learning by cheating" by (Chen et al. 2019).
      • balancing the dataset: by upsampling the rarely occurring angles, downsampling the common ones or by weighting the samples.

        • "Commonly, the collected datasets contain large amounts of repeated traffic situations and only few of those rare events."

        • The authors claim that only the joint distribution of inputs and outputs defines the rarity of a data point.

        • "Using more training data from CARLA Town1 decreases generalization ability in Town2. This illustrates that more data without more diversity is not useful."

        • Ideas for augmentation can be taken from the field of supervised Learning where it is already an largely-addressed topic.
  • About RL:
    • Policies can be first trained with IL and then fine-tuned with RL methods.

    • "This approach reduces the long training time of RL approaches and, as the RL-based fine-tuning happens online, also helps overcome the problem of IL models learning off-policy".

  • About domain adaptation and transfer from simulation to real world (sim2real).
    • Techniques from supervised learning, such as fine tuning, i.e. adapting the driving model to the new distribution, are rarely used.
    • Instead, one can instead adapt the incoming data and keep the driving model fixed.
      • A first idea is to transform real images into simulation-like images (the opposite - generating real-looking images - is challenging).
      • One can also extract the semantic segmentation of the scene from both the real and the simulated images and use it as the input for the driving policy.

• 2- Inputs.

  • In short:
    • Vision is key.
    • Lidar and HD-map are nice to have but expensive / tedious to maintain.
    • Additional inputs from independent modules (semantic segmentation, depth map, surface normals, optical flow and albedo) can improve the robustness.
  • About the inertia problem / causal confusion when for instance predicting the next ego-speed.

    • "As in the vast majority of samples the current [observed] and next speeds [to be predicted] are highly correlated, the model learns to base its speed prediction exclusively on current speed. This leads to the model being reluctant to change its speed, for example to start moving again after stopping behind another car or a at traffic light."

  • About affordances:

    • "Instead of parsing all the objects in the driving scene and performing robust localization (as modular approach), the system focuses on a small set of crucial indicators, called affordances."

• 3- Outputs.

  • "The outputs of the model define the level of understanding the model is expected to achieve."

  • Also related to the time horizon:

    • "When predicting instantaneous low-level commands, we are not explicitly forcing the model to plan a long-term trajectory."

  • Three types of predictions:
    • 3-1 Low-level commands.

      • "The majority of end-to-end models yield as output the steering angle and speed (or acceleration and brake commands) for the next timestep".

      • Low-level commands may be car-specific. For instance vehicles answer differently to the same throttle / steering commands.

        • "The function between steering wheel angle and the resulting turning radius depends on the car's geometry, making this measure specific to the car type used for recording."

      • [About the regression loss] "Many authors have recently optimized speed and steering commands using L1 loss (mean absolute error, MAE) instead of L2 loss (mean squared error, MSE)".

    • 3-2 Future waypoints or desired trajectories.
      • This higher-level output modality is independent of car geometry.
    • 3-3 Cost map, i.e. information about where it is safe to drive, leaving the trajectory generation to another module.
  • About multitask learning and auxiliary tasks:
    • The idea is to simultaneously train a separate set of networks to predict for instance semantic segmentation, optical flow, depth and other human-understandable representations from the camera feed.

    • "Based on the same extracted visual features that are fed to the decision-making branch (main task), one can also predict ego-speed, drivable area on the scene, and positions and speeds of other objects".

    • It offers more learning signals - at least for the shared layers.
    • And can also help understand the mistakes a model makes:

      • "A failure in an auxiliary task (e.g. object detection) might suggest that necessary information was not present already in the intermediate representations (layers) that it shared with the main task. Hence, also the main task did not have access to this information and might have failed for the same reason."

• 4- Evaluation: the difference between open-loop and close-loop.

  • 4-1 open-loop: like in supervised learning:
    • one question = one answer.
    • Typically, a dataset is split into training and testing data.
    • Decisions are compared with the recorded actions of the demonstrator, assumed to be the ground-truth.
  • 4-2 close-loop: like in decision processes:
    • The problem consists in a multi-step interaction with some environment.
    • It directly measures the model's ability to drive on its own.
  • Interesting facts: Good open-loop performance does not necessarily lead to good driving ability in closed-loop settings.

    • "Mean squared error (MSE) correlates with closed-loop success rate only weakly (correlation coefficient r = 0.39), so MAE, quantized classification error or thresholded relative error should be used instead (r > 0.6 for all three)."

    • About the balanced-MAE metric for open-loop evaluation, which correlates better with closed-loop performance than simple MAE.

      • "Balanced-MAE is computed by averaging the mean values of unequal-length bins according to steering angle. Because most data lies in the region around steering angle 0, equally weighting the bins grows the importance of rarely occurring (higher) steering angles."

• 5- Interpretability:

  • 5-1 Either on the trained model ...

    • "Sensitivity analysis aims to determine the parts of an input that a model is most sensitive to. The most common approach involves computing the gradients with respect to the input and using the magnitude as the measure of sensitivity."

    • VisualBackProp: which input pixels influence the cars driving decision the most.
  • 5-2 ... or already during training.

    • "visual attention is a built-in mechanism present already when learning. Where to attend in the next timestep (the attention mask), is predicted as additional output in the current step and can be made to depend on additional sources of information (e.g. textual commands)."

• About end-to-end neural nets and humans:

  • "[StarCraft, Dota 2, Go and Chess solved with NN]. Many of these solved tasks are in many aspects more complex than driving a car, a task that a large proportion of people successfully perform even when tired or distracted. A person can later recollect nothing or very little about the route, suggesting the task needs very little conscious attention and might be a simple behavior reflex task. It is therefore reasonable to believe that in the near future an end-to-end approach is also capable to autonomously control a vehicle."

The latent representation is enforced to predict the trajectories of both the ego vehicle and other vehicles in addition to the input image, using a multi-head network structure. Source.

Authors: Kargar, E., & Kyrki, V.

• Motivations:
  • 1- Reduce the dimensionality of the feature representation of the scene - used as input to some IL / RL policy.
    • This is to improve most mid-to-x approaches that encode and process a vehicle’s environment as multi-channel and quite high-dimensional bird view images.
    • -> The idea here is to learn an encoder-decoder.
    • The latent space has size 64 (way smaller than common 64 x 64 x N bird-views).
  • 2- Learn a latent representation faster / with fewer data.
    • A single head decoder would just consider reconstruction.
    • -> The idea here is to use have multiple heads in the decoder, i.e. make prediction of multiple auxiliary application relevant factors.

    • "The multi-head model can reach the single-head model’s performance in 20 epochs, one-fifth of training time of the single-head model, with full dataset."

    • "In general, the multi-heal model, using only 6.25% of the dataset, converges faster and perform better than single head model trained on the full dataset."

  • 3- Learn a policy faster / with fewer data.
• Two components to train:
  • 1- An encoder-decoder learns to produce a latent representation (encoder) coupled with a multiple-prediction-objective (decoder).
  • 2- A policy use the latent representation to predict low-level controls.
• About the encoder-decoder:
  • inputs: bird-view image containing:
    • Environment info, built from HD Maps and perception.
    • Ego trajectory: 10 past poses.
    • Other trajectory: 10 past poses.
    • It forms a 256 x 256 image, which is resized to 64 x 64 to feed them into the models
  • outputs: multiple auxiliary tasks:
    • 1- Reconstruction head: reconstructing the input bird-view image.
    • 2- Prediction head: 1s-motion-prediction for other agents.
    • 3- Planning head: 1s-motion-prediction for the ego car.
• About the policy:
  • In their example, the authors implement behaviour cloning, i.e. supervised learning to reproduce the decision of CARLA autopilot.
  • 1- steering prediction.
  • 2- acceleration classification - 3 classes.
• How to deal with the unbalanced dataset?
  • First, the authors note that no manual labelling is required to collect training data.
  • But the recorded steering angle is zero most of the time - leading to a highly imbalanced dataset.
  • Solution (no further detail):

    • "Create a new dataset and balance it using sub-sampling".

Illustration of the state distribution shift in behavioural cloning (BC) approaches. The models (e.g. neural networks) usually fail to generalize and instead extrapolate confidently yet incorrectly, resulting in arbitrary outputs and dangerous outcomes. Not to mention the compounding (or cascading) errors, inherent to the sequential decision making. Source.

Testing behaviours on scenarios such as roundabouts that are not present in the training set. Source.

Above - in their previous work, the authors introduced Deep imitative models (IM). The imitative planning objective is the log posterior probability of a state trajectory, conditioned on satisfying some goal G. The state trajectory that has the highest likelihood w.r.t. the expert model q(S given φ; θ) is selected, i.e. maximum a posteriori probability (MAP) estimate of how an expert would drive to the goal. This captures any inherent aleatoric stochasticity of the human behaviour (e.g., multi-modalities), but only uses a point-estimate of θ, thus q(s given φ;θ) does not quantify model (i.e. epistemic) uncertainty. φ denotes the contextual information (3 previous states and current LIDAR observation) and s denotes the agent’s future states (i.e. the trajectory). Bottom - in this works, an ensemble of models is used: q(s given φ; θk) where θk denotes the parameters of the k-th model (neural network). The Aggregation Operator operator is applied on the posterior p(θ given D). The previous work is one example of that, where a single θi is selected. Source.

To save computation and improve runtime to real-time, the authors use a trajectory library: they perform K-means clustering of the expert plan’s from the training distribution and keep 128 of the centroids, allegedly reducing the planning time by a factor of 400. During optimization, the trajectory space is limited to only that trajectory library. It makes me think of templates sometimes used for path-planning. I also see that as a way restrict the search in the trajectory space, similar to injecting expert knowledge about the feasibility of cars trajectories. Source.

Estimating the uncertainty is not enough. One should then forward that estimate to the planning module. This reminds me an idea of (McAllister et al., 2017) about the key benefit of propagating uncertainty throughout the AV framework. Source.

Authors: Tigas, P., Filos, A., Mcallister, R., Rhinehart, N., Levine, S., & Gal, Y.

• Previous work: "Deep Imitative Models for Flexible Inference, Planning, and Control" (see below).

  • The idea was to combine the benefits of imitation learning (IL) and goal-directed planning such as model-based RL (MBRL).
    • In other words, to complete planning based on some imitation prior, by combining generative modelling from demonstration data with planning.
    • One key idea of this generative model of expert behaviour: perform context-conditioned density estimation of the distribution over future expert trajectories, i.e. score the "expertness" of any plan of future positions.
  • Limitations:
    • It only uses a point-estimate of θ. Hence it fails to capture epistemic uncertainty in the model’s density estimate.

    • "Plans can be risky in scenes that are out-of-training-distribution since it confidently extrapolates in novel situations and lead to catastrophes".

• Motivations here:

  • 1- Develop a model that captures epistemic uncertainty.
  • 2- Estimating uncertainty is not a goal at itself: one also need to provide a mechanism for taking low-risk actions that are likely to recover in uncertain situations.
    • I.e. both aleatoric and epistemic uncertainty should be taken into account in the planning objective.
    • This reminds me the figure of (McAllister et al., 2017) about the key benefit of propagating uncertainty throughout the AV framework.

• One quote about behavioural cloning (BC) that suffers from state distribution shift (co-variate shift):

  • "Where high capacity parametric models (e.g. neural networks) usually fail to generalize, and instead extrapolate confidently yet incorrectly, resulting in arbitrary outputs and dangerous outcomes".

• One quote about model-free RL:

  • "The specification of a reward function is as hard as solving the original control problem in the first place."

• About epistemic and aleatoric uncertainties:

  • "Generative models can provide a measure of their uncertainty in different situations, but robustness in novel environments requires estimating epistemic uncertainty (e.g., have I been in this state before?), where conventional density estimation models only capture aleatoric uncertainty (e.g., what’s the frequency of times I ended up in this state?)."

• How to capture uncertainty about previously unseen scenarios?

  • Using an ensemble of density estimators and aggregate operators over the models’ outputs.

    • "By using demonstration data to learn density models over human-like driving, and then estimating its uncertainty about these densities using an ensemble of imitative models".

  • The idea it to take the disagreement between the models into consideration and inform planning.

    • "When a trajectory that was never seen before is selected, the model’s high epistemic uncertainty pushes us away from it. During planning, the disagreement between the most probable trajectories under the ensemble of imitative models is used to inform planning."

The visualization of 3D detection, motion forecasting as well as learned cost-map volume offers interpretability. A set of candidate trajectories is sampled, first considering the geometrical path and then then speed profile. The trajectory with the minimum learned cost is selected. Source.

Source.

Authors: Zeng W., Luo W., Suo S., Sadat A., Yang B., Casas S. & Urtasun R.

• Motivation is to bridge the gap between the traditional engineering stack and the end-to-end driving frameworks.

  • 1- Develop a learnable motion planner, avoiding the costly parameter tuning.
  • 2- Ensure interpretability in the motion decision. This is done by offering an intermediate representation.
  • 3- Handle uncertainty. This is allegedly achieved by using a learnt, non-parametric cost function.
  • 4- Handle multi-modality in possible trajectories (e.g changing lane vs keeping lane).

• One quote about RL and IRL:

  • "It is unclear if RL and IRL can scale to more realistic settings. Furthermore, these methods do not produce interpretable representations, which are desirable in safety critical applications".

• Architecture:

  • Input: raw LIDAR data and a HD map.
  • 1st intermediate result: An "interpretable" bird’s eye view representation that includes:
    • 3D detections.
    • Predictions of future trajectories (planning horizon of 3 seconds).
    • Some spatio-temporal cost volume defining the goodness of each position that the self-driving car can take within the planning horizon.
  • 2nd intermediate result: A set of diverse physically possible trajectories (candidates).
    • They are Clothoid curves being sampled. First building the geometrical path. Then the speed profile on it.

    • "Note that Clothoid curves can not handle circle and straight line trajectories well, thus we sample them separately."

  • Final output: The trajectory with the minimum learned cost.

• Multi-objective:

  • 1- Perception Loss - to predict the position of vehicles at every time frame.
    • Classification: Distinguish a vehicle from the background.
    • Regression: Generate precise object bounding boxes.
  • 2- Planning Loss.

    • "Learning a reasonable cost volume is challenging as we do not have ground-truth. To overcome this difficulty, we minimize the max-margin loss where we use the ground-truth trajectory as a positive example, and randomly sampled trajectories as negative examples."

    • As stated, the intuition behind is to encourage the demonstrated trajectory to have the minimal cost, and others to have higher costs.
    • The model hence learns a cost volume that discriminates good trajectories from bad ones.

The main idea is to use Learning from Interventions (LfI) in order to ensure safety and improve data efficiency, by intervening on sub-goals rather than trajectories. Both top-level policy (that generates sub-goals) and bottom-level policy are jointly learnt. Source.

Authors: Bi, J., Dhiman, V., Xiao, T., & Xu, C.

• Motivations:
  • 1- Improve data-efficiency.
  • 2- Ensure safety.
• One term: "Learning from Interventions" (LfI).
  • One way to classify the "learning from expert" techniques is to use the frequency of expert’s engagement.
    • High frequency -> Learning from Demonstrations.
    • Medium frequency -> learning from Interventions.
    • Low frequency -> Learning from Evaluations.
  • Ideas of LfI:

    • "When an undesired state is detected, another policy is activated to take over actions from the agent when necessary."

    • Hence the expert overseer only intervenes when it suspects that an unsafe action is about to be taken.
  • Two issues:
    • 1- LfI (as for LfD) learn reactive behaviours.

      • "Learning a supervised policy is known to have 'myopic' strategies, since it ignores the temporal dependence between consecutive states".

      • Maybe one option could be to stack frames or to include the current speed in the state. But that makes the state space larger.
    • 2- The expert only signals after a non-negligible amount of delay.
• One idea to solve both issues: Hierarchy.
  • The idea is to split the policy into two hierarchical levels, one that generates sub-goals for the future and another that generates actions to reach those desired sub-goals.
  • The motivation is to intervene on sub-goals rather than trajectories.
  • One important parameter: k
    • The top-level policy predicts a sub-goal to be achieved k steps ahead in the future.
    • It represents a trade-off between:
      • The ability for the top-level policy to predict sub-goals far into the future.
      • The ability for the bottom-level policy to follow it correctly.
  • One question: How to deal with the absence of ground- truth sub-goals ?
    • One solution is "Hindsight Experience Replay", i.e. consider an achieved goal as a desired goal for past observations.
    • The authors present additional interpolation techniques.
    • They also present a Triplet Network to train goal-embeddings (I did not understand everything).

The encoder is trained to reconstruct RGB, depth and segmentation, i.e. to learn scene understanding. It is augmented with optical flow for temporal information. As noted, such representations could be learned simultaneously with the driving policy, for example, through distillation. But for efficiency, this was pre-trained (Humans typically also have ~30 hours of driver training before taking the driving exam. But they start with huge prior knowledge). Interesting idea: the navigation command is injected as multiple locations of the control part. Source.

Driving data is inherently heavily imbalanced, where most of the captured data will be driving near-straight in the middle of a lane. Any naive training will collapse to the dominant mode present in the data. No data augmentation is performed. Instead, during training, the authors sample data uniformly across lateral and longitudinal control dimensions. Source.

Authors: Hawke, J., Shen, R., Gurau, C., Sharma, S., Reda, D., Nikolov, N., Mazur, P., Micklethwaite, S., Griffiths, N., Shah, A. & Kendall, A.

• Motivations:
  • 1- Learn both steering and speed via Behavioural Cloning.
  • 2- Use raw sensor (camera) inputs, rather than intermediate representations.
  • 3- Train and test on dense urban environments.
• Why "conditional"?
  • A route command (e.g. turn left, go straight) resolves the ambiguity of multi-modal behaviours (e.g. when coming at an intersection).

  • "We found that inputting the command multiple times at different stages of the network improves robustness of the model".

• Some ideas:
  • Provide wider state observability through multiple camera views (single camera disobeys navigation interventions).
  • Add temporal information via optical flow.
    • Another option would be to stack frames. But it did not work well.
  • Train the primary shared encoders and auxiliary independent decoders for a number of computer vision tasks.

    • "In robotics, the test data is the real-world, not a static dataset as is typical in most ML problems. Every time our cars go out, the world is new and unique."

• One concept: "Causal confusion".
  • A good video about Causal Confusion in Imitation Learning showing that "access to more information leads to worse generalisation under distribution shift".

  • "Spurious correlations cannot be distinguished from true causes in the demonstrations. [...] For example, inputting the current speed to the policy causes it to learn a trivial identity mapping, making the car unable to start from a static position."

  • Two ideas during training:
    • Using flow features to make the model use explicit motion information without learning the trivial solution of an identity mapping for speed and steering.
    • Add random noise and use dropout on it.
  • One alternative is to explicitly maintain a causal model.
  • Another alternative is to learn to predict the speed, as detailed in "Exploring the Limitations of Behavior Cloning for Autonomous Driving".
• Output:
  • The model decides of a "motion plan", i.e. not directly the low-level control?
  • Concretely, the network gives one prediction and one slope, for both speed and steering, leading to two parameterised lines.
• Two types of tests:
  • 1- Closed-loop (i.e. go outside and drive).
    • The number and type of safety-driver interventions.
  • 2- Open-loop (i.e., evaluating on an offline dataset).
    • The weighted mean absolute error for speed and steering.
      • As noted, this can serve as a proxy for real world performance.

  • "As discussed by [34] and [35], the correlation between offline open-loop metrics and online closed-loop performance is weak."

• About the training data:
  • As stated, they are two levers to increase the performance:
    • 1- Algorithmic innovation.
    • 2- Data.
  • For this IL approach, 30 hours of demonstrations.

  • "Re-moving a quarter of the data notably degrades performance, and models trained with less data are almost undriveable."

• Next steps:
  • I find the results already impressive. But as noted:

    • "The learned driving policies presented here need significant further work to be comparable to human driving".

  • Ideas for improvements include:
    • Add some predictive long-term planning model. At the moment, it does not have access to long-term dependencies and cannot reason about the road scene.
    • Learn not only from demonstration, but also from mistakes.
      • This reminds me the concept of ChauffeurNet about "simulate the bad rather than just imitate the good".
    • Continuous learning: Learning from corrective interventions would also be beneficial.
  • The last point goes in the direction of adding learning signals, which was already done here.
    • Imitation of human expert drivers (supervised learning).
    • Safety driver intervention data (negative reinforcement learning) and corrective action (supervised learning).
    • Geometry, dynamics, motion and future prediction (self-supervised learning).
    • Labelled semantic computer vision data (supervised learning).
    • Simulation (supervised learning).

The IL models were trained on a straight road and tested on roads with high curvature. PS-GAIL is effective only while surrounded by other vehicles, while the RAIL policy remained stably within the bounds of the road thanks to the additional rewards terms included into the learning process.. Source.

Authors: Lange, B. A., & Brannon, W. D.

• One motivation: Compare the robustness (domain adaptation) of three IL techniques:
  • 1- Generative Adversarial Imitation Learning (GAIL).
  • 2- Parameter Sharing GAIL (PS-GAIL).
  • 3- Reward Augmented Imitation Learning (RAIL).
• One take-away: This student project builds a good overview of the different IL algorithms and why these algorithms came out.
  • Imitation Learning (IL) aims at building an (efficient) policy using some expert demonstrations.
  • Behavioural Cloning (BC) is a sub-class of IL. It treats IL as a supervised learning problem: a regression model is fit to the state/action space given by the expert.

    • Issue of distribution shift: "Because data is not infinite nor likely to contain information about all possible state/action pairs in a continuous state/action space, BC can display undesirable effects when placed in these unknown or not well-known states."

    • "A cascading effect is observed as the time horizon grows and errors expand upon each other."

  • Several solutions (not exhaustive):
    • 1- DAgger: Ask the expert to say what should be done in some encountered situations. Thus iteratively enriching the demonstration dataset.
    • 2- IRL: Human driving behaviour is not modelled inside a policy, but rather capture into a reward/cost function.
      • Based on this reward function, an (optimal) policy can be derived with classic RL techniques.
      • One issue: It can be computationally expensive.
    • 3- GAIL (I still need to read more about it):

      • "It fits distributions of states and actions given by an expert dataset, and a cost function is learned via Maximum Causal Entropy IRL."

      • "When the GAIL-policy driven vehicle was placed in a multi-agent setting, in which multiple agents take over the learned policy, this algorithm produced undesirable results among the agents."

  • PS-GAIL is therefore introduced for multi-agent driving models (agents share a single policy learnt with PS-TRPO).

    • "Though PS-GAIL yielded better results in multi-agent simulations than GAIL, its results still led to undesirable driving characteristics, including unwanted trajectory deviation and off-road duration."

  • RAIL offers a fix for that: the policy-learning process is augmented with two types of reward terms:
    • Binary penalties: e.g. collision and hard braking.
    • Smoothed penalties: "applied in advance of undesirable actions with the theory that this would prevent these actions from occurring".
    • I see that technique as a way to incorporate knowledge.
• About the experiment:
  • The three policies were originally trained on the straight roadway: cars only consider the lateral distance to the edge.
  • In the "new" environment, a road curvature is introduced.
  • Findings:

    • "None of them were able to fully accommodate the turn in the road."

    • PS-GAIL is effective only while surrounded by other vehicles.
    • The smoothed reward augmentation helped RAIL, but it was too late to avoid off-road (the car is already driving too fast and does not dare a hard brake which is strongly penalized).
    • The reward function should therefore be updated (back to reward engineering 😅), for instance adding a harder reward term to prevent the car from leaving the road.

The main idea is to decompose the imitation learning (IL) process into two stages: 1- Learn to act. 2- Learn to see. Source.

mid-to-mid learning: Based on a processed bird’s-eye view map, the privileged agent predicts a sequence of waypoints to follow. This desired trajectory is eventually converted into low-level commands by two PID controllers. It is also worth noting how this privileged agent serves as an oracle that provides adaptive on-demand supervision to train the sensorimotor agent across all possible commands. Source.

Example of privileged map supplied to the first agent. And details about the lateral PID controller that produces steering commands based on a list of target waypoints. Source.

Authors: Chen, D., Zhou, B., Koltun, V. & Krähenbühl, P

• One motivation: decomposing the imitation learning (IL) process into two stages:
  • Direct IL (from expert trajectories to vision-based driving) conflates two difficult tasks:
    • 1- Learning to see.
    • 2- Learning to act.
• One term: "Cheating".
  • 1- First, train an agent that has access to privileged information:

    • "This privileged agent cheats by observing the ground-truth layout of the environment and the positions of all traffic participants."

    • Goal: The agent can focus on learning to act (it does not need to learn to see because it gets direct access to the environment’s state).
  • 2- Then, this privileged agent serves as a teacher to train a purely vision-based system (abundant supervision).
    • Goal: Learning to see.
• 1- First agent (privileged agent):
  • Input: A processed bird’s-eye view map (with ground-truth information about lanes, traffic lights, vehicles and pedestrians) together with high-level navigation command and current speed.
  • Output: A list of waypoints the vehicle should travel to.
  • Hence mid-to-mid learning approach.
  • Goal: imitate the expert trajectories.
  • Training: Behaviour cloning (BC) from a set of recorded expert driving trajectories.
    • Augmentation can be done offline, to facilitate generalization.
    • The agent is thus placed in a variety of perturbed configurations to learn how to recover
    • E.g. facing the sidewalk or placed on the opposite lane, it should find its way back onto the road.
• 2- Second agent (sensorimotor agent):
  • Input: Monocular RGB image, current speed, and a high-level navigation command.
  • Output: A list of waypoints.
  • Goal: Imitate the privileged agent.
• One idea: "White-box" agent:
  • The internal state of the privileged agent can be examined at will.
    • Based on that, one could test different high-level commands: "What would you do now if the command was [follow-lane] [go left] [go right] [go straight]".
  • This relates to conditional IL: all conditional branches are supervised during training.
• Another idea: "online learning" and "on-policy supervision":

  • "“On-policy” refers to the sensorimotor agent rolling out its own policy during training."

    • Here, the decision of the second agents are directly implemented (close-loop).
    • And an oracle is still available for the newly encountered situation (hence on-policy), which also accelerates the training.
    • This is an advantage of using a simulator: it would be difficult/impossible in the physical world.
  • Here, the second agent is first trained off-policy (on expert demonstration) to speed up the learning (offline BC), and only then go on-policy:

    • "Finally, we train the sensorimotor agent on-policy, using the privileged agent as an oracle that provides adaptive on-demand supervision in any state reached by the sensorimotor student."

    • The sensorimotor agent can thus be supervised on all its waypoints and across all commands at once.
  • It resembles the Dataset aggregation technique of DAgger:

    • "This enables automatic DAgger-like training in which supervision from the privileged agent is gathered adaptively via online rollouts of the sensorimotor agent."

• About the two benchmarks:
  • 1- Original CARLA benchmark (2017).
  • 2- NoCrash benchmark (2019).
  • Interesting idea for timeout:

    • "The time limit corresponds to the amount of time needed to drive the route at a cruising speed of 10 km/h".

• Another idea: Do not directly output low-level commands.
  • Instead, predict waypoints and speed targets.
  • And rely on two PID controllers to implement them.

    • 1- "We fit a parametrized circular arc to all waypoints using least-squares fitting and then steer towards a point on the arc."

    • 2- "A longitudinal PID controller tries to match a target velocity as closely as possible [...] We ignore negative throttle commands, and only brake if the predicted velocity is below some threshold (2 km/h)."

The main motivation is to combine the benefits of IL (to imitate some expert demonstrations) and goal-directed planning (e.g. model-based RL). Source.

φ represents the scene. It consists of the current lidar scan, previous states in the trajectory as well as the current traffic light state. Source.

From left to right: Point, Line-Segment and Region (small and wide) Final State Indicators used for planning. Source.

Comparison of features and implementations. Source.

Authors: Rhinehart, N., McAllister, R., & Levine, S.

• Main motivation: combine the benefits of imitation learning (IL) and goal-directed planning such as model-based RL (MBRL).

  • Especially to generate interpretable, expert-like plans with offline learning and no reward engineering.
  • Neither IL nor MBRL can do so.
  • In other words, it completes planning based on some imitation prior.

• One concept: "Imitative Models"

  • They are _"probabilistic predictive models able to plan interpretable expert-like trajectories to achieve new goals".
  • As for IL -> use expert demonstration:
    • It generates expert-like behaviors without reward function crafting.
    • The model is learnt "offline" also means it avoids costly online data collection (contrary to MBRL).
    • It learns dynamics desirable behaviour models (as opposed to learning the dynamics of possible behaviour done by MBRL).
  • As for MBRL -> use planning:
    • It achieves new goals (goals that were not seen during training). Therefore, it avoids the theoretical drift shortcomings (distribution shift) of vanilla behavioural cloning (BC).
    • It outputs (interpretable) plan to them at test-time, which IL cannot.
    • It does not need goal labels for training.
  • Binding IL and planning:
    • The learnt imitative model q(S|φ) can generate trajectories that resemble those that the expert might generate.
      • These manoeuvres do not have a specific goal. How to direct our agent to goals?
    • General tasks are defined by a set of goal variables G.
      • At test time, a route planner provides waypoints to the imitative planner, which computes expert-like paths for each candidate waypoint.
    • The best plan is chosen according to the planning objective (e.g. prefer routes avoiding potholes) and provided to a low-level PID-controller in order to produce steering and throttle actions.
    • In other words, the derived plan (list of set-points) should be:
      • Maximizing the similarity to the expert demonstrations (term with q)
      • Maximizing the probability of reaching some general goals (term with P(G)).
    • How to represent goals?
      • dim=0 - with points: Final-State Indicator.
      • dim=1 - with lines: Line-Segment Final-State Indicator.
      • dim=2 - with areas (regions): Final-State Region Indicator.

• How to deal with traffic lights?

  • The concept of smart waypointer is introduced.

  • "It removes far waypoints beyond 5 meters from the vehicle when a red light is observed in the measurements provided by CARLA".

  • "The planner prefers closer goals when obstructed, when the vehicle was already stopped, and when a red light was detected [...] The planner prefers farther goals when unobstructed and when green lights or no lights were observed."

• About interpretability and safety:

  • "In contrast to black-box one-step IL that predicts controls, our method produces interpretable multi-step plans accompanied by two scores. One estimates the plan’s expertness, the second estimates its probability to achieve the goal."

    • The imitative model can produce some expert probability distribution function (PDF), hence offering superior interpretability to one-step IL models.
    • It is able to score how likely a trajectory is to come from the expert.
    • The probability to achieve a goal is based on some "Goal Indicator methods" (using "Goal Likelihoods"). I must say I did not fully understand that part
  • The safety aspect relies on the fact that experts were driving safely and is formalized as a "plan reliability estimation":

    • "Besides using our model to make a best-effort attempt to reach a user-specified goal, the fact that our model produces explicit likelihoods can also be leveraged to test the reliability of a plan by evaluating whether reaching particular waypoints will result in human-like behavior or not."

    • Based on this idea, a classification is performed to recognize safe and unsafe plans, based on the planning criterion.

• About the baselines:

  • Obviously, the proposed approach is compared to the two methods it aims at combining.
  • About MBRL:
    • 1- First, a forward dynamics model is learnt using given observed expert data.
      • It does not imitate the expert preferred actions, but only models what is physically possible.
    • 2- The model then is used to plan a reachability tree through the free-space up to the waypoint while avoiding obstacles:
      • Playing with the throttle action, the search expands each state node and retains the 50 closest nodes to the target waypoint.
      • The planner finally opts for the lowest-cost path that ends near the goal.

    • "The task of evoking expert-like behavior is offloaded to the reward function, which can be difficult and time-consuming to craft properly."

  • About IL: It used Conditional terms on States, leading to CILS.
    • S for state: Instead of emitting low-level control commands (throttle, steering), it outputs set-points for some PID-controller.
    • C for conditional: To navigate at intersections, waypoints are classified into one of several directives: {Turn left, Turn right, Follow Lane, Go Straight}.
      • This is inspired by "End-to-end driving via conditional imitation learning" - (Codevilla et al. 2018) - detailed below.

Some figures:

End-to-Mid approach: 3 inputs with different levels of abstraction are used to predict the future positions on a fixed 2s-horizon of the ego vehicle and the neighbours. The ego trajectory is be implemented by an external PID controller - Therefore, not end-to-end. Source.

The past 3D-bounding boxes of the road users in the current reference are projected back in the current camera space. The past positions of ego and other vehicles are projected into some grid-map called proximity map. The image and the proximity map are concatenated to form context feature vector C. This context encoding is concatenated with the ego encoding, then fed into branches corresponding to the different high-level goals - conditional navigation goal. Source.

Illustration of the distribution shift in imitation learning. Source.

VisualBackProp highlights the image pixels which contributed the most to the final results - Traffic lights and their colours are important, together with highlights lane markings and curbs when there is a significant lateral deviation. Source.

Authors: Buhet, T., Wirbel, E., & Perrotton, X.

• Previous works:
  • "Imitation Learning for End to End Vehicle Longitudinal Control with Forward Camera" - (George, Buhet, Wirbel, Le-Gall, & Perrotton, 2018).
  • "End to End Vehicle Lateral Control Using a Single Fisheye Camera" (Toromanoff, M., Wirbel, E., Wilhelm, F., Vejarano, C., Perrotton, X., & Moutarde, F. 2018).
• One term: "End-To-Middle".
  • It is opposed to "End-To-End", i.e. it does not output "end" control signals such as throttle or steering but rather some desired trajectory, i.e. a mid-level representation.
    • Each trajectory is described by two polynomial functions (one for x, the other for y), therefore the network has to predict a vector (x0, ..., x4, y0, ..., y4) for each vehicle.
    • The desired ego-trajectory is then implemented by an external controller (PID). Therefore, not end-to-end.
  • Advantages of end-to-mid: interpretability for the control part + less to be learnt by the net.
  • This approach is also an instance of "Direct perception":

    • "Instead of commands, the network predicts hand-picked parameters relevant to the driving (distance to the lines, to other vehicles), which are then fed to an independent controller".

  • Small digression: if the raw perception measurements were first processed to form a mid-level input representation, the approach would be said mid-to-mid. An example is ChauffeurNet, detailed on this page as well.
• About Ground truth:
  • The expert demonstrations do not come from human recordings but rather from CARLA autopilot.
  • 15 hours of driving in Town01 were collected.
  • As for human demonstrations, no annotation is needed.
• One term: "Conditional navigation goal".
  • Together with the RGB images and the past positions, the network takes as input a navigation command to describe the desired behaviour of the ego vehicle at intersections.
  • Hence, the future trajectory of the ego vehicle is conditioned by a navigation command.
    • If the ego-car is approaching an intersection, the goal can be left, right or cross, else the goal is to keep lane.
    • That means lane-change is not an option.

  • "The last layers of the network are split into branches which are masked with the current navigation command, thus allowing the network to learn specific behaviours for each goal".

• Three ingredients to improve vanilla end-to-end imitation learning (IL):
  • 1- Mix of high and low-level input (i.e. hybrid input):
    • Both raw signal (images) and partial environment abstraction (navigation commands) are used.
  • 2- Auxiliary tasks:
    • One head of the network predicts the future trajectories of the surrounding vehicles.
      • It differs from the primary task which should decide the 2s-ahead trajectory for the ego car.
      • Nevertheless, this secondary task helps: "Adding the neighbours prediction makes the ego prediction more compliant to traffic rules."
    • This refers to the concept of "Privileged learning":

      • "The network is partly trained with an auxiliary task on a ground truth which is useful to driving, and on the rest is only trained for IL".

  • 3- Label augmentation:
    • The main challenge of IL is the difference between train and online test distributions. This is due to the difference between
      • Open-loop control: decisions are not implemented.
      • Close-loop control: decisions are implemented, and the vehicle can end in a state absent from the train distribution, potentially causing "error accumulation".
    • Data augmentation is used to reduce the gap between train and test distributions.
      • Classical randomization is combined with label augmentation: data similar to failure cases is generated a posteriori.
    • Three findings:

    • "There is a significant gap in performance when introducing the augmentation."

    • "The effect is much more noticeable on complex navigation tasks." (Errors accumulate quicker).

    • "Online test is the real significant indicator for IL when it is used for active control." (The common offline evaluation metrics may not be correlated to the online performance).

• Baselines:
  • Conditional Imitation learning (CIL): "End-to-end driving via conditional imitation learning" video - (Codevilla, F., Müller, M., López, A., Koltun, V., & Dosovitskiy, A. 2017).
    • CIL produces instantaneous commands.
  • Conditional Affordance Learning (CAL): "Conditional affordance learning for driving in urban environments" video - (Sauer, A., Savinov, N. & Geiger, A. 2018).
    • CAL produces "affordances" which are then given to a controller.
• One word about the choice of the simulator.
  • A possible alternative to CARLA could be DeepDrive or the LGSVL simulator developed by the Advanced Platform Lab at the LG Electronics America R&D Centre. This looks promising.

One figure:

The trust or uncertainty in one decision can be measured based on the probability mass function around its mode. Source.

The measures of uncertainty based on mutual information can be used to issue warnings to the driver and perform safety / emergency manoeuvres. Source.

As noted by the authors: while the variance can be useful in collision avoidance, the wide variance of HMC causes a larger proportion of trajectories to fall outside of the safety boundary when a new weather is applied. Source.

Authors: Michelmore, R., Wicker, M., Laurenti, L., Cardelli, L., Gal, Y., & Kwiatkowska, M

• One related work:
  • NVIDIA’s PilotNet [DAVE-2] where expert demonstrations are used together with supervised learning to map from images (front camera) to steering command.
  • Here, human demonstrations are collected in the CARLA simulator.
• One idea: use distribution in weights.
  • The difference with PilotNet is that the neural network applies the "Bayesian" paradigm, i.e. each weight is described by a distribution (not just a single value).
  • The authors illustrate the benefits of that paradigm, imagining an obstacle in the middle of the road.
    • The Bayesian controller may be uncertain on the steering angle to apply (e.g. a 2-tail or M-shape distribution).
    • A first option is to sample angles, which turns the car either right or left, with equal probability.
    • Another option would be to simply select the mean value of the distribution, which aims straight at the obstacle.
    • The motivation of this work is based on that example: "derive some precise quantitative measures of the BNN uncertainty to facilitate the detection of such ambiguous situation".
• One definition: "real-time decision confidence".
  • This is the probability that the BNN controller is certain of its decision at the current time.
  • The notion of trust can therefore be introduced: the idea it to compute the probability mass in a ε−ball around the decision π(observation) and classify it as certain if the resulting probability is greater than a threshold.
    • It reminds me the concept of trust-region optimisation in RL.
    • In extreme cases, all actions are equally distributed, π(observation) has a very high variance, the agent does not know what to do (no trust) and will randomly sample an action.
• How to get these estimates? Three Bayesian inference methods are compared:
  • Monte Carlo dropout (MCD).
  • Mean-field variational inference (VI).
  • Hamiltonian Monte Carlo (HMC).
• What to do with this information?

  • "This measure of uncertainty can be employed together with commonly employed measures of uncertainty, such as mutual information, to quantify in real time the degree that the model is confident in its predictions and can offer a notion of trust in its predictions."

    • I did not know about "mutual information" and liked the explanation of Wikipedia about the link of MI to entropy and KL-div.
      • I am a little bit confused: in what I read, the MI is function of two random variables. What are they here? The authors rather speak about the predictive distribution exhibited by the predictive distribution.
  • Depending on the uncertainty level, several actions are taken:
    • mutual information warnings slow down the vehicle.
    • standard warnings slow down the vehicle and alert the operator of potential hazard.
    • severe warnings cause the car to safely brake and ask the operator to take control back.
• Another definition: "probabilistic safety", i.e. the probability that a BNN controller will keep the car "safe".
  • Nice, but what is "safe"?
  • It all relies on the assumption that expert demonstrations were all "safe", and measures the how much of the trajectory belongs to this "safe set".
  • I must admit I did not fully understand the measure on "safety" for some continuous trajectory and discrete demonstration set:
    • A car can drive with a large lateral offset from the demonstration on a wide road while being "safe", while a thin lateral shift in a narrow street can lead to an "unsafe" situation.
    • Not to mention that the scenario (e.g. configuration of obstacles) has probably changed in-between.
    • This leads to the following point with an interesting application for scenario coverage.
• One idea: apply changes in scenery and weather conditions to evaluate model robustness.
  • To check the generalization ability of a model, the safety analysis is re-run (offline) with other weather conditions.
  • As noted in conclusion, this offline safety probability can be used as a guide for active learning in order to increase data coverage and scenario representation in training data.

One figure:

Conditional Imitation Learning is extended with a ResNet architecture and Speed prediction (CILRS). Source.

Authors: Codevilla, F., Santana, E., Antonio, M. L., & Gaidon, A.

• One term: “CILRS” = Conditional Imitation Learning extended with a ResNet architecture and Speed prediction.
• One Q&A: How to include in E2E learning information about the destination, i.e. to disambiguate imitation around multiple types of intersections?
  • Add a high-level navigational command (e.g. take the next right, left, or stay in lane) to the tuple <observation, expert action> when building the dataset.
• One idea: learn to predict the ego speed (mediated perception) to address the inertia problem stemming from causal confusion (biased correlation between low speed and no acceleration - when the ego vehicle is stopped, e.g. at a red traffic light, the probability it stays static is indeed overwhelming in the training data).
  • A good video about Causal Confusion in Imitation Learning showing that "access to more information leads to worse generalisation under distribution shift".
• Another idea: The off-policy (expert) driving demonstration is not produced by a human, but rather generated from an omniscient "AI" agent.
• One quote:

"The more common the vehicle model and color, the better the trained agent reacts to it. This raises ethical challenges in automated driving".

Some figures:

Examples of affordances, i.e. attributes of the environment which limit the space of allowed actions. A1, A2 and A3 are predefined observation areas. Source.

The presented direct perception method predicts a low-dimensional intermediate representation of the environment - affordance - which is then used in a conventional control algorithm. The affordance is conditioned for goal-directed navigation, i.e. before each intersection, it receives an instruction such as go straight, turn left or turn right. Source.

The feature maps produced by a CNN feature extractor are stored in a memory and consumed by task-specific layers (one affordance has one task block). Every task block has its own specific temporal receptive field - it decides how much of the memory it needs. This figure also illustrates how the navigation command is used as switch between trained submodules. Source.

Authors: Sauer, A., Savinov, N., & Geiger, A.

• One term: "Direct perception" (DP):
  • The goal of DP methods is to predict a low-dimensional intermediate representation of the environment which is then used in a conventional control algorithm to manoeuvre the vehicle.
  • With this regard, DP could also be said end-to-mid. The mapping to learn is less complex than end-to-end (from raw input to controls).
  • DP is meant to combine the advantages of two other commonly-used approaches: modular pipelines MP and end-to-end methods such as imitation learning IL or model-free RL.
  • Ground truth affordances are collected using CARLA. Several augmentations are performed.
• Related work on affordance learning and direct perception.
  • Deepdriving: Learning affordance for direct perception in autonomous driving by (Chen, Seff, Kornhauser, & Xiao, 2015).
  • Deepdriving works on highway.
  • Here, the idea is extended to urban scenarios (with traffic signs, traffic lights, junctions) considering a sequence of images (not just one camera frame) for temporal information.
• One term: "Conditional Affordance Learning" (CAL):
  • "Conditional": The actions of the agent are conditioned on a high-level command given by the navigation system (the planner) prior to intersections. It describes the manoeuvre to be performed, e.g., go straight, turn left, turn right.
  • "Affordance": Affordances are one example of DP representation. They are attributes of the environment which limit the space of allowed actions. Only 6 affordances are used for CARLA urban driving:
    • Distance to vehicle (continuous).
    • Relative angle (continuous and conditional).
    • Distance to centre-line (continuous and conditional).
    • Speed Sign (discrete).
    • Red Traffic Light (discrete - binary).
    • Hazard (discrete - binary).
      • The Class Weighted Cross Entropy is the loss used for discrete affordances to put more weights on rare but important occurrences (hazard occurs rarely compared to traffic light red).
  • "Learning": A single neural network trained with multi-task learning (MTL) predicts all affordances in a single forward pass (~50ms). It only takes a single front-facing camera view as input.
• About the controllers: The path-velocity decomposition is applied. Hence two controllers are used in parallel:
  • 1- throttle and brake
    • Based on the predicted affordances, a state is "rule-based" assigned among: cruising, following, over limit, red light, and hazard stop (all are mutually exclusive).
    • Based on this state, the longitudinal control signals are derived, using PID or threshold-predefined values.
    • It can handle traffic lights, speed signs and smooth car-following.
    • Note: The Supplementary Material provides details insights on controller tuning (especially PID) for CARLA.
  • 2- steering is controlled by a Stanley Controller, based on two conditional affordances: distance to centreline and relative angle.
• One idea: I am often wondering what timeout I should set when testing a scenario with CARLA. The author computes this time based on the length of the pre-defined path (which is actually easily accessible):

  • "The time limit equals the time needed to reach the goal when driving along the optimal path at 10 km/h"

• Another idea: Attention Analysis.
  • For better understanding on how affordances are constructed, the attention of the CNN using gradient-weighted class activation maps (Grad-CAMs).
  • This "visual explanation" reminds me another technique used in end-to-end approaches, VisualBackProp, that highlights the image pixels which contributed the most to the final results.
• Baselines and results:
  • Compared to CARLA-based Modular Pipeline (MP), Conditional Imitation Learning (CIL) and Reinforcement Learning (RL), CAL particularly excels in generalizing to the new town.
• Where to provide the high-level navigation conditions?
  • The authors find that "conditioning in the network has several advantages over conditioning in the controller".
  • In addition, in the net, it is preferable to use the navigation command as switch between submodules rather than an input:

    • "We observed that training specialized submodules for each directional command leads to better performance compared to using the directional command as an additional input to the task networks".

Some figures:

One particular latent variable ^y is explicitly supervised to predict steering control. Another interesting idea: augmentation is based on domain knowledge - if a method used to the middle-view is given some left-view image, it should predict some correction to the right. Source.

For each new image, empirical uncertainty estimates are computed by sampling from the variables of the latent space. These estimates lead to the D statistic that indicates whether an observed image is well captured by our trained model, i.e. novelty detection. Source.

In a subsequent work, the VAE is conditioned onto the road topology. It serves multiple purposes such as localization and end-to-end navigation. The routed or unrouted map given as additional input goes toward the mid-to-end approach where processing is performed and/or external knowledge is embedded. Source. See this video temporal for evolution of the predictions.

Authors: Amini, A., Schwarting, W., Rosman, G., Araki, B., Karaman, S., & Rus, D.

• One issue raised about vanilla E2E:
  • The lack a measure of associated confidence in the prediction.
  • The lack of interpretation of the learned features.
  • Having said that, the authors present an approach to both understand and estimate the confidence of the output.
  • The idea is to use a Variational Autoencoder (VAE), taking benefit of its intermediate latent representation which is learnt in an unsupervised way and provides uncertainty estimates for every variable in the latent space via their parameters.
• One idea for the VAE: one particular latent variable is explicitly supervised to predict steering control.
  • The loss function of the VAE has therefore 3 parts:
    • A reconstruction loss: L1-norm between the input image and the output image.
    • A latent loss: KL-divergence between the latent variables and a unit Gaussian, providing regularization for the latent space.
    • A supervised latent loss: MSE between the predicted and actual curvature of the vehicle’s path.
• One contribution: "Detection of novel events" (which have not been sufficiently trained for).
  • To check if an observed image is well captured by the trained model, the idea is to propagate the VAE’s latent uncertainty through the decoder and compare the result with the original input. This is done by sampling (empirical uncertainty estimates).
  • The resulting pixel-wise expectation and variance are used to compute a sort of loss metric D(x, ˆx) whose distribution for the training-set is known (approximated with a histogram).
  • The image x is classified as novel if this statistic is outside of the 95th percentile of the training distribution and the prediction can finally be "untrusted to produce reliable outputs".

  • "Our work presents an indicator to detect novel images that were not contained in the training distribution by weighting the reconstructed image by the latent uncertainty propagated through the network. High loss indicates that the model has not been trained on that type of image and thus reflects lower confidence in the network’s ability to generalize to that scenario."

• A second contribution: "Automated debiasing against learned biases".
  • As for the novelty detection, it takes advantage of the latent space distribution and the possibility of sampling from the most representative regions of this space.
  • Briefly said, the idea it to increase the proportion of rarer datapoints by dropping over-represented regions of the latent space to accelerate the training (sampling efficiency).
  • This debiasing is not manually specified beforehand but based on learned latent variables.
• One reason to use single frame prediction (as opposed to RNN):

  • ""Note that only a single image is used as input at every time instant. This follows from original observations where models that were trained end-to-end with a temporal information (CNN+LSTM) are unable to decouple the underlying spatial information from the temporal control aspect. While these models perform well on test datasets, they face control feedback issues when placed on a physical vehicle and consistently drift off the road.""

• One idea about augmentation (also met in the Behavioral Cloning Project of the Udacity Self-Driving Car Engineer Nanodegree):

  • "To inject domain knowledge into our network we augmented the dataset with images collected from cameras placed approximately 2 feet to the left and right of the main centre camera. We correspondingly changed the supervised control value to teach the model how to recover from off-centre positions."

• One note about the output:

  • "We refer to steering command interchangeably as the road curvature: the actual steering angle requires reasoning about road slip and control plant parameters that change between vehicles."

• Previous and further works:
  • "Spatial Uncertainty Sampling for End-to-End control" - (Amini, Soleimany, Karaman, & Rus, 2018)
  • "Variational End-to-End Navigation and Localization" - (Amini, Rosman, Karaman, & Rus, 2019)
    • One idea: incorporate some coarse-grained roadmaps with raw perceptual data.
      • Either unrouted (just containing the drivable roads). Output = continuous probability distribution over steering control.
      • Or routed (target road highlighted). Output = deterministic steering control to navigate.
    • How to evaluate the continuous probability distribution over steering control given the human "scalar" demonstration?

      • "For a range of z-scores over the steering control distribution we compute the number of samples within the test set where the true (human) control output was within the predicted range."

    • About the training dataset: 25 km of urban driving data.

Two figures:

Different layers composing the mid-level representation. Source.

Training architecture around ChauffeurNet with the different loss terms, that can be grouped into environment and imitation losses. Source.

Authors: Bansal, M., Krizhevsky, A., & Ogale, A.

• One term: "mid-level representation"
  • The decision-making task (between perception and control) is packed into one single "learnable" module.
    • Input: the representation divided into several image-like layers:
      • Map features such as lanes, stop signs, cross-walks...; Traffic lights; Speed Limit; Intended route; Current agent box; Dynamic objects; Past agent poses.
      • Such a representation is generic, i.e. independent of the number of dynamic objects and independent of the road geometry/topology.
      • I discuss some equivalent representations seen at IV19.
    • Output: intended route, i.e. the future poses recurrently predicted by the introduced ChauffeurNet model.
  • This architecture lays between E2E (from pixels directly to control) and fully decomposed modular pipelines (decomposing planning in multiple modules).
  • Two notable advantages over E2E:
    • It alleviates the burdens of learning perception and control:
      • The desired trajectory is passed to a controls optimizer that takes care of creating the low-level control signals.
      • Not to mention that different types of vehicles may possibly utilize different control outputs to achieve the same driving trajectory.
    • Perturbations and input data from simulation are easier to generate.
• One key finding: "pure imitation learning is not sufficient", despite the 60 days of continual driving (30 million examples).
  • One quote about the "famous" distribution shift (deviation from the training distribution) in imitation learning:

  "The key challenge is that we need to run the system closed-loop, where errors accumulate and induce a shift from the training distribution."

  • The training data does not have any real collisions. How can the agent efficiently learn to avoid them if it has never been exposed during training?
  • One solution consists in exposing the model to non-expert behaviours, such as collisions and off-road driving, and in adding extra loss functions.
    • Going beyond vanilla cloning.
      • Trajectory perturbation: Expose the learner to synthesized data in the form of perturbations to the expert’s driving (e.g. jitter the midpoint pose and heading)
        • One idea for future works is to use more complex augmentations, e.g. with RL, especially for highly interactive scenarios.
      • Past dropout: to prevent using the history to cheat by just extrapolating from the past rather than finding the underlying causes of the behaviour.
      • Hence the concept of tweaking the training data in order to “simulate the bad rather than just imitate the good”.
    • Going beyond the vanilla imitation loss.
      • Extend imitation losses.
      • Add environment losses to discourage undesirable behaviour, e.g. measuring the overlap of predicted agent positions with the non-road regions.
      • Use imitation dropout, i.e. sometimes favour the environment loss over the imitation loss.

Some figures:

The state consists in 51 features divided into 3 groups: The core features include hand-picked features such as Speed, Curvature and Lane Offset. The LIDAR-like beams capture the surrounding objects in a fixed-size representation independent of the number of vehicles. Finally, 3 binary indicator features identify when the ego vehicle encounters undesirable states - collision, drives off road, and travels in reverse. Source.

As for common adversarial approaches, the objective function in GAIL includes some sigmoid cross entropy terms. The objective is to fit ψ for the discriminator. But this objective function is non-differentiable with respect to θ. One solution is to optimize πθ separately using RL. But what for reward function? In order to drive πθ into regions of the state-action space similar to those explored by the expert πE, a surrogate reward ˜r is generated from D_ψ based on samples and TRPO is used to perform a policy update of πθ. Source.

Authors: Kuefler, A., Morton, J., Wheeler, T., & Kochenderfer, M.

• One term: the problem of "cascading errors" in behavioural cloning (BC).
  • BC, which treats IL as a supervised learning problem, tries to fit a model to a fixed dataset of expert state-action pairs. In other words, BC solves a regression problem in which the policy parameterization is obtained by maximizing the likelihood of the actions taken in the training data.
  • But inaccuracies can lead the stochastic policy to states that are underrepresented in the training data (e.g., an ego-vehicle edging towards the side of the road). And datasets rarely contain information about how human drivers behave in such situations.
  • The policy network is then forced to generalize, and this can lead to yet poorer predictions, and ultimately to invalid or unseen situations (e.g., off-road driving).
  • "Cascading Errors" refers to this problem where small inaccuracies compound during simulation and the agent cannot recover from them.
    • This issue is inherent to sequential decision making.
  • As found by the results:

    • "The root-weighted square error results show that the feedforward BC model has the best short-horizon performance, but then begins to accumulate error for longer time horizons."

    • "Only GAIL (and of course IDM+MOBIL) are able to stay on the road for extended stretches."

• One idea: RL provides robustness against "cascading errors".
  • RL maximizes the global, expected return on a trajectory, rather than local instructions for each observation. Hence more appropriate for sequential decision making.
  • Also, the reward function r(s_t, a_t) is defined for all state-action pairs, allowing an agent to receive a learning signal even from unusual states. And these signals can establish preferences between mildly undesirable behaviour (e.g., hard braking) and extremely undesirable behaviour (e.g., collisions).
    • In contrast, BC only receives a learning signal for those states represented in a labelled, finite dataset.
    • Because handcrafting an accurate RL reward function is often difficult, IRL seems promising. In addition, the imitation (via the recovered reward function) extends to unseen states: e.g. a vehicle that is perturbed toward the lane boundaries should know to return toward the lane centre.
• Another idea: use GAIL instead of IRL:

  • "IRL approaches are typically computationally expensive in their recovery of an expert cost function. Instead, recent work has attempted to imitate expert behaviour through direct policy optimization, without first learning a cost function."

  • "Generative Adversarial Imitation Learning" (GAIL) implements this idea:

    • "Expert behaviour can be imitated by training a policy to produce actions that a binary classifier mistakes for those of an expert."

    • "GAIL trains a policy to perform expert-like behaviour by rewarding it for “deceiving” a classifier trained to discriminate between policy and expert state-action pairs."

  • One contribution is to extend GAIL to the optimization of recurrent neural networks (GRU in this case).
• One concept: "Trust Region Policy Optimization".
  • Policy-gradient RL optimization with "Trust Region" is used to optimize the agent's policy πθ, addressing the issue of training instability of vanilla policy-gradient methods.

    • "TRPO updates policy parameters through a constrained optimization procedure that enforces that a policy cannot change too much in a single update, and hence limits the damage that can be caused by noisy gradient estimates."

  • But what reward function to apply? Again, we do not want to do IRL.
  • Some "surrogate" reward function is empirically derived from the discriminator. Although it may be quite different from the true reward function optimized by expert, it can be used to drive πθ into regions of the state-action space similar to those explored by πE.
• One finding: Should previous actions be included in the state s?

  • "The previous action taken by the ego vehicle is not included in the set of features provided to the policies. We found that policies can develop an over-reliance on previous actions at the expense of relying on the other features contained in their input."

  • But on the other hand, the authors find:

  • "The GAIL GRU policy takes similar actions to humans, but oscillates between actions more than humans. For instance, rather than outputting a turn-rate of zero on straight road stretches, it will alternate between outputting small positive and negative turn-rates".

  • "An engineered reward function could also be used to penalize the oscillations in acceleration and turn-rate produced by the GAIL GRU".

• Some interesting interpretations about the IDM and MOBIL driver models (resp. longitudinal and lateral control).
  • These commonly-used rule-based parametric models serve here as baselines:

  • "The Intelligent Driver Model (IDM) extended this work by capturing asymmetries between acceleration and deceleration, preferred free road and bumper-to-bumper headways, and realistic braking behaviour."

  • "MOBIL maintains a utility function and 'politeness parameter' to capture intelligent driver behaviour in both acceleration and turning."

Left: The contribution of each feature in the linear reward model differs between the Maximum Entropy (ME) and the Feature Matching (FM) algorithms. The FM algorithm is inconsistent across levels and has a higher intercept to parameter weight ratio compared with the estimated weights using the ME. Besides, why does it penalize all lateral distances and all speeds in these overtaking scenarios? Right: good idea how to visualize reward function for state of dimension 5. Source.

Authors: Alsaleh, R., & Sayed, T.

• In short: A simple but good illustration of IRL concepts using Maximum Entropy (ME) and Feature Matching (FM) algorithms.
  • It reminds me some experiments I talk about in this video: "From RL to Inverse Reinforcement Learning: Intuitions, Concepts + Applications to Autonomous Driving".
• Motivations, here:
  • 1- Work in non-motorized shared spaces, in this case a cyclist-pedestrian zone.
    • It means high degrees of freedom in motions for all participants.
    • And offers complex road-user interactions (behaviours different than on conventional streets).
  • 2- Model the behaviour of cyclists in this share space using agent-based modelling.
    • agent-based as opposed to physics-based prediction models such as social force model (SFM) or cellular automata (CA).
    • The agent is trying to maximize an unknown reward function.
    • The recovery of that reward function is the core of the paper.
• First, 2 interaction types are considered:
  • The cyclist following the pedestrian.
  • The cyclist overtaking the pedestrian.
  • This distinction avoids the search for a 1-size-fits-all model.
• About the MDP:
  • The cyclist is the agent.
  • state (absolute for the cyclist or relative compared to the pedestrian):
    • longitudinal distance
    • lateral distance
    • angle difference
    • speed difference
    • cyclist speed
  • state discretization:

    • "Discretized for each interaction type by dividing each state feature into 6 levels based on equal frequency observation in each level."

    • This non-constant bin-width partially addresses the imbalanced dataset.
    • 6^5 = 7776 states.
  • action:
    • acceleration
    • yaw rate
  • action discretization:

    • "Dividing the acceleration into five levels based on equal frequency observation in each level."

    • 5^2 = 25 actions.
  • discount factor:

    • "A discount factor of 0.975 is used assuming 10% effect of the reward at a state 3 sec later (90 time steps) from the current state."

• About the dataset.
  • Videos of two streets in Vancouver, for a total of 39 hours.
  • 228 cyclist and 276 pedestrian trajectories are extracted.
• IRL.
  • The two methods assume that the reward is a linear combination of features. Here features are state components.
  • 1- Feature Matching (FM).
    • It matches the feature counts of the expert trajectories.
    • The authors do not details the max-margin part of the algorithm.
  • 2- Maximum Entropy (ME).
    • It estimates the reward function parameters by maximizing the likelihood of the expert demonstrations under the maximum entropy distribution.
    • Being probabilistic, it can account for non-optimal observed behaviours.
• The recovered reward model can be used for prediction - How to measure the similarity between two trajectories?
  • 1- Mean Absolute Error (MAE).
    • It compares elements of same indices in the two sequences.
  • 2- Hausdorff Distance.

    • "It computes the largest distance between the simulated and the true trajectories while ignoring the time step alignment".

• Current limitations:
  • 1-to-1 interactions, i.e. a single pedestrian/cyclist pair.
  • Low-density scenarios.

  • "[in future works] neighbor condition (i.e. other pedestrians and cyclists) and shared space density can be explicitly considered in the model."

Instead of the costly RL optimisation step at each iteration of the vanilla IRL, the idea is to randomly sample a massive of policies in advance and then to pick one of them as the optimal policy. In case the sampled policy set does not contain the optimal policy, exploration of policy is introduced as well for supplement. Source.

The approximation used in Kuderer et al. (2015) is applied here to compute the second term of gradient about the expected feature values. Source.

Authors: Xin, L., Li, S. E., Wang, P., Cao, W., Nie, B., Chan, C., & Cheng, B.

• Reminder: Goal of IRL = Recover the reward function of an expert from demonstrations (here trajectories).
• Motivations, here:
  • 1- Improve the efficiency of "weights updating" in the iterative routine of IRL.
    • More precisely: generating optimal policy using model-free RL suffers from low sampling efficiency and should therefore be avoided.
    • Hence the term "accelerated" IRL.
  • 2- Embed human knowledge where restricting the search space (policy space).
• One idea: "Pre-designed policy subspace".

  • "An intuitive idea is to randomly sample a massive of policies in advance and then to pick one of them as the optimal policy instead of finding it via RL optimisation."

• How to construct the policies sub-space?
  • Human knowledge about vehicle controllers is used.
  • Parametrized linear controllers are implemented:
    • acc = K1∆d + K2∆v + K3*∆a, where ∆ are relative to the leading vehicle.
    • By sampling tuples of <K1, K2, K3> coefficients, 1 million (candidates) policies are generated to form the sub-space.
• Section about Max-Entropy IRL (btw. very well explained, as for the section introducing IRL):

  • "Ziebart et al. (2008) employed the principle of maximum entropy to resolve ambiguities in choosing trajectory distributions. This principle maximizes the uncertainty and leads to the distribution over behaviors constrained to matching feature expectations, while being no more committed to any particular trajectory than this constraint requires".

  • "Maximizing the entropy of the distribution over trajectories subject to the feature constraints from expert’s trajectories implies to maximize the likelihood under the maximum entropy (exponential family) distributions. The problem is convex for MDPs and the optima can be obtained using gradient-based optimization methods".

  • "The gradient [of the Lagrangian] is the difference between empirical feature expectations and the learners expected feature expectations."

• How to compute the second term of this gradient?
  • It implies integrating over all possible trajectories, which is infeasible.
  • As Kuderer et al. (2015), one can compute the feature values of the most likely trajectory as an approximation of the feature expectation.

  • "With this approximation, only the optimal trajectory associated to the optimal policy is needed, in contrast to regarding the generated trajectories as a probability distribution."

• About the features.
  • As noted in my experiments about IRL, they serve two purposes (in feature-matching-based IRL methods):
    • 1- In the reward function: they should represent "things we want" and "things we do not want".
    • 2- In the feature-match: to compare two policies based on their sampled trajectories, they should capture relevant properties of driving behaviours.
  • Three features for this longitudinal acceleration task:
    • front-veh time headway.
    • long. acc.
    • deviation to speed limit.
• Who was the expert?
  • Expert followed a modified linear car-following (MLCF) model.
• Results.
  • Iterations are stopped after 11 loops.
  • It would have been interesting for comparison to test a "classic" IRL method where RL optimizations are applied.

Both behavioural planner and trajectory optimizer share the same cost function, whose weigth parameters are learnt from demonstration. Source.

Authors: Sadat, A., Ren, M., Pokrovsky, A., Lin, Y., Yumer, E., & Urtasun, R.

• Main motivation:
  • Design a decision module where both the behavioural planner and the trajectory optimizer share the same objective (i.e. cost function).
  • Therefore "joint".

  • "[In approaches not-joint approaches] the final trajectory outputted by the trajectory planner might differ significantly from the one generated by the behavior planner, as they do not share the same objective".

• Requirements:
  • 1- Avoid time-consuming, error-prone, and iterative hand-tuning of cost parameters.
    • E.g. Learning-based approaches (BC).
  • 2- Offer interpretability about the costs jointly imposed on these modules.
    • E.g. Traditional modular 2-stage approaches.
• About the structure:
  • The driving scene is described in W (desired route, ego-state, map, and detected objects). Probably W for "World"?
  • The behavioural planner (BP) decides two things based on W:
    • 1- A high-level behaviour b.
      • The path to converge to, based on one chosen manoeuvre: keep-lane, left-lane-change, or right-lane-change.
      • The left and right lane boundaries.
      • The obstacle side assignment: whether an obstacle should stay in the front, back, left, or right to the ego-car.
    • 2- A coarse-level trajectory τ.
    • The loss has also a regularization term.
    • This decision is "simply" the argmin of the shared cost-function, obtained by sampling + selecting the best.
  • The "trajectory optimizer" refines τ based on the constraints imposed by b.
    • E.g. an overlap cost will be incurred if the side assignment of b is violated.
  • A cost function parametrized by w assesses the quality of the selected <b, τ> pair:
    • cost = w^T . sub-costs-vec(τ, b, W).
    • Sub-costs relate to safety, comfort, feasibility, mission completion, and traffic rules.
• Why "learnable"?
  • Because the weight vector w that captures the importance of each sub-cost is learnt based on human demonstrations.

    • "Our planner can be trained jointly end-to-end without requiring manual tuning of the costs functions".

  • They are two losses for that objective:
    • 1- Imitation loss (with MSE).
      • It applies on the <b, τ> produced by the BP.
    • 2- Max-margin loss to penalize trajectories that have small cost and are different from the human driving trajectory.
      • It applies on the <τ> produced by the trajectory optimizer.

      • "This encourages the human driving trajectory to have smaller cost than other trajectories".

      • It reminds me the max-margin method in IRL where the weights of the reward function should make the expert demonstration better than any other policy candidate.

Author: Wang, P., Liu, D., Chen, J., & Chan, C.-Y.

In Adversarial IRL (AIRL), the discriminator tries to distinguish learnt actions from demonstrated expert actions. Action-masking is applied, removing some action combinations that are not preferable, in order to reduce the unnecessary exploration. Finally, the reward function of the discriminator is extended with some manually-designed semantic reward to help the agent successfully complete the lane change and not to collide with other objects. Source.

• One related concept (detailed further on this page): Generative Adversarial Imitation Learning (GAIL).
  • An imitation learning method where the goal is to learn a policy against a discriminator that tries to distinguish learnt actions from expert actions.
• Another concept used here: Guided Cost Learning (GCL).
  • A Max-Entropy IRL method that makes use of importance sampling (IS) to approximate the partition function (the term in the gradient of the log-likelihood function that is hard to compute since it involves an integral of over all possible trajectories).
• One concept introduced: Adversarial Inverse Reinforcement Learning (AIRL).
  • It combines GAIL with GCL formulation.

    • "It uses a special form of the discriminator different from that used in GAIL, and recovers a cost function and a policy simultaneously as that in GCL but in an adversarial way."

  • Another difference is the use of a model-free RL method to compute the new optimal policy, instead of model-based guided policy search (GPS) used in GCL:

    • "As the dynamic driving environment is too complicated to learn for the driving task, we instead use a model-free policy optimization method."

  • One motivation of AIRL is therefore to cope with changes in the dynamics of environment and make the learnt policy more robust to system noises.
• One idea: Augment the learned reward with some "semantic reward" term to improve learning efficiency.
  • The motivation is to manually embed some domain knowledge, in the generator reward function.

  • "This should provide the agent some informative guidance and assist it to learn fast."

• About the task:

  • "The task of our focus includes a longitudinal decision – the selection of a target gap - and a lateral decision – whether to commit the lane change right now."

  • It is a rather "high-level" decision:
    • A low-level controller, consisting of a PID for lateral control and sliding-mode for longitudinal control, is the use to execute the decision.
  • The authors use some action-masking technics where only valid action pairs are allowed to reduce the agent’s unnecessary exploration.

Author: Hjaltason, B.

About the features: The φ are distances read from the origin of a vision field and are represented by red dotted lines. They take value in [0, 1], where φi = 1 means the dotted line does not hit any object and φi = 0 means it hits an object at origin. In this case, two objects are inside the front vision field. Hence φ1 = 0.4 and φ2 = 0.6. Source.

Example of max-margin IRL. Source.

• A good example of max-margin IRL:

  • "There are two classes: The expert behaviour from data gets a label of 1, and the "learnt" behaviours a label of -1. The framework performs a max-margin optimization step to maximise the difference between both classes. The result is an orthogonal vector wi from the max margin hyperplane, orthogonal to the estimated expert feature vector µ(πE)".

  • From this new R=w*f, an optimal policy is derived using DDPG.
  • Rollouts are performed to get an estimated feature vector that is added to the set of "learnt" behaviours.
  • The process is repeated until convergence (when the estimated values w*µ(π) are close enough).
• Note about the reward function:
  • Here, r(s, a, s') is also function of the action and the next state.
  • Here a post about different forms of reward functions.

Authors: Arora, S., & Doshi, P.

Trying to generalize and classify IRL methods. Source.

I learnt about state visitation frequency: ψ(π)(s) and the feature count expectation: µ(π)(φ). Source.

• This large review does not focus on AD applications, but it provides a good picture of IRL and can give ideas. Here are my take-aways.
• Definition:

  • "Inverse reinforcement learning (IRL) is the problem of modeling the preferences of another agent using its observed behavior [hence class of IL], thereby avoiding a manual specification of its reward function."

• Potential AD applications of IRL:
  • Decision-making: If I find your underlying reward function, and I consider you as an expert, I can imitate you.
  • Prediction: If I find your underlying reward function, I can imagine what you are going to do
• I start rethinking Imitation Learning. The goal of IL is to derive a policy based on some (expert) demonstrations.
  • Two branches emerge, depending on what structure is used to model the expert behaviour. Where is that model captured?
    • 1- In a policy.
      • This is a "direct approach". It includes BC and its variants.
      • The task is to learn that state -> action mapping.
    • 2- In a reward function.
      • Core assumption: Each driver has an internal reward function and acts optimally w.r.t. it.
      • The main task it to learn that reward function (IRL), which captures the expert's preferences.
      • The second step consists in deriving the optimal policy for this derived reward function.

        As Ng and Russell put it: "The reward function, rather than the policy, is the most succinct, robust, and transferable definition of the task"

  • What happens if some states are missing in the demonstration?
    • 1- Direct methods will not know what to do. And will try to interpolate from similar states. This could be risky. (c.f. distributional shift problem and DAgger).

      • "If a policy is used to describe a task, it will be less succinct since for each state we have to give a description of what the behaviour should look like". From this post

    • 2- IRL methods acts optimally w.r.t. the underlying reward function, which could be better, since it is more robust.
      • This is particularly useful if we have an expert policy that is only approximately optimal.
      • In other words, a policy that is better than the "expert" can be derived, while having very little exploration. This "minimal exploration" property is useful for tasks such as AD.
      • This is sometimes refers to as Apprenticeship learning.
• One new concept I learnt: State-visitation frequency (it reminds me some concepts of Markov chains).
  • Take a policy π. Let run the agent with it. Count how often it sees each state. This is called the state-visitation frequency (note it is for a specific π).
  • Two ideas from there:
    • Iterating until this state-visitation frequency stops changing yields the converged frequency function.
    • Multiplying that converged state-visitation frequency with reward gives another perspective to the value function.
      • The value function can now be seen as a linear combination of the expected feature count µ(φk)(π) (also called successor feature).
• One common assumption: -> "The solution is a weighted linear combination of a set of reward features".
  • This greatly reduces the search space.

  • "It allowed the use of feature expectations as a sufficient statistic for representing the value of trajectories or the value of an expert’s policy."

• Known IRL issues (and solutions):
  • 1- This is an under-constrained learning problem.

    • "Many reward functions could explain the observations".

    • Among them, they are highly "degenerate" functions with all reward values zero.
    • One solution is to impose constraints in the optimization.
      • For instance try to maximize the sum of "value-margins", i.e. the difference between the value functions of the best and the second-best actions.

      • "mmp makes the solution policy have state-action visitations that align with those in the expert’s demonstration."

      • "maxent distributes probability mass based on entropy but under the constraint of feature expectation matching."

    • Another common constraint is to encourage the reward function to be as simple as possible, similar to L1 regularization in supervised learning.
  • 2- Two incomplete models:
    • 2.1- How to deal with incomplete/absent model of transition probabilities?
    • 2.2- How to select the reward features?

      • "[One could] use neural networks as function approximators that avoid the cumbersome hand-engineering of appropriate reward features".

    • "These extensions share similarity with model-free RL where the transition model and reward function features are also unknown".

  • 3- How to deal with noisy demonstrations?
    • Most approaches assume a Gaussian noise and therefore apply Gaussian filters.
• How to classify IRL methods?
  • It can be useful to ask yourself two questions:
    • 1- What are the parameters of the Hypothesis R function`?
      • Most approaches use the "linear approximation" and try to estimate the weights of the linear combination of features.
    • 2- What for "Divergence Metric", i.e. how to evaluate the discrepancy to the expert demonstrations?

      • "[it boils down to] a search in reward function space that terminates when the behavior derived from the current solution aligns with the observed behavior."

      • How to measure the closeness or the similarity to the expert?
        • 1- Compare the policies (i.e. the behaviour).
          • E.g. how many <state, action> pairs are matching?

          • "A difference between the two policies in just one state could still have a significant impact."

        • 2- Compare the value functions (they are defined over all states).
          • The authors mention the inverse learning error (ILE) = || V(expert policy) - V(learnt policy) || and the value loss (use as a margin).
  • Classification:
    • Margin-based optimization: Learn a reward function that explains the demonstrated policy better than alternative policies by a margin (address IRL's "solution ambiguity").
      • The intuition here is that we want a reward function that clearly distinguishes the optimal policy from other possible policies.
    • Entropy-based optimization: Apply the "maximum entropy principle" (together with the "feature expectations matching" constraint) to obtain a distribution over potential reward functions.
    • Bayesian inference to derive P(^R|demonstration).
      • What for the likelihood P(<s, a> | ˆR)? This probability is proportional to the exponentiated value function: exp(Q[s, a]).
    • Regression and classification.

Author: Weiss, E.

The assumption-free reward function that uses a simple polynomial form based on state and action values at each time step does better at minimizing both safety and mobility objectives, even though it does not incorporate human knowledge of typical reward function structures. About Pareto optimum: at these points, it becomes impossible to improve in the minimization of one objective without worsening our minimization of the other objective). Source.

• What?
  • A Project from the Stanford course (AA228/CS238 - Decision Making under Uncertainty). Examples of student projects can be found here.
• My main takeaway:
  • A simple problem that illustrates the need for (learning more about) IRL.
• The merging task is formulated as a simple MDP:
  • The state space has size 3 and is discretized: lat + long ego position and long position of the other car.
  • The other vehicle transitions stochastically (T) according to three simple behavioural models: fast, slow, average speed driving.
  • The main contribution concerns the reward design: how to shape the reward function for this multi-objective (trade-off safety / efficiency) optimization problem?
• Two reward functions (R) are compared:

  • 1- "The first formulation models rewards based on our prior knowledge of how we would expect autonomous vehicles to operate, directly encoding human values such as safety and mobility into this problem as a positive reward for merging, a penalty for merging close to the other vehicle, and a penalty for staying in the on-ramp."

  • 2- "The second reward function formulation assumes no prior knowledge of human values and instead comprises a simple degree-one polynomial expression for the components of the state and the action."

    • The parameters are tuned using a sort of grid search (no proper IRL).
• How to compare them?
  • Since both T and R are known, a planning (as opposed to learning) algorithm can be used to find the optimal policy. Here value iteration is implemented.
  • The resulting agents are then evaluated based on two conflicting objectives:

    • "Minimizing the distance along the road at which point merging occurs and maximizing the gap between the two vehicles when merging."

  • Next step will be proper IRL:

    • "We can therefore conclude that there may exist better reward functions for capturing optimal driving policies than either the intuitive prior knowledge reward function or the polynomial reward function, which doesn’t incorporate any human understanding of costs associated with safety and efficiency."

Authors: Tian, R., Li, N., Kolmanovsky, I., Yildiz, Y., & Girard, A.

• This paper builds on several works (also analysed further below):

  • "Adaptive Game-Theoretic Decision Making for Autonomous Vehicle Control at Roundabouts" - (Tian, Li, Li, et al., 2019).
  • "Game Theoretic Modeling of Vehicle Interactions at Unsignalized Intersections and Application to Autonomous Vehicle Control" - (N. Li, Kolmanovsky, Girard, & Yildiz, 2018).
  • "Game-theoretic modeling of driver and vehicle interactions for verification and validation of autonomous vehicle control systems" - (N. Li et al., 2016).

• Addressed problem: unsignalized intersections with heterogenous driving styles (k in [0, 1, 2])

  • The problem is formulated using the level-k game-theory formalism (See analysed related works for more details).

• One idea: use imitation learning (IL) to obtain an explicit level-k control policy.

  • A level-k policy is a mapping pi: <ego state, other's states, ego k> -> <sequence of ego actions>.
  • The ego-agent maintains belief over the level k of other participants. These estimates are updated using maximum likelihood and Bayes rule.
  • A first attempt with supervised learning on a fix dataset (behavioural cloning) was not satisfying enough due to its drift shortcomings:

    • "A small error may cause the vehicle to reach a state that is not exactly included in the dataset and, consequently, a large error may occur at the next time step."

  • The solution is to also aggregate experience sampled from the currently learnt policy.
    • The DAgger algorithm (Dataset Aggregation) is used in this work.
    • One point I did not understand: I am surprised that no initial "off-policy" demonstrations is used. The dataset D is initialized as empty.
    • The policy is represented by a neural network.

In the rule-based stochastic driver model describing the other agents, 2 thresholds are introduced: The reaction threshold, sampled from the range {−1.5m, 0.4m}, describes whether or not the agent reacts to the ego car. The aggression threshold, uniformly sampled {−2.2, 1.1m}, describes how the agent reacts. Source.

Two tree searches are performed: The first step is to identify a target merging gap based on the probability of a successful merge for each of them. The second search involves forward simulation and collision checking for multiple ego and traffic intentions. In practice the author found that ''the coarse tree - i.e. with intention only - was sufficient for long term planning and only one intention depth needed to be considered for the fine-grained search''. This reduces this second tree to a matrix game. Source.

Author: Isele, D.

• Three motivations when working on decision-making for merging in dense traffic:
  • 1- Prefer game theory approaches over rule-based planners.
    • To avoid the frozen robot issue, especially in dense traffic.

    • "If the ego car were to wait for an opening, it may have to wait indefinitely, greatly frustrating drivers behind it".

  • 2- Prefer the stochastic game formulation over MDP.
    • Merging in dense traffic involves interacting with self-interested agents ("self-interested" in the sense that they want to travel as fast as possible without crashing).

    • "MDPs assume agents follow a set distribution which limits an autonomous agent’s ability to handle non-stationary agents which change their behaviour over time."

    • "Stochastic games are an extension to MDPs that generalize to multiple agents, each of which has its own policy and own reward function."

    • In other words, stochastic games seen more appropriate to model interactive behaviours, especially in the forward rollout of tree search:
      • An interactive prediction model based on the concept of counterfactual reasoning is proposed.
      • It describes how behaviour might change in response to ego agent intervention.
  • 3- Prefer tree search over neural networks.

    • "Working with the game trees directly produces interpretable decisions which are better suited to safety guarantees, and ease the debugging of undesirable behaviour."

    • In addition, it is possible to include stochasticity for the tree search.
      • More precisely, the probability of a successful merge is computed for each potential gap based on:
        • The traffic participant’s willingness to yield.
        • The size of the gap.
        • The distance to the gap (from our current position).
• How to model other participants, so that they act "intelligently"?

  • "In order to validate our behaviour we need interactive agents to test against. This produces a chicken and egg problem, where we need to have an intelligent agent to develop and test our agent. To address this problem, we develop a stochastic rule-based merge behaviour which can give the appearance that agents are changing their mind."

  • This merging-response driver model builds on the ideas of IDM, introducing two thresholds (c.f. figure):
    • One threshold governs whether or not the agent reacts to the ego car,
    • The second threshold determines how the agent reacts.

    • "This process can be viewed as a rule-based variant of negotiation strategies: an agent proposes he/she go first by making it more dangerous for the other, the other agent accepts by backing off."

• How to reduce the computational complexity of the probabilistic game tree search, while keeping safely considerations ?
  • The forward simulation and the collision checking are costly operations. Especially when the depth of the tree increases.
  • Some approximations include reducing the number of actions (for both the ego- and the other agents), reducing the number of interacting participants and reducing the branching factor, as can been seen in the steps of the presented approach:
    • 1- Select an intention class based on a coarse search. - the ego-actions are decomposed into a sub-goal selection task and a within-sub-goal set of actions.
    • 2- Identify the interactive traffic participant. - it is assumed that at any given time, the ego-agent interacts with only one other agent.
    • 3- Predict other agents’ intentions. - working with intentions, the continuous action space can be discretized. It reminds me the concept of temporal abstraction which reduces the depth of the search.
    • 4- Sample and evaluate the ego intentions. - a set of safe (absence of collision) ego-intentions can be generated and assessed.
    • 5- Act, observe, and update our probability models. - the probability of safe successful merge.

The agent maintain belief on the k parameter for other vehicles and updates it at each step. Source.

Authors: Sankar, G. S., & Han, K.

• One related work (described further below): Decision making in dynamic and interactive environments based on cognitive hierarchy theory: Formulation, solution, and application to autonomous driving by (Li, S., Li, N., Girard, A., & Kolmanovsky, I. 2019).
• One framework: "level-k game-theoretic framework".
  • It is used to model the interactions between vehicles, taking into account the rationality of the other agents.
  • The agents are categorized into hierarchical structure of their cognitive abilities, parametrized with a reasoning depth k in [0, 1, 2].
    • A level-0 vehicle considers the other vehicles in the traffic scenario as stationary obstacles, hence being "aggressive".
    • A level-1 agent assumes other agents are at level-0. ...
  • This parameter k is what the agent must estimate to model the interaction with the other vehicles.
• One term: "disturbance set".
  • This set, denoted W, describe the uncertainty in the position estimate of other vehicle (with some delta, similar to the variance in Kalman filters).
  • It should capture both the uncertainty about the transition model and the uncertainty about the driver models.
  • This set is considered when taking action using a "feedback min-max strategy".
    • I must admit I did not fully understand the concept. Here is a quote:

    • "The min-max strategy considers the worst-case disturbance affecting the behaviour/performance of the system and provides control actions to mitigate the effect of the worst-case disturbance."

  • The important idea is to adapt the size of this W set in order to avoid over-conservative behaviours (compared to reachable-set methods).
    • This is done based on the confidence in the estimated driver model (probability distribution of the estimated k) for the other vehicles.
      • If the agent is sure that the other car follows model 0, then it should be "fully" conservative.
      • If the agent is sure it follows level 1, then it could relax its conservatism (i.e. reduce the size of the disturbance set) since it is taken into consideration.
• I would like to draw some parallels:
  • With (PO)MDP formulation: for the use of a transition model (or transition function) that is hard to define.
  • With POMDP formulation: for the tracking of believes about the driver model (or intention) of other vehicles.
    • The estimate of the probability distribution (for k) is updated at every step.
  • With IRL: where the agent can predict the reaction of other vehicles assuming they act optimally w.r.t a reward function it is estimating.
  • With MPC: the choice of the optimal control following a receding horizon strategy.

• Note:
  • this 190-page thesis is also referenced in the sections for prediction and planning.
  • I really like how the author organizes synergies between three modules that are split and made independent in most modular architectures:
    • (1) driver model
    • (2) behaviour prediction
    • (3) decision-making

Author: Sierra Gonzalez, D.

• Related work: there are close concepts to the approach of (Kuderer et al., 2015) referenced below.
• One idea: encode the driving preferences of a human driver with a reward function (or cost function), mentioning a quote from Abbeel, Ng and Russell:

“The reward function, rather than the policy or the value function, is the most succinct, robust, and transferable definition of a task”.

• Other ideas:

  • Use IRL to avoid the manual tuning of the parameters of the reward model. Hence learn a cost/reward function from demonstrations.
  • Include dynamic features, such as the time-headway, in the linear combination of the cost function, to take the interactions between traffic participants into account.
  • Combine IRL with a trajectory planner based on "conformal spatiotemporal state lattices".
    • The motivation is to deal with continuous state and action spaces and handle the presence of dynamic obstacles.
    • Several advantages (I honestly did not understand that point): the ability to exploit the structure of the environment, to consider time as part of the state-space and respect the non-holonomic motion constraints of the vehicle.

• One term: "planning-based motion prediction".

  • The resulting reward function can be used to generate trajectory (for prediction), using optimal control.
  • Simply put, it can be assumed that each vehicle in the scene behaves in the "risk-averse" manner encoded by the model, i.e. choosing actions leading to the lowest cost / highest reward.
  • This method is also called "model-based prediction" since it relies on a reward function or on the models of an MDP.
  • This prediction tool is not used alone but rather coupled with some DBN-based manoeuvre estimation (detailed in the section on prediction).

Kernel functions are used on the continuous state space to obtain a smooth reward function using linear function approximation. Source.

As often, the divergence metric (to measure the gap between one candidate and the expert) is the expected value function estimated on sampled trajectories. Example of how to use 2 other candidate policies. I am still confused that each of their decision is based on a state seen by the expert, i.e. they are not building their own full trajectory. Source.

Authors: Gao, H., Shi, G., Xie, G., & Cheng, B.

• One idea: A simple and "educationally relevant" application to IRL and a good implementation of the algorithm of (Ng A. & Russell S., 2000): Algorithms for Inverse Reinforcement Learning.
  • Observe human behaviours during a "car following" task, assume his/her behaviour is optimal w.r.t. an hidden reward function, and try to estimate that function.
  • Strong assumption: no lane-change, no overtaking, no traffic-light. In other worlds, just concerned about the longitudinal control.
• Which IRL method?
  • Maximum-margin. Prediction aim at learning a reward function that explains the demonstrated policy better than alternative policies by a margin.
  • The "margin" is there to address IRL's solution ambiguity.
• Steps:
  • 1- Define a simple 2d continuous state space s = (s0, s1).
    • s0 = ego-speed divided into 15 intervals (each centre will serve to build means for Gaussian kernel functions).
    • s1 = dist-to-leader divided into 36 intervals (same remark).
    • A normalization is additionally applied.
  • 2- Feature transformation: Map the 2d continuous state to a finite number of features using kernel functions.
    • I recommend this short video about feature transformation using kernel functions.
    • Here, Gaussian radial kernel functions are used:
      • Why "radial"? The closer the state to the centre of the kernel, the higher the response of the function. And the further you go, the larger the response "falls".
      • Why "Gaussian"? Because the standard deviation describes how sharp that "fall" is.
      • Note that this functions are 2d: mean = (the centre of one speed interval, the centre of one dist interval).
    • The distance of the continuous state s = (s0, s1) to each of the 15*36=540 means s(i, j) can be computed.
    • This gives 540 kernel features f(i, j) = K(s, s(i, j)).
  • 3- The one-step reward is assumed to be linear combination of that features.
    • Given a policy, a trajectory can be constructed.
      • This is a list of states. This list can be mapped to a list of rewards.
      • The discounted sum of this list leads to the trajectory return, seen as expected Value function.
    • One could also form 540 lists for this trajectory (one per kernel feature). Then reduce them by discounted_sum(), leading to 540 V_f(i, j) per trajectory.
      • The trajectory return is then a simple the linear combination: theta(i, j) * V_f(i, j).
    • This can be computed for the demonstrating expert, as well as for many other policies.
    • Again, the task it to tune the weights so that the expert results in the largest values, against all possible other policies.
  • 4- The goal is now to find the 540 theta(i, j) weights parameters solution of the max-margin objective:
    • One goal: costly single-step deviation.
      • Try to maximize the smallest difference one could find.
        • I.e. select the best non-expert-policy action and try to maximize the difference to the expert-policy action in each state.
        • max[over theta] min[over π] of the sum[over i, j] of theta(i, j) * [f_candidate(i, j) - f_expert(i, j)].
      • As often the value function serves as "divergence metric".
    • One side heuristic to remove degenerate solutions:

      • "The reward functions with many small rewards are more natural and should be preferred". from here.

      • Hence a regularization constraint (a constraint, not a loss like L1!) on the theta(i, j).
    • The optimization problem with strict constraint is transformed into an optimization problem with "inequality" constraint.
      • Violating constraints is allowed by penalized.
      • As I understood from my readings, that relaxes the linear assumption in the case the true reward function cannot be expressed as a linear combination of the fixed basis functions.
    • The resulting system of equations is solved here with Lagrange multipliers (linear programming was recommended in the orginal max-margin paper).
  • 5- Once the theta(i, j) are estimated, the R can be expressed.
• About the other policy "candidates":

  • "For each optimal car-following state, one of the other car-following actions is randomly selected for the solution".

  • In other words, in V(expert) > V(other_candidates) goal, "other_candidates" refers to random policies.
  • It would have been interesting to have "better" competitors, for instance policies that are optional w.r.t. the current estimate of R function. E.g. learnt with RL algorithms.
    • That would lead to an iterative process that stops when R converges.

Source.

Authors: He, X., Xu, D., Zhao, H., Moze, M., Aioun, F., & Franck, G.

• One idea: couple learning and sampling for motion planning.
  • More precisely, learn from human demonstrations (offline) how to weight different contributions in a cost function (as opposed to hand-crafted approaches).
  • This cost function is then used for trajectory planning (online) to evaluate and select one trajectory to follow, among a set of candidates generated by sampling methods.
• One detail: the weights of the optimal cost function minimise the sum of [prob(candidate) * similarities(demonstration, candidate)].
  • It is clear to me how a cost can be converted to some probability, using softmax().
  • But for the similarity measure of a trajectory candidate, how to compute "its distance to the human driven one at the same driving situation"?
  • Should the expert car have driven exactly on the same track before or is there any abstraction in the representation of the situation?
  • How can it generalize at all if the similarity is estimated only on the location and velocity? The traffic situation will be different between two drives.
• One quote:

"The more similarity (i.e. less distance) the trajectory has with the human driven one, the higher probability it has to be selected."

Source.

Authors: Kuderer, M., Gulati, S., & Burgard, W.

• One important contribution: Deal with continuous features such as integral of jerk over the trajectory.
• One motivation: Derive a cost function from observed trajectories.
  • The trajectory object is first mapped to some feature vector (speed, acceleration ...).
• One Q&A: How to then derive a cost (or reward) from these features?
  • The authors assume the cost function to be a linear combination of the features.
  • The goal is then about learning the weights.
  • They acknowledge in the conclusion that it may be a too simple model. Maybe neural nets could help to capture some more complex relations.
• One concept: "Feature matching":

  • "Our goal is to find a generative model p(traj| weights) that yields trajectories that are similar to the observations."

  • How to define the "Similarity"?
    • The "features" serve as a measure of similarity.
• Another concept: "ME-IRL" = Maximum Entropy IRL.
  • One issue: This "feature matching" formulation is ambiguous.
    • There are potentially many (degenerated) solutions p(traj| weights). For instance weights = zeros.
  • One idea is to introduce an additional goal:
    • In this case: "Among all the distributions that match features, they to select the one that maximizes the entropy."
  • The probability distribution over trajectories is in the form exp(-cost[features(traj), θ]), to model that agents are exponentially more likely to select trajectories with lower cost.
• About the maximum likelihood approximation in MaxEnt-IRL:
  • The gradient of the Lagrangian cost function turns to be the difference between two terms:
    • 1- The empirical feature values (easy to compute from the recorded).
    • 2- The expected feature values (hard to compute: it requires integrating over all possible trajectories).
      • An approximation is made to estimate the expected feature values: the authors compute the feature values of the "most" likely trajectory, instead of computing the expectations by sampling.
    • Interpretation:

      • "We assume that the demonstrations are in fact generated by minimizing a cost function (IOC), in contrast to the assumption that demonstrations are samples from a probability distribution (IRL)".

• One related work:
  • "Learning to Predict Trajectories of Cooperatively Navigating Agents" by (Kretzschmar, H., Kuderer, M., & Burgard, W., 2014).