Paper list of Reinforcement Learning (RL) applied on transportation

• RL-for-Transportation
  • Ride-sourcing system
    • Survey
    • Dataset
    • Competition
    • Book
    • Paper
      • Order dispatching
        • 1. A Taxi Order Dispatch Model based On Combinatorial Optimization. 2017. KDD
        • 2. Large-scale order dispatch in on-demand ride-hailing platforms: A learning and planning approach. 2018. KDD
        • 3. A Deep Value-network Based Approach for Multi-Driver Order Dispatching. 2019. KDD
        • 4. Efficient Ridesharing Order Dispatching with Mean Field Multi-Agent Reinforcement Learning. 2019. WWW
        • 5. Multi-Agent Reinforcement Learning for Order-dispatching via Order-Vehicle Distribution Matching. 2019. CIKM
      • Order delaying
        • 1. Learning to delay in ride-sourcing systems: a multi-agent deep reinforcement learning framework. 2019. TKDE
        • 2. Optimizing matching time intervals for ride-hailing services using reinforcement learning. 2021. TRC
      • Order pooling
        • 1. Optimizing Taxi Carpool Policies via Reinforcement Learning and Spatio-Temporal Mining. 2018. Big Data
        • 2. DeepPool: Distributed Model-free Algorithm for Ride-sharing using Deep Reinforcement Learning. 2019. ITS
        • 3. AdaPool: A Diurnal-Adaptive Fleet Management Framework using Model-Free Deep Reinforcement Learning and Change Point Detection. 2021.
        • 4. An Integrated Decomposition and Approximate Dynamic Programming Approach for On-Demand Ride Pooling. 2018. ITS
        • 5. Neural Approximate Dynamic Programming for On-Demand Ride-Pooling. 2020. AAAI
        • 6. Conditional Expectation Based Value Decomposition For Scalable On-demand Ride Pooling
      • Order pricing
        • 1. InBEDE: Integrating Contextual Bandit with TD Learning for Joint Pricing and Dispatch of Ride-Hailing Platforms. 2019. ICDM
      • Vehicle relocation
        • 1. A Cost-Effective Recommender System for Taxi Drivers. 2014. KDD
        • 2. Optimizing Taxi Driver Profit Efficiency: A Spatial Network-based Markov Decision Process Approach. 2015. Big Data
        • 3. Optimal Passenger-Seeking Policies on E-hailing Platforms Using Markov Decision Process and Imitation Learning. 2020. TRC
        • 4. MOVI: A Model-Free Approach to Dynamic Fleet Management. 2018. INFOCOM
        • 5. Credit Assignment For Collective Multiagent RL With Global Rewards. 2018. NIPS
        • 6. Efficient Large-Scale Fleet Management via Multi-Agent Deep Reinforcement Learning. 2018. KDD
        • 7. Real-world Ride-hailing Vehicle Repositioning using Deep Reinforcement Learning. 2020. NIPS
      • Joint dispatching and relocation
        • 1. CoRide: Joint Order Dispatching and Fleet Management for Multi-Scale Ride-Hailing Platforms. 2019. CIKM
        • 2. Value Function is All You Need: A Unified Learning Framework for Ride Hailing Platforms. 2021. KDD
        • 3. An Integrated Reinforcement Learning and Centralized Programming Approach for Online Taxi Dispatching. 2021. TNNLS
        • 4. Path-based dynamic pricing for vehicle allocation in ridesharing systems with fully compliant drivers. 2019. TRB
  • Intersection control
    • Survey
    • Dataset
    • Competition
    • Paper
      • Single-agent
        • 1. IntelliLight: A Reinforcement Learning Approach for Intelligent Traffic Light Control. 2018. KDD
        • 2. Learning Traffic Signal Control from Demonstrations. 2019. CIKM
        • 3. PressLight: Learning Max Pressure Control to Coordinate Traffic Signals in Arterial Network. 2019. KDD
        • 4. PDLight: A Deep Reinforcement Learning Traffic Light Control Algorithm with Pressure and Dynamic Light Duration. 2020
        • 5. Learning Phase Competition for Traffic Signal Control. 2019. CIKM
        • 6. Toward A Thousand Lights: Decentralized Deep Reinforcement Learning for Large-Scale Traffic Signal Control. 2020. AAAI
        • 7. AttendLight: Universal Attention-Based Reinforcement Learning Model for Traffic Signal Control. 2020. NIPS
        • 8. GeneraLight: Improving Environment Generalization of Traffic Signal Control via Meta Reinforcement Learning. 2020. CIKM
        • 9. MetaLight: Value-Based Meta-Reinforcement Learning for Traffic Signal Control. 2020. AAAI
      • Multi-agent
        • 1. CoLight: Learning Network-level Cooperation for Traffic Signal Control. 2019. CIKM
        • 2. Multi-agent Reinforcement Learning for Networked System Control. 2020. ICLR
        • 3. Meta Variationally Intrinsic Motivated Reinforcement Learning for Decentralized Traffic Signal Control. 2021
        • 4. Hierarchically and Cooperatively Learning Traffic Signal Control. 2021. AAAI

1. Reinforcement Learning for Ridesharing: A Survey. 2021

1. DiDi GAIA Dataset

1. Learning to Dispatch and Reposition on a Mobility-on-Demand Platform

1. Approximate Dynamic Programming: Solving the curses of dimensionality. Powell, W. B. (2007).

1. predict cancellation probability of vehicle-order pair $p_{ij}$
2. maximize total success rate:
   1. $a_{ij}$: matching decision
   2. NP hard combinatorial optimization
      1. HillClimbing Algorithm

1. bipartite matching + evaluation
   1. weight: advantage trick
      1. discounted reward + furture value - current value
   2. state: time & space zone (no contextual information)
   3. value: policy evaluation (offline)

1. bipartite matching + policy evaluation
   1. weight: advantage value - distance
   2. state:
      1. time & space zone + supply-demand info
      2. coarse coding with hierarchical hexagon grid
   3. value (offline)
      1. policy evaluation: with supply-demand info
      2. distillation: marginalize time-space value

1. MARL (on policy)
   1. state: contextual information
   2. acrion: active order pool
      1. mean action: defined as number of neighbor drivers
   3. reward:
      1. order fare
      2. pick distance
      3. destination supply-demand gap

1. MARL (on policy)
   1. state: contextual features
   2. TD loss + KL regulizer
      1. regulizer: distribution between order and vehicles

1. MARL (on policy)
   1. state: contextual features
   2. action: {0,1}, match or hold
   3. reward: customer waiting time
      1. weighted global + individual reward

1. RL (off policy)
   1. state: global grid-based state -> flatten
   2. action: {0,1}, match or hold
   3. reward:
      1. matching wating time
      2. pickup wiating time

1. RL (on policy)
   1. state: time & space grid
   2. action: wait, TK1, TK2
      1. wait: stay current location
      2. TK1: pick orders within max pick time
      3. TK2: TK1 + larger pick time for second order
   3. reward: effective distance traveled

1. RL (on policy)
   1. state: global supply-demand profile map
   2. action (sequentially): follow shorest path
      1. find another customer
      2. next zone
   3. reward
      1. served customer
      2. detour time

1. The same as DeepPool
   1. consider the change of MDP （with different models）
   2. online Dirichlet change point detection (ODCP) to detect changes

1. ADP
   1. decision
      1. routes is determined using the shortest-path strategy
      2. one decision one assignment
      3. linear approximation for value function
      4. linear assignment problem
         1. dual update

1. ADP
   1. decision
      1. follow shorest path
      2. generate feasible order combinations
      3. value approximation: individual value
         1. low-dimensional embedding for each location of vehicle
      4. linear assignment
         1. TD update

1. like NeuralADP
   1. average of neighbor value to approximate indivadule value:
   2. each neighbor value is the conditional expectation of action
   3. final approximation

1. RL (on-policy)
   1. use TD learning to calculate furture value
   2. use Contextual Bandit to give price

1. recommend route for vacant vehicles
   1. segment profit
      1. earning:
      2. cost:
   2. expected route profits:
   3. algorithm
      1. Brute-Force based MNP Recommendation
      2. Recursive Recommendation Strategy
   4. multi-driver routes recommendation
      1. recommend the route with the lowest correlationship to the second driver

1. Defined as MDP:
   1. calibrate pick probability (discounted by number of taxis)
   2. passenger destination probability
2. solving
   1. Rolling Horizon:
   2. DP approach
      1. discounting pick prob:

1. similar to last one
   1. long-horizon
   2. discounted prob of competing drivers

1. RL (on-policy)
   1. state: heatmap + CNN
   2. making decision sequentially for each vehicle

1. MARL (on policy)
   1. centralize critic + decentralized policy
   2. conut based global state
   3. difference policy gradient
      1. Wonderful Life Utility (WLU)
         1. with and without agent i
      2. Aristrocratic Utility (AU)
         1. fix actions of other agents, marginalize agent i
         2. conterfactual
   4. approximating central xritic
      1. linear combination
         1. individual information
         2. f without info of other agents
      2. first-order approximation
         1. those coefficients are evaluated at the overall state-action counts

1. RL (on-policy)
   1. state + contextual features
   2. action: neighbor girds
      1. sequentially make decision
      2. avoid moving in conflict directions
         1. add collaborative context indicating directions of previous vehicles
      3. avoid moving to low-value grid

1. Policy evaluation (off-line)
   1. time-space value function
      1. dual policy evaluation
         1. conditional value: V(s|b)
         2. marginal value: V(s)
      2. Value-based Policy Search (VPS)
         1. one-step:
         2. two-step:
      3. implementation
         1. step selection: small works well
         2. long search: choose global top points
         3. contextual value:
         4. SD regulizer
            1. add destination supply-demand gap

1. MARL (on-policy), sequentially decision making
   1. hierarchical strucutre
      1. upper level
         1. generate encoding of env using RNN
      2. lower level
         1. using info from upper level, generate prob of different grids
         2. dispatching and relocating
   2. reward
      1. gap between manager’s entropy and global average entropy
      2. KL divergence of supplt and demand
   3. coordination
      1. using attention to aggregate info of neighbor grids

1. policy evaluation (off-line)
2. on-line updateing
   1. using current transitions
3. ensemble of offline and online value
4. dispatching: bipartite matching
5. relocaintg:

1. RL (on-policy)
   1. centralized programming model
      1. planning in both dispatching and relocating
   2. TD learning for updating value function

1. ADP (marco level)
   1. decision
   2. path based pricing (market cleaning)
   3. routing after distaching (constrained zone choice)
   4. order sharing
   5. relocation
2. piece-wise linear approximation of value function

1. A survey on traffic signal control methods. 2019
2. Recent advances in reinforcement learning for traffic signal control: A survey of models and evaluation. 2021

1. Cityflow: A multi-agent reinforcement learning environment for large scale city traffic scenario
2. Reinforcement Learning for Traffic Signal Control

1. City Brain Challenge

1. state:
2. action: {1,0}
   1. change to the next phase
   2. keep phase
3. reward:
   1. wighted reward of (queue length, delay, waiting time, light switches, number of vehicles, and travel time)
4. algorithm:
   1. DQN

1. imitation learning
   1. actor: ![](pic/2021-10-25-19-38-48.png)
   2. critic:![](pic/2021-10-25-19-39-39.png)

1. state:
   1. current pahse (one-hot)
   2. number of vehicles
2. action:
   1. pre-defined phases
3. reward:
   1. pressure for movement:
   2. total pressure:
4. algorithm:
   1. DQN

1. reward
   1. Pressure with Remaining Capacity of Outgoing Lane:

1. state
   1. current phase (one-hot)
   2. number of vehicles
2. action:
   1. pre-defined phases
3. reward:
   1. queue length
4. invariance:
   1. flip and rotation

1. FRAP + pressure reward
2. reward:
   1. pressure based on queuing vehicles

1. state
   1. traffic characteristics in each lane
2. action
   1. pre-defined phases
3. reward
   1. pressure
4. algorithm:
   1. PG + MC
5. invariance
   1. topology

1. gradient-based meta learning
   1. training agent in clusetered environments
   2. meta-training

1. FRAP + gradient-based meta-learning

1. information aggregation:
2. GAT to aggregation neighbor intersection information

1. differentiable commnication
   1. 

1. intrinsic reward
   1. error of predict neighbor reward and transitions
2. latent variable policy
   1. RNN encoded environment

1. Hierarchy
   1. select sub-policies with different reward function
2. weighted local and neighbor reward
   1. adaptive weighting mechanism