The Trajectory Planning Method for Intelligent Vehicles Using Reinforcement Learning and Risk-Reachable Sets
Safe RL and constrained optimization for structured-road trajectory planning.
Abstract
Trajectory planning is a key technology for ensuring the safety, comfort, and efficiency of autonomous vehicles. This study investigates the influence of spatially distributed driving risks on trajectory planning and proposes a framework combining risk-reachable sets, safe reinforcement learning, and constrained optimization.
First, a risk-reachable-set construction method is developed by integrating vehicle motion uncertainty with a driving-risk field. Gaussian-process regression is used to model surrounding-vehicle trajectories, while a probability-density-based risk assessment and Conditional Value at Risk are employed to identify high-risk boundaries over time and space.
Second, a safe reinforcement-learning method is proposed to generate initial trajectories. Polynomial trajectory parameters form the action space, and a constrained Markov decision process is solved using proximal policy optimization with a Lagrangian formulation. The generated trajectories are further projected onto the risk-reachable sets to improve safety and feasibility.
Third, a constrained iterative linear quadratic regulator is developed for trajectory optimization. Vehicle kinematics, comfort, efficiency, and safety are incorporated into the objective and constraints, while a barrier-function method ensures that trajectory points remain within safe regions.
Finally, a simulation platform is established, and high-speed merging experiments demonstrate that the proposed framework avoids high-risk areas while maintaining trajectory comfort, efficiency, and vehicle safety.
Keywords
- Intelligent Vehicles
- Trajectory Planning
- Safe Reinforcement Learning
- Risk-Reachable Sets
- Constrained Iterative Linear Quadratic Regulator
Methods
1. Risk Field and Risk-Reachable Set
A vehicle kinematic model and forward simulation produce nominal trajectories under Gaussian noise. Separate Gaussian-process regressors model lateral and longitudinal position uncertainty, which is converted into a time-aware driving-risk field. Forward reachability and CVaR then remove high-risk positions to form a safe, feasible risk-reachable set.
2. Safe RL Initial Trajectory
Trajectory planning is formulated as a constrained Markov decision process. PPO-Lagrangian learns a policy from reward and safety-cost signals, selects target positions, and produces longitudinal and lateral polynomial trajectories. A set-based projection corrects any trajectory point or speed that falls outside the risk-reachable set.
3. Constrained Trajectory Refinement
CILQR starts from the projected feasible trajectory and uses the risk-reachable set as a positional boundary. The objective balances jerk, steering-rate change, curvature, lateral offset, and safety distance; velocity, acceleration, steering, and obstacle constraints are handled with exponential barrier functions.
Experiments
Compared with a Lattice planner in high-speed merging, the proposed planner generates safer and smoother trajectories.It also improves driving efficiency by reducing travel time and increasing average speed.
Results
| Metric | Lattice Baseline | Proposed Planner | Change |
|---|---|---|---|
| Average speed (m/s) | 10.17 | 16.34 | +60.7% |
| Maximum jerk (m/s3) | 11.98 | 5.75 | −52.0% |
| Maximum curvature (m−1) | 0.02 | 0.01 | −50.0% |
| Travel time (s) | 7.40 | 4.80 | −35.1% |
| Maximum risk | 0.999 | 0.266 | −73.4% |
| Average risk | 0.417 | 0.074 | −82.3% |
The proposed planner increased average speed to 16.34 m/s while reducing travel time to 4.80 s. It also produced smoother trajectories, cutting maximum jerk by 52.0% and maximum curvature by 50.0%. Most importantly, the risk-aware reachable-set constraint reduced maximum risk by 73.4% and average risk by 82.3% relative to the Lattice baseline.