Human-Aligned Safe RL for Highway On-Ramp Merging

2026

Adaptive Constraint Regulation for Human Preference-Aware Safe Reinforcement Learning of On-Ramp Merging

Jingjia Teng, Wenjie Huang, Shijie Yuan, and 5 more authors

Machines, May 2026

Abs DOI Bib HTML Code

Safe reinforcement learning for autonomous driving requires balancing safety constraints with driving efficiency. This paper proposes an adaptive constraint regulation framework for human preference-aware safe reinforcement learning in highway on-ramp merging scenarios. The method dynamically adjusts constraint limits according to traffic conditions and driver risk preferences, combined with an action shielding mechanism that filters unsafe decisions before execution. A hierarchical architecture separates high-level maneuver decisions from low-level model predictive control, ensuring both safety compliance and driving efficiency. Simulation experiments across varying traffic densities demonstrate that the proposed approach achieves a superior safety–efficiency trade-off compared to fixed-constraint baselines, maintaining collision-free merging while improving travel efficiency.
@article{teng2026adaptive, title = {Adaptive Constraint Regulation for Human Preference-Aware Safe Reinforcement Learning of On-Ramp Merging}, author = {Teng, Jingjia and Huang, Wenjie and Yuan, Shijie and Hu, Manjiang and Qin, Hongmao and Li, Yang and Bian, Yougang and Li, Bai}, journal = {Machines}, volume = {14}, number = {6}, pages = {605}, year = {2026}, month = may, publisher = {MDPI}, doi = {10.3390/machines14060605}, }

Animated MPC-predicted vehicle trajectories during an on-ramp merging episode. — MPC-predicted trajectories during an on-ramp merging episode.

Abstract

Autonomous on-ramp merging requires a practical safety–efficiency trade-off and a policy that reflects driver risk preference. This work proposes a hierarchical human-aligned safe RL framework: a constrained Markov decision process adapts its permissible safety cost to traffic density and user preference; discrete SAC learns high-level manoeuvres; and MPC executes low-level motion. Before execution, an action shield predicts trajectories, checks conflicts with surrounding traffic, and replaces unsafe decisions. Simulations across low-, medium-, and high-density traffic show substantially fewer safety violations while retaining efficient merges.

Keywords

Autonomous Driving
On-Ramp Merging
Safe Reinforcement Learning
Action Shielding
Model Predictive Control
Human Risk Preference

Methods

Hierarchical architecture for preference-aware safe reinforcement learning and model predictive control in highway on-ramp merging. — Hierarchical safe-merging architecture: high-level reinforcement-learning decisions are safety-checked before MPC generates low-level vehicle controls.

1. MPC Prediction and Motion Execution

For a candidate high-level decision, MPC first maps the action to a reference speed and trajectory, transforms the reference from Frenet to Cartesian coordinates, and solves a quadratic program. In prediction mode it returns the ego vehicle’s future states for safety checking; in execution mode it returns the optimized control sequence.

Policy, critic, cost, and target-network construction used by the human-aligned safe reinforcement-learning algorithm. — Network construction for the discrete SAC policy, twin reward critics, cost critic, and target networks.

2. Human-Aligned Safe Reinforcement Learning

Traffic density and the user’s risk preference determine the CMDP cost limit through fuzzy control. The discrete SAC policy samples a manoeuvre, stores its shielded transition with reward and cost, and updates twin Q-networks, the cost network, policy, temperature, Lagrangian multiplier, and target networks. The constrained objective makes safety an explicit learning requirement rather than only a reward term.

Example fuzzy-control inference that maps risk preference and traffic density to a cost limit of 0.0595. — Example Mamdani fuzzy-control inference. The aggregated fuzzy output is defuzzified into a CMDP cost limit of 0.0595.

3. Action Shielding Module

The action shield uses MPC-predicted states to detect collisions, an unsafe continuation after the vehicle has merged, or a failure to reach the merge objective. When a decision is unsafe or invalid, it is replaced with a safe alternative before the low-level controller executes it, improving safety during both exploration and deployment.

Action shielding replaces actions that cause a collision, an unexpected decision, or a failure to merge with safe alternatives. — Action shielding in three typical unsafe situations: collision, unexpected decision after merging, and failure to merge. Unsafe actions are replaced before execution.

Experiments

The experiments use a highway-env on-ramp scenario with low-, medium-, and high-density traffic. The following values reproduce the implementation settings and reported evaluation results from the accompanying paper.

Training curves comparing Dueling DQN, SACD, PPO, and SACD-lambda-TM for crash ratio, average cost, and average reward. — Training comparison with Dueling DQN, SACD, and PPO. Curves show crash ratio, average cost, and average reward; shaded bands indicate the 95% confidence interval across five random seeds.

Ablation training curves comparing SACD, SACD-lambda, SACD-lambda-M, and SACD-lambda-TM for crash ratio, average cost, and average reward. — Ablation training comparison of SACD, SACD-λ, SACD-λ-M, and SACD-λ-TM. Curves show crash ratio, average cost, and average reward; shaded bands indicate the 95% confidence interval across five random seeds.

RL hyperparameters

Parameter	Value
Optimizer	Adam
Policy / critic / cost learning rate	1e-4
Temperature learning rate	1e-4
Initial Lagrangian multiplier	1.0
Lagrangian multiplier learning rate	1e-4
Replay-buffer size	1e5
Batch size	256

Constraint-free RL baselines and the proposed method

Method	Success rate (%)			Collision rate			Average cost			Average time (s)
Method	High	Medium	Low	High	Medium	Low	High	Medium	Low	High	Medium	Low
Dueling DQN	87.0	94.3	99.0	0.013	0.005	0.005	0.50	0.28	0.08	11.78	11.47	10.85
SACD	94.5	97.5	99.2	0.010	0.008	0.005	0.44	0.25	0.10	11.59	11.33	10.82
PPO	99.5	97.7	99.2	0.003	0.018	0.008	0.01	0.03	0.01	12.36	11.62	10.95
SACD-λ-TM (proposed)	99.0	99.5	99.3	0.003	0.005	0.005	0.02	0.02	0.02	11.87	11.46	10.96

Results

Comparison with the lattice-based planner

Method	Success rate (%)			Collision rate			Average time (s)
Method	High	Medium	Low	High	Medium	Low	High	Medium	Low
Lattice	98.5	99.0	97.3	0.012	0.010	0.027	10.44	10.12	9.93
SACD-λ-TM (proposed)	99.0	99.5	99.3	0.003	0.005	0.005	11.87	11.46	10.96

Ablation study

Method	Success rate (%)			Collision rate			Average cost			Average time (s)
Method	High	Medium	Low	High	Medium	Low	High	Medium	Low	High	Medium	Low
SACD	94.5	97.5	99.2	0.010	0.008	0.005	0.44	0.25	0.10	11.59	11.33	10.82
SACD-λ	97.2	97.3	99.3	0.003	0.005	0.005	0.23	0.13	0.04	12.08	11.54	10.99
SACD-λ-M	98.3	98.8	98.9	0.010	0.008	0.003	0.02	0.02	0.02	12.33	11.58	11.02
SACD-λ-TM (proposed)	99.0	99.5	99.3	0.003	0.005	0.005	0.02	0.02	0.02	11.87	11.46	10.96