Beyond RMSE: Do machine-learned models of road user interaction produce human-like behavior?

Autonomous vehicles use a variety of sensors and machine-learned models to predict the behavior of surrounding road users. Most of the machine-learned models in the literature focus on quantitative error metrics like the root mean square error (RMSE) to learn and report their models' capabilities. This focus on quantitative error metrics tends to ignore the more important behavioral aspect of the models, raising the question of whether these models really predict human-like behavior. Thus, we propose to analyze the output of machine-learned models much like we would analyze human data in conventional behavioral research. We introduce quantitative metrics to demonstrate presence of three different behavioral phenomena in a naturalistic highway driving dataset: 1) The kinematics-dependence of who passes a merging point first 2) Lane change by an on-highway vehicle to accommodate an on-ramp vehicle 3) Lane changes by vehicles on the highway to avoid lead vehicle conflicts. Then, we analyze the behavior of three machine-learned models using the same metrics. Even though the models' RMSE value differed, all the models captured the kinematic-dependent merging behavior but struggled at varying degrees to capture the more nuanced courtesy lane change and highway lane change behavior. Additionally, the collision aversion analysis during lane changes showed that the models struggled to capture the physical aspect of human driving: leaving adequate gap between the vehicles. Thus, our analysis highlighted the inadequacy of simple quantitative metrics and the need to take a broader behavioral perspective when analyzing machine-learned models of human driving predictions.


page 1

page 9


Comparing merging behaviors observed in naturalistic data with behaviors generated by a machine learned model

There is quickly growing literature on machine-learned models that predi...

A Stackelberg Game Theoretic Model of Lane-Merging

Merging in the form of a mandatory lane-change is an important issue in ...

Learning Personalized Discretionary Lane-Change Initiation for Fully Autonomous Driving Based on Reinforcement Learning

In this article, the authors present a novel method to learn the persona...

The PREVENTION Challenge: How Good Are Humans Predicting Lane Changes?

While driving on highways, every driver tries to be aware of the behavio...

Predicting Future Lane Changes of Other Highway Vehicles using RNN-based Deep Models

In the event of sensor failure, it is necessary for autonomous vehicles ...

Estimating the Probability That a Vehicle Reaches a Near-Term Goal State Using Multiple Lane Changes

This paper presents a model to estimate the probability of reaching a ne...

Generic Vehicle Tracking Framework Capable of Handling Occlusions Based on Modified Mixture Particle Filter

Accurate and robust tracking of surrounding road participants plays an i...

Please sign up or login with your details

Forgot password? Click here to reset