Sequential Forecasting of 100,000 Points

by   Xinshuo Weng, et al.

Predicting the future is a crucial first step to effective control, since systems that can predict the future can select plans that lead to desired outcomes. In this work, we study the problem of future prediction at the level of 3D scenes, represented by point clouds captured by a LiDAR sensor, i.e., directly learning to forecast the evolution of >100,000 points that comprise a complete scene. We term this Scene Point Cloud Sequence Forecasting (SPCSF). By directly predicting the densest-possible 3D representation of the future, the output contains richer information than other representations such as future object trajectories. We design a method, SPCSFNet, evaluate it on the KITTI and nuScenes datasets, and find that it demonstrates excellent performance on the SPCSF task. To show that SPCSF can benefit downstream tasks such as object trajectory forecasting, we present a new object trajectory forecasting pipeline leveraging SPCSFNet. Specifically, instead of forecasting at the object level as in conventional trajectory forecasting, we propose to forecast at the sensor level and then apply detection and tracking on the predicted sensor data. As a result, our new pipeline can remove the need of object trajectory labels and enable large-scale training with unlabeled sensor data. Surprisingly, we found our new pipeline based on SPCSFNet was able to outperform the conventional pipeline using state-of-the-art trajectory forecasting methods, all of which require future object trajectory labels. Finally, we propose a new evaluation procedure and two new metrics to measure the end-to-end performance of the trajectory forecasting pipeline. Our code will be made publicly available at


page 1

page 2

page 3

page 4


Unsupervised Sequence Forecasting of 100,000 Points for Unsupervised Trajectory Forecasting

Predicting the future is a crucial first step to effective control, sinc...

Forecasting from LiDAR via Future Object Detection

Object detection and forecasting are fundamental components of embodied ...

PnPNet: End-to-End Perception and Prediction with Tracking in the Loop

We tackle the problem of joint perception and motion forecasting in the ...

Staged Contact-Aware Global Human Motion Forecasting

Scene-aware global human motion forecasting is critical for manifold app...

Argoverse 2: Next Generation Datasets for Self-Driving Perception and Forecasting

We introduce Argoverse 2 (AV2) - a collection of three datasets for perc...

Multi-Camera Trajectory Forecasting with Trajectory Tensors

We introduce the problem of multi-camera trajectory forecasting (MCTF), ...

Multiple Object Forecasting: Predicting Future Object Locations in Diverse Environments

This paper introduces the problem of multiple object forecasting (MOF), ...

Please sign up or login with your details

Forgot password? Click here to reset