Code for CVPR2019 Towards Natural and Accurate Future Motion Prediction of Humans and Animals

Overview

Motion prediction with Hierarchical Motion Recurrent Network

Introduction

This work concerns motion prediction of articulate objects such as human, fish and mice. Given a sequence of historical skeletal joints locations, we model the dynamics of the trajectory as kinematic chains of SE(3) group actions, parametrized by se(3) Lie algebra parameters. A sequence to sequence model employing our novel Hierarchical Motion Recurrent (HMR) Network as the decoder is employed to learn the temporal context of input pose sequences so as to predict future motion.

Instead of adopting the conventional Euclidean L2 loss function for the 3D coordinates, we propose a geodesic regression loss layer on the SE(3) manifold which provides the following advantages.

  • The SE(3) representation respects the anatomical and kinematic constraints of the skeletal model, i.e. bone lengths and physical degrees of freedom at the joints.
  • Spatial relations underlying the joints are fully captured.
  • Subtleties of temporal dynamics are better modelled in the manifold space than Euclidean space due to the absence of redundancy and constraints in the Lie algebra parameterization.

Train and Predict

The main file is found in motion_prediction.py.
To train and predict on default setting, execute the following with python 3.

python motion_prediction.py
FLAGS Default value Possible values Remarks
dataset --dataset Human Human, Fish, Mouse
datatype --datatype lie lie, xyz
action --action all all, actions listed below
training --training=1 0, 1
visualize --visualize=1 0, 1
longterm --longterm=0 0, 1 Super long-term prediction exceeding 60s.
dataset: Human
action: walking, eating or smoking.

To train and predict for different settings, simply set different value for the flags. An example of long term prediction for walking on the Human dataset is given below.

python motion_prediction.py --action walking --longterm=1

Possible actions for Human 3.6m

["directions", "discussion", "eating", "greeting", "phoning",
 "posing", "purchases", "sitting", "sittingdown", "smoking",
 "takingphoto", "waiting", "walking", "walkingdog", "walkingtogether"]

The configuration file is found in training_config.py. There are choices of different LSTM architectures as well as different loss functions that can be chosen in the configuration.

Checkpoint and Output

checkpoints are saved in:

./checkpoint/dataset[Human, Fish, Mouse]/datatype[lie, xyz]_model(_recurrent-steps_context-window_hidden-size)_loss/action/inputWindow_outputWindow

outputs are saved in:

./output/dataset[Human, Fish, Mouse]/datatype[lie, xyz]_model_(_recurrent-steps_context-window_hidden-size)_loss/action/inputWindow_outputWindow

*[ ] denotes possible arguments and ( ) is specific for our HMR model

Comments
  • Reproducing the results

    Reproducing the results

    Hi, I am unable to replicate the results in the paper. the final results reported is always higher than the results we are getting using multiple tf versions and using the same seeds. For example, we ran motion_prediction --action discussion --visualize 0 and get 0.32 & 0.62 & 1.10 & 1.31 & 1.87 & 2.04 & 2.20 & 2.44 The results reported in the paper are: HMR 0.29 0.55 0.83 0.94 1.35 1.49 1.61 1.72

    The validation loss never reduced beyond Best Validation Loss: 0.31701756060123426 in the first epoch.

    Could you please let me know what tf.set_random_seed(112858) you set or if you are able to reproduce the results?

    opened by hondaathma 7
  • The paper and the github topic are inconsistent

    The paper and the github topic are inconsistent

    I I found the paper titled Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction, but your github title is Motion prediction with Hierarchical Motion Recurrent Network, and the project name is lie-group-motion-prediction. The paper explicitly uses Stiefel manifold parameterization of rotation instead of lie-group. Or maybe Stiefel manifold parameterization of rotation is the idea of lie-group? Finally, is the code described in this paper consistent with this github code? Looking forward to your reply thank you very much

    opened by logiclj 5
  • Hard to download the code and data

    Hard to download the code and data

    Hi, This is a great work, I would like to reproduct this project, however, it is hard to download not only by command"git" but also download directly, the documents are always incomplete after "downloading", and I try to use the proxy and it doesn't work, so could you please upload the project to the google cload or any other place for the download operation! Thanks again! Yours, Hu

    opened by hujb48 3
  • Super long term prediction

    Super long term prediction

    Hello, I was wondering how to produce the results for 1000 frames output as in the video given on your site? Could you please upload the trained model? I tried to train the model by setting longterm to true and here is my output for walking: Video Thanks in advance.

    opened by zaverichintan 3
  • Something wrong when i use xyz data

    Something wrong when i use xyz data

    When I set "datatype" as "xyz", I get follow issue, you may forget to set the "chain_idx"

    Traceback (most recent call last): File "/home/cz5270/Lie-Group-Motion-Prediction/src/motion_prediction.py", line 316, in tf.app.run() File "/home/cz5270/anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef) File "/home/cz5270/anaconda3/envs/tf/lib/python3.7/site-packages/absl/app.py", line 300, in run _run_main(main, args) File "/home/cz5270/anaconda3/envs/tf/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main sys.exit(main(argv)) File "/home/cz5270/Lie-Group-Motion-Prediction/src/motion_prediction.py", line 218, in main train() File "/home/cz5270/Lie-Group-Motion-Prediction/src/motion_prediction.py", line 44, in train prediction = models.seq2seq(x, dec_in, config, True) File "/home/cz5270/Lie-Group-Motion-Prediction/src/models.py", line 102, in seq2seq outputs, states = tf.contrib.legacy_seq2seq.rnn_decoder(dec_in, enc_state, dec_cell, loop_function=lf) File "/home/cz5270/anaconda3/envs/tf/lib/python3.7/site-packages/tensorflow/contrib/legacy_seq2seq/python/ops/seq2seq.py", line 152, in rnn_decoder output, state = cell(inp, state) File "/home/cz5270/Lie-Group-Motion-Prediction/src/models.py", line 184, in call output = xyz_resize(output, self.config) File "/home/cz5270/Lie-Group-Motion-Prediction/src/models.py", line 374, in xyz_resize index = config.chain_idx AttributeError: 'train_Config' object has no attribute 'chain_idx'

    opened by mysteryzoz 2
  • Issue in reproducing the results

    Issue in reproducing the results

    Hello, I tried to run the linearized loss by making changes to the loss function as on: Issue

    The numbers I am getting on evaluation are very weird and high for the Mean Joint Error. I have attached the files for the reference. Could you provide the checkpoints for model weights to reproduce the result? Errors_walking.xlsx

    opened by zaverichintan 2
  • Confusion in data loading

    Confusion in data loading

    For humans - The data is loaded from h3.6m for lie datatype whereas the logic of loading lie data exists in read_human function in read_data.py. Why is a different function created to read lie datatype ?

    opened by zaverichintan 1
  • Geodesic loss

    Geodesic loss

    Hi, I realized in your code " loss_functions.py" there are "l2_loss" and ' linearizedlie_loss', I would like to ask do you have the code for the geodesic loss image mentioned in your paper Investigating Pose Representations and Motion Contexts Modeling for 3D Motion Prediction.

    opened by joylinmimi 0
  • how to calculate errors in 3D coordinate

    how to calculate errors in 3D coordinate

    After restoring your pretrained model and setting the random seed, I calculate the 3D coordinate error between y_t_xyz and y_p_xyz using the following L2-norm, code

    def xyz_error(gt, out):
        '''
        gt: B, 100, 66
        '''
        batch_size, seq_len, features = gt.shape
        t_3d = np.zeros(seq_len)
    
        gt = gt.reshape(batch_size, seq_len, -1, 3) # B, 100, 22, 3
        out = out.reshape(batch_size, seq_len, -1, 3) # B, 100, 22, 3
        for k in np.arange(0, seq_len):
            t_3d[k] = np.mean(np.linalg.norm(gt[:, k, :, :].reshape(-1, 3) - out[:, k, :, :].reshape(-1, 3), 2, axis=1))
        return t_3d
    

    the results are larger than those in your paper, I want to know how do you calculate errors in 3D coordinate ? have you scaled the source data?

    3d

    eular

    opened by Droliven 3
Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimation. In CVPR 2022.

Nonuniform-to-Uniform Quantization This repository contains the training code of N2UQ introduced in our CVPR 2022 paper: "Nonuniform-to-Uniform Quanti

Zechun Liu 60 Dec 28, 2022
This repository contains the code for the paper "Hierarchical Motion Understanding via Motion Programs"

Hierarchical Motion Understanding via Motion Programs (CVPR 2021) This repository contains the official implementation of: Hierarchical Motion Underst

Sumith Kulal 40 Dec 5, 2022
Exploring Versatile Prior for Human Motion via Motion Frequency Guidance (3DV2021)

Exploring Versatile Prior for Human Motion via Motion Frequency Guidance This is the codebase for video-based human motion reconstruction in human-mot

Jiachen Xu 5 Jul 14, 2022
Code for the paper Learning the Predictability of the Future

Learning the Predictability of the Future Code from the paper Learning the Predictability of the Future. Website of the project in hyperfuture.cs.colu

Computer Vision Lab at Columbia University 139 Nov 18, 2022
PyTorch code for the paper "FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras"

FIERY This is the PyTorch implementation for inference and training of the future prediction bird's-eye view network as described in: FIERY: Future In

Wayve 406 Dec 24, 2022
SLAMP: Stochastic Latent Appearance and Motion Prediction

SLAMP: Stochastic Latent Appearance and Motion Prediction Official implementation of the paper SLAMP: Stochastic Latent Appearance and Motion Predicti

Kaan Akan 34 Dec 8, 2022
MAU: A Motion-Aware Unit for Video Prediction and Beyond, NeurIPS2021

MAU (NeurIPS2021) Zheng Chang, Xinfeng Zhang, Shanshe Wang, Siwei Ma, Yan Ye, Xinguang Xiang, Wen GAo. Official PyTorch Code for "MAU: A Motion-Aware

ZhengChang 20 Nov 25, 2022
Kaggle Lyft Motion Prediction for Autonomous Vehicles 4th place solution

Lyft Motion Prediction for Autonomous Vehicles Code for the 4th place solution of Lyft Motion Prediction for Autonomous Vehicles on Kaggle. Discussion

null 44 Jun 27, 2022
[arXiv] What-If Motion Prediction for Autonomous Driving ❓🚗💨

WIMP - What If Motion Predictor Reference PyTorch Implementation for What If Motion Prediction [PDF] [Dynamic Visualizations] Setup Requirements The W

William Qi 96 Dec 29, 2022
Waymo motion prediction challenge 2021: 3rd place solution

Waymo motion prediction challenge 2021: 3rd place solution ?? Technical report ??️ Presentation ?? Announcement ??Motion Prediction Channel Website ??

null 158 Jan 8, 2023
Stochastic Scene-Aware Motion Prediction

Stochastic Scene-Aware Motion Prediction [Project Page] [Paper] Description This repository contains the training code for MotionNet and GoalNet of SA

Mohamed Hassan 31 Dec 9, 2022
Multi-Person Extreme Motion Prediction

Multi-Person Extreme Motion Prediction Implementation for paper Wen Guo, Xiaoyu Bie, Xavier Alameda-Pineda, Francesc Moreno-Noguer, Multi-Person Extre

GUO-W 38 Nov 15, 2022
[CVPR 2021] Forecasting the panoptic segmentation of future video frames

Panoptic Segmentation Forecasting Colin Graber, Grace Tsai, Michael Firman, Gabriel Brostow, Alexander Schwing - CVPR 2021 [Link to paper] We propose

Niantic Labs 44 Nov 29, 2022
Using a Seq2Seq RNN architecture via TensorFlow to predict future Bitcoin prices

Recurrent Bitcoin Network A Data Science Thesis Project About This repository contains the source code for implementing Bitcoin price prediciton using

Frizu 6 Sep 8, 2022
Forecasting for knowable future events using Bayesian informative priors (forecasting with judgmental-adjustment).

What is judgyprophet? judgyprophet is a Bayesian forecasting algorithm based on Prophet, that enables forecasting while using information known by the

AstraZeneca 56 Oct 26, 2022
TANL: Structured Prediction as Translation between Augmented Natural Languages

TANL: Structured Prediction as Translation between Augmented Natural Languages Code for the paper "Structured Prediction as Translation between Augmen

null 98 Dec 15, 2022
Code for "Reconstructing 3D Human Pose by Watching Humans in the Mirror", CVPR 2021 oral

Reconstructing 3D Human Pose by Watching Humans in the Mirror Qi Fang*, Qing Shuai*, Junting Dong, Hujun Bao, Xiaowei Zhou CVPR 2021 Oral The videos a

ZJU3DV 178 Dec 13, 2022
Price-Prediction-For-a-Dream-Home - A machine learning based linear regression trained model for house price prediction.

Price-Prediction-For-a-Dream-Home ROADMAP TO THIS LINEAR REGRESSION BASED HOUSE PRICE PREDICTION PREDICTION MODEL Import all the dependencies of the p

DIKSHA DESWAL 1 Dec 29, 2021