BASH - Biomechanical Animated Skinned Human

Related tags

Deep Learning visualization biomechanics animated human-pose-estimation opensim 3d-models statistical-model scape
Overview

BASH - Biomechanical Animated Skinned Human

BASH Teaser

Schleicher, R., Nitschke, M., Martschinke, J., Stamminger, M., Eskofier, B., Klucken, J., Koelewijn, A. (2021). BASH: Biomechanical Animated Skinned Human for Visualization of Kinematics and Muscle Activity. 16th International Conference on Computer Graphics Theory and Applications (GRAPP), 2021.

https://www.scitepress.org/Papers/2021/102106/102106.pdf

BASH Model

Converting a OpenSim [1] format file (.osim + .mot) to the SCAPE [2] framework. Visualization tool to inspect the animated model in 3D.

Processing Pipeline

Input Model: OpenSim

  • Parser
  • Model (.osim)
  • Scale factors (.xml)
  • Motion (.mot)
  • Muscle Activation (.sto)

Baseline model Design for a new Musculoskeltal Model (in Blender)

  • modeling
  • import SCAPE mesh
  • rig and skin skeleton (same hierarchy as musucloskeletal model)
  • place markers (same set as musculoskeletal model)
  • export model (.dae reorders vertices...) => mesh, markers & weights files

Scaling

  • performed automatically, applied correctly to the hierachy, applied in bone space
  • use .xml file or my estimation (defined in settings.h)
  • scaled vs generic in ./data/cache/mesh/

Initial Pose Matching

  • computed automatically using OpenSim's IK solver
  • cached in ./data/cache/mapping for debugging the resulting .mot file

Pose Transformation

  • calculated beforehand (needed the mesh for projection to SCAPE)
  • uses pose mapping projection and kinematic transformations, applied in world space
  • cached in ./data/cache/mesh/

Projection into SCAPE space

  • projection to scape space (only relative rotations)
  • rigid alignment to adjust translation
  • cached in ./data/cache/mesh/

Visualization of Muscle Activation

  • computed at run-time
  • color coding in Fragment Shader

Settings

  • settings.h for keyshortcuts, constants and other configurations

Project structure and dependencies

  • SCAPE: The main Windows-Application that handles the model conversion and visualization

  • External dependencies (minimum required version):

  • SFML (>= 2.5.1)

  • glew (>= 2.1.0)

  • glm (>= 0.9.9.5)

  • Assimp (>= 3.0.0)

  • OpenSim and SimbodyTK (>= 4.0)

  • libRender: A custom framework used for creating a window and render a 3D-application in it

  • External dependencies (minimum required version):

  • SFML (>= 2.5.1)

  • glew (>= 2.1.0)

  • glm (>= 0.9.9.5)

  • libSCAPE: The SCAPE framework to load the SCAPE binary data and create a mesh in pose and shape

  • External dependencies (minimum required version):

  • SuitSparse package: suitsparse + amd + umfpack (>= 5.1.2)

  • GSL (>= 2.4)

SCAPE Framework

  • Implementation in ´SCAPE.h´
  • Model parameters
  • Pose: Rotation vector for each part ('numParts = 16') in three-dimensional twist subvectors (the axis is determined by the vector's direction and the angle is determined by the vector's magnitude.
  • Shape: Scalar PCA coefficients ('numVecs = 20') to modify body proportions like height, size and gender etc.

Building platform x64

  • OpenSim can only be built in 64bit. So we have to use the x64 Platform in order to use their API.
  • Include and link all dependencies in x64.
  • Build the SCAPE framework in x64.
  • Define the flag '#define SAVE_MATRIX 0' once to write new binaries in the correct format (64bit wording).
  • The folder 'data\default_scape_data' should contain the binary files: 'matrixDGrad.bin', 'SCAPE_DGrad_numeric.bin', 'SCAPE_DGrad_symbolic.bin', 'SCAPE_pose.bin'.

Example result

OpenSim's visualization compared to our visualization (data set: straight running [3]): Example

References

[1] Seth, A., Hicks, J. L., Uchida, T. K., Habib, A., Dembia,C. L., Dunne, J. J., Ong, C. F., DeMers, M. S., Ra-jagopal, A., Millard, M., et al. (2018). OpenSim: Sim-ulating musculoskeletal dynamics and neuromuscularcontrol to study human and animal movement. PLoSComputational Biology, 14(7):1–20.

[2] Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers,J., and Davis, J. (2005). SCAPE: Shape Completionand Animation of People. InACM Transactions onGraphics, volume 24, pages 408–416.

[3] Nitschke, M., Dorschky, E., Heinrich, D., Schlarb, H., Eskofier, B. M., Koelewijn, A. D., and Van den Bogert, A. J. (2020). Efficient trajectory optimization for curved running using a 3D musculoskeletal model with implicit dynamics. Scientific Reports, 10(17655).

Comments
  • How to define the markers set in SCAPE?

    How to define the markers set in SCAPE?

    Hello, I am learning about OpenSim and SMPL model. I read another issue markers.obj.

    We placed the markers on the surface of the baseline model's mesh by hand and exported their coordinates

    Explore the model in Blender, find the positions on the skin to place the markers that share the same name in your osim model? How to evaluate the markers in Blender? Thank you.

    opened by hwy1992129 5
  • markers.obj

    markers.obj

    Hello,

    I am trying to use your package with some of my data. I managed to build everything on windows (I think), and I am now loading my data. I get the following error: Error: No corresponding marker was found: AnkL @ line 180 in file: c:\users\u0101727\documents\myrepositories\bash-model\src\model.cpp, which is expected since there is no such marker in my model. My question is therefore the following: can I use a different marker set with your package? I see that the markers of the baseline model are defined in markers.obj, so I could potentially adjust this file. But I don't know where the values come from. I can also try to adjust my marker set to reflect what is in markers.obj. Any thoughts on that?

    I have also tried loading the data provided with Nitschke et al. (2020), and I think it worked fine. At least I get a lot of .ply files in cache\mesh\.... Is it what it is supposed to output? How are you then converting those files into animation as in your example? Sorry for the basic questions, but it is not super clear to me how to use the tool and what to expect as output.

    Thanks in advance for your help, I hope I can make it work - it looks great!

    opened by antoinefalisse 4
  • Can't load sto from 2020_Nitschke_RunMaD.zip.... and some questions on the result..

    Can't load sto from 2020_Nitschke_RunMaD.zip.... and some questions on the result..

    Hi, Finally, I let BASH-Model run in my pc. Since it has an error, as mentioned in my previous issue, I modify that line into OpenSim::TimeSeriesTable stoTable = OpenSim::TimeSeriesTable(filepath); I downloaded 2020_Nitschke_RunMaD.zip to test the program. I can't load any sto to run it... I only can run the osim with mot file....

    Besides, without the sto, I got the result below. image There are some cyan color lines near the arms. Do they represent the arms from the OpenSim skeleton (runMaD.osim)? Does it mean the skeleton can't be under the SCAPE surface model? image

    opened by hwy1992129 3
  • Need help for the installation........

    Need help for the installation........

    Hello, For the dependencies, SuitSparse package: suitsparse + amd + umfpack (>= 5.1.2). The official SuitSparse doesn't support Windows well. Are you using the unoffical SuitSparse for windows version? There is a "libmwlapack.lib" I failed to get it... Does it come from the matlab version of lapack? I am using the SDK from OpenSim 4.3 binary version. There is an error in the code in osim.cpp.... Severity Code Description Project File Line Suppression State Error C2440 'initializing': cannot convert from 'OpenSim::DataAdapter::OutputTables' to 'OpenSim::TimeSeriesTable_<SimTK::Real>' SCAPE E:\innovision2021\BASH-Model-main\BASH-Model-main\src\osim.cpp 94

    opened by hwy1992129 2
  • Issues on the versions of reading sto and the sto files in 2020_Nitschke_RunMaD.zip

    Issues on the versions of reading sto and the sto files in 2020_Nitschke_RunMaD.zip

    I finally let the program work with Muscle Activation. It has an errorif you are not using OpenSim 4.0 API. Change the code to OpenSim::TimeSeriesTable stoTable = OpenSim::TimeSeriesTable(filepath); may fix the error in OpenSim 4.1+ API. The *.sto files from 2020_Nitschke_RunMaD.zip don't work in the program. The reason behind it may be the problem of format.....??? I open the *.sto using excel. The *.sto file from 2020_Nitschke_RunMaD.zip look like the following figure. image I modified the *.sto file and the program works well with Muscle Activation... image image

    opened by hwy1992129 1
  • gsl.lib windows

    gsl.lib windows

    Hello,

    I'd like to test your tool but I'm struggling to get things built on windows. I was just wondering if that's something you tried out and, if so, you found an easy way to get gsl.lib? Your tool looks really great so I am really willing to give it a try.

    Thanks

    opened by antoinefalisse 1
  • How to obtain the osim file?

    How to obtain the osim file?

    Thank you very much for your perfect work. I'm new to this field, so I want to ask some basic questions. How to obtain the visualization like the gif? When I run the start.bat on windows, it said that I need to drag the .osim file. I know the .osim file is used for OpenSim, but is there any ready-made .osim file for visualization? In details, How to obtain the osim file by ourselves?

    Thank you very much.

    opened by Xianjin111 1
  • FOV in deg instead of rad.

    FOV in deg instead of rad.

    Hi,

    It is not an issue per se, but I think the FOV set here is in degrees, whereas glm::perspective expects a FOV in radian here. Interestingly, it does not have a huge effect on the rendering. Yet the values of the projection matrix P are wrong should anyone would like to use them. Thought that could be useful.

    Best, Antoine

    opened by antoinefalisse 1
Owner
Machine Learning and Data Analytics Lab FAU
Public projects of the Machine Learning and Data Analytics Lab at the Friedrich-Alexander-University Erlangen-Nürnberg
Machine Learning and Data Analytics Lab FAU
GitHub
FwordCTF 2021 Infrastructure and Source code of Web/Bash challenges

FwordCTF 2021 You can find here the source code of the challenges I wrote (Web and Bash) in FwordCTF 2021 and the source code of the platform with our

Kahla 5 Nov 25, 2022
[CVPR2021] UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicles

UAV-Human Official repository for CVPR2021: UAV-Human: A Large Benchmark for Human Behavior Understanding with Unmanned Aerial Vehicle Paper arXiv Res

null 129 Jan 4, 2023
Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors, CVPR 2021

Human POSEitioning System (HPS): 3D Human Pose Estimation and Self-localization in Large Scenes from Body-Mounted Sensors Human POSEitioning System (H

Aymen Mir 66 Dec 21, 2022
Human Action Controller - A human action controller running on different platforms.

Human Action Controller (HAC) Goal A human action controller running on different platforms. Fun Easy-to-use Accurate Anywhere Fun Examples Mouse Cont

null 27 Jul 20, 2022
Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Python scripts for performing 3D human pose estimation using the Mobile Human Pose model in ONNX.

Ibai Gorordo 99 Dec 31, 2022
StyleGAN-Human: A Data-Centric Odyssey of Human Generation

StyleGAN-Human: A Data-Centric Odyssey of Human Generation Abstract: Unconditional human image generation is an important task in vision and graphics,

stylegan-human 762 Jan 8, 2023
The project is an official implementation of our CVPR2019 paper "Deep High-Resolution Representation Learning for Human Pose Estimation"

Deep High-Resolution Representation Learning for Human Pose Estimation (CVPR 2019) News [2020/07/05] A very nice blog from Towards Data Science introd

Leo Xiao 3.9k Jan 5, 2023
PaddleRobotics is an open-source algorithm library for robots based on Paddle, including open-source parts such as human-robot interaction, complex motion control, environment perception, SLAM positioning, and navigation.

简体中文 | English PaddleRobotics paddleRobotics是基于paddle的机器人开源算法库集，包括人机交互、复杂运动控制、环境感知、slam定位导航等开源算法部分。 人机交互 主动多模交互技术TFVT-HRI 主动多模交互技术是通过视觉、语音、触摸传感器等输入机器人

null 185 Dec 26, 2022
EasyMocap is an open-source toolbox for markerless human motion capture from RGB videos.

EasyMocap is an open-source toolbox for markerless human motion capture from RGB videos. In this project, we provide the basic code for fitt

ZJU3DV 2.2k Jan 5, 2023
Human head pose estimation using Keras over TensorFlow.

RealHePoNet: a robust single-stage ConvNet for head pose estimation in the wild.

Rafael Berral Soler 71 Jan 5, 2023
Repo for CVPR2021 paper "QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information"

QPIC: Query-Based Pairwise Human-Object Interaction Detection with Image-Wide Contextual Information by Masato Tamura, Hiroki Ohashi, and Tomoaki Yosh

null 105 Dec 23, 2022
Deep Dual Consecutive Network for Human Pose Estimation (CVPR2021)

Deep Dual Consecutive Network for Human Pose Estimation （CVPR2021） Introduction This is the official code of Deep Dual Consecutive Network for Human P

null 295 Dec 29, 2022
Bottom-up Human Pose Estimation

Introduction This is the official code of Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation. This paper has been accepted to CVPR2

null 108 Dec 1, 2022
Code for "3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop"

PyMAF This repository contains the code for the following paper: 3D Human Pose and Shape Regression with Pyramidal Mesh Alignment Feedback Loop Hongwe

Hongwen Zhang 450 Dec 28, 2022
《Unsupervised 3D Human Pose Representation with Viewpoint and Pose Disentanglement》(ECCV 2020) GitHub: [fig9]

Unsupervised 3D Human Pose Representation [Paper] The implementation of our paper Unsupervised 3D Human Pose Representation with Viewpoint and Pose Di

null 42 Nov 24, 2022
Repository for the paper "PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation", CVPR 2021.

PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation Code repository for the paper: PoseAug: A Differentiable Pose Augme

Pyjcsx 328 Dec 17, 2022
Source code for the GPT-2 story generation models in the EMNLP 2020 paper "STORIUM: A Dataset and Evaluation Platform for Human-in-the-Loop Story Generation"

Storium GPT-2 Models This is the official repository for the GPT-2 models described in the EMNLP 2020 paper [STORIUM: A Dataset and Evaluation Platfor

Nader Akoury 27 Dec 20, 2022
This is an official implementation of our CVPR 2021 paper "Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression" (https://arxiv.org/abs/2104.02300)

Bottom-Up Human Pose Estimation Via Disentangled Keypoint Regression Introduction In this paper, we are interested in the bottom-up paradigm of estima

HRNet 367 Dec 27, 2022
Official PyTorch implementation of "IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videos", CVPRW 2021

IntegralAction: Pose-driven Feature Integration for Robust Human Action Recognition in Videos Introduction This repo is official PyTorch implementatio

Gyeongsik Moon 29 Sep 24, 2022