Hello, I compiled the Android version, found that can not be compiled successfully, lack of Library dependency, can you provide the dependent source code? Since last year, I found your paper is waiting for your demo. Until recently, I saw that you posted it to GitHub. I am very interested in the application of AI in game, but I don't know much about neural network, so I can learn from your demo first.

I use the "D1_001_KAN01_001.bvh" (only the one) to train the model. This is my Motion Editor setting I want to test whether the Motion Exporter output data can be correct training. However, i failed. Can you help find the problem? This is my scene. This is upside down.

There are 3 questions about SIGGRAPH 2017 paper:

1. How do I create my own training data, given that I have VR devices like leg bands, hand controller, and headset?
2. Is it possible to include hand and head input from VR devices into the model too (instead of only Up/Down/Left/Right input)?
3. What is the training set size for the model provided in this repository?

I will need to read more about the paper. Any advice is appreciated. Thank you!

How can I let the AI move the character using spline waypoints like the one you showed at the end of your video ? Do you have any Slack group or Gitter or something like that ??

Hi! I am new to unity and this project. And I got one problem for your help : ( After I imported BVH file using Data Processing/BVH importer in the demo_adam (and demo_original) , and I wanted to export the BVH file to input.txt and output.txt as the training data. The Motion Exporter just shows that "no motion editor found in scene". I have no clue to solve this out. Is there missing scene.unity file in this project?

Hello, I even tried to directly built and ran project of unity in SIGGRAPH_2017 without editing. But in the program have built, the character doesn't respond to my input. And one more question, how could I use the weights I trained in PFNN to unity project? Thank you so much. By the way, I really appreciate your work in AI4Animation. Hope you have a nice day : )

Hi, I tried to convert the raw motion capture data in MANN into the format for neural nerwork input, using the data processing scripts you provided in Unity (BVHImporter, MotionExporter etc.). However, the resulted Input.txt and Output.txt does not align with the ones you provided, both in trajectory and bone parameter fields. Training MANN with my own txts also displayed weird results, proving that the generated files are wrong.

I looked through the code and everything seems to be right. I haven't alter any of the code you released, and only clicked "Export" botton and added Trajectory Module for each clip in the Motion Editor panel (leaving alone the Style Module because the annotation of styles does not affect other parameters).

Do you have any idea where things might go wrong?

Thank you! :)

Hello, I saw a video about this project and would like to try it, but found no instruction on how to do it.

I assume it is played in Unity game engine but I tried "Adding" the "Adam" folder in the Projects screen but it said that it was an invalid path.

Can you please provide some instructions?

With regards, John

Hi! I tried to check your Demo project but after I followed your instructions and stored the network parameters the wolf character is not moving if a press the control buttons. I am new to Unity, maybe it is a trivial one. Do you have any suggestions? (Windows 10, Unity 2018.2.0f2)

Hi, thanks for your awesome work! I noticed that there is neither pretrained weights nor training data for the Adam model in PFNN. Just want to confirm whether these data are unreleased, in case I missed them somewhere.

Not sure what's going on here, but I've managed to... well, mangle the dog. Or make him dance, not entirely sure.

1. Make the dog fully sit (hold V for a few seconds)
2. Release V
3. Wait 2 seconds
4. Tap V
5. Repeat steps 3 and 4 repeatedly.

Is this an issue with the network, or just the Unity/demo code?

Hi Sebastian,

Thank you for posting this amazing work!

I am trying to reproduce the results from scratch (the raw .bvh files -> asset data -> input.txt/output.txt for training). But the results I reproduced have some problems：

1. The agent cannot turn left or right when running, I mean when I press Shift+W+E/Q at the same time, the action of agent will crash;
2. The posture of the agent is unnatural when sitting down, and there is always an offset from the marked contacts point; And when I directly export your data as training data pressing Export Data button in MotionExporter, the results don't have these problems.

This is my data processing:

1. Use BVH Importer to import .bvh files which comes from MotionCapture.zip. When importing some .bvh files, the Flip option is checked, such as Jump, RunTurn, WalkTurn etc.. The data is saved in Assets/MotionCapture_reproduce/;
2. Copy the *.unity files in Assets/MotionCapture/ and overwrite the *.unity files in Assets/MotionCapture_reproduce/. I found that many actions need to create objects by myself, such as Amchair, Avoid, Sit etc., which would be time-consuming to do manually, so I directly copied them;
3. Click the Import option in the figure below, and in the 'public void Import()' in MotionEditor.cs, I added the following code to import the Modules, Sequences, Export, Framerate, MirrorAxis, and Offset parameters from Assets/MotionCapture/ and copied to the data in Assets/MotionCapture_reproduce/;
4. Use the MotionExporter in the unity scene to export my own data;

At the same time, I also noticed that the data in Assets/MotionCapture/ is far more than the .bvh files from MotionCapture.zip, but I don't know how the extra data came from, so I only used the data in MotionCapture.zip to reproduce the results;

So, is my data processing missing something? Or which step is wrong? Where does the extra data in Assets/MotionCapture/ come from? Can you elaborate your data processing from the raw .bvh files to asset data?

Need your help. Thanks a lot!

Hello, Thank you for your great work!

Can you detail how to extract the velocity? Or the code? It seems to be the acceleration that is extracted in the readme?

This code is included with a comment saying it is for 60hz (BioAnimation_Adam.cs):

//Trajectory for 60 Hz framerate
private const int Framerate = 60;
private const int Points = 111;
private const int PointSamples = 12;
private const int PastPoints = 60;
private const int FuturePoints = 50;
private const int RootPointIndex = 60;
private const int PointDensity = 10;

Is it possible to change these values to have it run at 30hz? If so, aside from changing the 'Framerate' variable to 30, what other values need to be changed?

Hello Sebastian,

I''ve sent you email twice but have not recieved any response. I am not sure if there is any connection problem and I try to connect you via github issues.

I found there might be two mistakes in the paper.

1. In Eq. 5, the range of S should be (-pi, pi), so I think S should be out of the innermost scope: $A\cdot sin(2pi(F\cdot Tau-S)+B (5) -> A\cdot sin(2pi(F\cdot Tau)-S)+B (5.1) $, or divided by 2pi before passing it in Eq. (5). I found it hard to predict S to be similar to the curve in fig. 3 using Eq. 5 directly. I tested Eq. (5) and Eq. (5.1) and plotted the figures after training for 5 epoches. Eq 5: Eq 5.1:

2. In Eq 9, the next phase is calculated by interpolating two phases, and multiply it by $A_{t+dt}$: However, the phase $P_t$ and $P_{t+dt}$ also contains the information about the amplitudes, so the expected prediction of $A_{t+dt}$ should be very similar to one, which means scale the interpolated result. I am not sure if the amplitudes should be predicted by the difference between two frames, just like the frequency does?

Besides, I am not sure how to handle the loss of the phase. I employ the mse loss between the predicted phase in eq 9 and the groundtruth phase, but the frequency of the predicted phase is inaccurate. Do we have to employ loss on the amplitudes and frequence additionally? Which method do you use?

I will appreciate it if you correct me if I am wrong. I look forward to hearing from you.

Best regards,

Xiangjun Tang

Hi, Sebastian

How are you? We are following closely with your research work on applying Deep Learning onto character animation, and I want to say they are great work! We are reading your Siggraph 2022 paper "DeepPhase: Periodic Autoencoders for Learning Motion Phase Manifolds" and trying to reproduce the work, but got stuck on some questions. I am wondering if you could help me with these detailed questions.

1. What's the kernel-size of the convolultional layer?
2. What method did you use to initialize the weights?
3. What are the validation/test loss you achieved after you finished training?
4. If I change the kernal size, there are quite a few occations that loss became nan，do you know what could be the reason for this?
5. In the paper, does every channel connect to a unique fully connected layer? What's the activation function of the fully connected layer?
6. Does the FFT layer has weights to learn as well?
7. The sampling time for a time window is 2 second, correct? Also the T in "f" in formula (3) is also 2 seconds, right?

We used your dataset from paper "Neural State Machine for Character-Scene Interactions",but the lowest loss we could get is 0.2. We think it is too high and don't find a way to reduce it. Can you shed some light on this?

Avoid 18863（5.24min） Carry 53094（14.75min） Crouch 7659 （2.13min） Door 58479（16.24min） Jump 4511 （1.25min） Loco 59859（16.63min） Sit 199472（55.41min） total: 401937 (111min)

Thanks a lot!