Excuse me I have a question in the docs capertar in the script.js file can't export any functions or new functions within that script.js file, when I want to use in another js file, the question is why is it? can't you export those functions from that script.js file?

Improvement of Leg and Arm Rotation

Hi Rich, It took some time and I am not 100% satisfied with this solution, but I think it is an improvement. What has changed:

• Types for left and right
• Unit test for upper and lower legs
• Alternative calculation for the leg rotation
• Alternative calculation for the arm rotation
• Introduction of a Euler Class to keep track of the rotation order (especially import for the arm rotations)
• A little code cleaning
• The use of constants for PI and TWO_PI

I'm looking forward to your feedback!

Hi, thanks for your awesome project!

I am wondering whether it's possible to calling the kalidokit face solver from a node.js service.

The story: a windows executable(exe) send an image to a nodejs web service, which runs mediapipe and kalidokit

Can you give me some suggestions?

Thanks!

I don't know if this is 100% correct, but it seems the brow landmark point indices are mirrored, or at least do not match the pupil side, which messes up the pupils solve because the brows are the bounding box of the pupils.

I don't know if it matters but I'm using holistic latest model.

In my testing I found the pupils math is completely messed up because of this bug -- it is "corrected" by some additional averaging of both eyes, which "fixes" the error but adds a lot of jitter to the pupil tracking. https://github.com/yeemachine/kalidokit/blob/73f202663c32673f7e21c8869bf02f81223bc340/src/FaceSolver/calcEyes.js#L97

Add yarn.lock file when using yarn to install dependencies and fix the error, showing "Cannot find type definition file for 'node'." message when yarn tsc -p . building. Error messages shown below:

> [email protected] build:lib
> tsc -p .

../../node_modules/@types/plist/index.d.ts:7:23 - error TS2688: Cannot find type definition file for 'node'.

7 /// <reference types="node" />
                        ~~~~

../../node_modules/@types/plist/index.d.ts:21:10 - error TS2580: Cannot find name 'Buffer'. Do you need to install type definitions for node? Try `npm i --save-dev @types/node`.

21   Date | Buffer | PlistObject | PlistArray;
            ~~~~~~

../../node_modules/@types/responselike/index.d.ts:6:23 - error TS2688: Cannot find type definition file for 'node'.

6 /// <reference types="node" />
                        ~~~~

../../node_modules/@types/responselike/index.d.ts:8:33 - error TS2307: Cannot find module 'http' or its corresponding type declarations.

8 import { IncomingMessage } from 'http';
                                  ~~~~~~

../../node_modules/@types/responselike/index.d.ts:9:24 - error TS2307: Cannot find module 'stream' or its corresponding type declarations.

9 import { Stream } from 'stream';
                         ~~~~~~~~

../../node_modules/@types/responselike/index.d.ts:19:11 - error TS2580: Cannot find name 'Buffer'. Do you need to install type definitions for node? Try `npm i --save-dev @types/node`.

19     body: Buffer;
             ~~~~~~

../../node_modules/@types/responselike/index.d.ts:31:15 - error TS2580: Cannot find name 'Buffer'. Do you need to install type definitions for node? Try `npm i --save-dev @types/node`.

31         body: Buffer,
                 ~~~~~~

../../node_modules/xmlbuilder/typings/index.d.ts:2:26 - error TS2307: Cannot find module 'stream' or its corresponding type declarations.

2 import { Writable } from 'stream';
                           ~~~~~~~~


Found 8 errors.

error Command failed with exit code 2.```

when i open the site (selfhosted) i get the error: Failed to acquire camera feed: AbortError: Starting videoinput failed I tested it with Firefox. In chrome i get the error: Failed to acquire camera feed: NotReadableError: Could not start video source

I install kalidokit by npm. And can not find types for it.

But i clone this repo and build it myself. I find inde.d.ts, etc. (by tsc)

Did you forget to re-release the new version?

currently kalidokit package has following hard dependencies:

    "dependencies": {
        "@mediapipe/camera_utils": "^0.3.1632432234",
        "@mediapipe/drawing_utils": "^0.3.1620248257",
        "@mediapipe/holistic": "^0.5.1635989137",
        "@pixiv/three-vrm": "^0.6.7",
        "three": "^0.133.0"
    },

which means they will all be installed when installing kalidokit which makes it at least 100x bigger than it should be

and in reality, all of those dependencies are needed only for demos and not needed at all for actual solver. please remove hard dependencies - you can make them optional instead

for example, i prefer to use my own implementation of ml processing (using different mediapipe models and using custom processing instead of mediapipe official modules) and using babylonjs instead of threejs

Is there any particular reason to use MediaPipe? Why not use another method like VIBE, which directly gives output in the format of 3D rotation data for joints (instead of 3D Cartesian coordinates like MediaPipe). This way, there wouldn't be the need to perform the complex kinematic calculations which is the majority of the code of this kalidokit library.

Hi ~ Thanks for your great work ! i have encounter some problems when reading your source code. The main confusion is that the output landmarks from mediapipe FaceMesh is in Screen Coordinates as mention at face_mesh#face-transform-module, but you have use these landmarks to calculate Euler angles in your rollPitchYaw method. The pipeline for FaceSolver is look like:

scree coords 3D landmarks -> multiply imagesize(still screen coords) -> calcHead() -> rollPitchYaw() use scree coords

The landmarks detected by mediapipe is not in World Space nor Model Space, I just can not understand the calculation logic about the pipeline above. Can you tell me why this is valid? It will make a great help to me, many thanks ~

When running npm run test or npm run dev to start up the demo projects in the docs folder, vite fails to start the dev server, claiming a missing dependency in the Live2D folder.

Despite the error claiming that script.js could not be resolved, I've verified that the script.js file is present in docs/live2d, and I haven't touched it or index.html in any way. The only change I've made so far is removing the pre-build script from package.json to workaround #19

Platform: Windows 10

Steps to reproduce:

1. Clone Kalidokit repository
2. Remove prebuild script from package.json
3. Run "npm install"
4. Run "npm run test"

Full error output:

C:\Projects\Programming\kalidokit>npm run test

> [email protected] test C:\Projects\Programming\kalidokit
> npm run build && npm run dev


> [email protected] build C:\Projects\Programming\kalidokit
> vite build && npm run build:lib

vite v2.6.0 building for production...
Unknown input options: exports. Allowed options: acorn, acornInjectPlugins, cache, context, experimentalCacheExpiry, external, inlineDynamicImports, input, makeAbsoluteExternalsRelative, manualChunks, maxParallelFileReads, moduleContext, 
onwarn, perf, plugins, preserveEntrySignatures, preserveModules, preserveSymlinks, shimMissingExports, strictDeprecations, treeshake, watch
✓ 13 modules transformed.
dist/kalidokit.es.js   31.87 KiB / gzip: 6.99 KiB
dist/kalidokit.umd.js   16.10 KiB / gzip: 5.38 KiB

> [email protected] build:lib C:\Projects\Programming\kalidokit
> tsc -p .


> [email protected] dev C:\Projects\Programming\kalidokit
> vite docs


  vite v2.6.0 dev server running at:

error when starting dev server:
Error: The following dependencies are imported but could not be resolved:

  script.js (imported by C:/Projects/Programming/kalidokit/docs/live2d/index.html)

Are they installed?
    at optimizeDeps (C:\Projects\Programming\kalidokit\node_modules\vite\dist\node\chunks\dep-a8e01c9f.js:71146:15)     
    at processTicksAndRejections (internal/process/task_queues.js:93:5)
    at async runOptimize (C:\Projects\Programming\kalidokit\node_modules\vite\dist\node\chunks\dep-a8e01c9f.js:74997:48)
    at async Server.httpServer.listen (C:\Projects\Programming\kalidokit\node_modules\vite\dist\node\chunks\dep-a8e01c9f.js:75013:21)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] dev: `vite docs`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] dev script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

Hello, Thanks for your component that fill the gap between MediaPipe and Avatar Puppetering.

I made a small JSFiddle using ReadyPlayer Me Avatar with MediaPipe and Kalidokit https://jsfiddle.net/c0gxfk9y/1/

I only setup the Kalidokit.Pose.solve() I don't know if I correctly apply the pose Rig ? The animations seems OK but the arms are upside down.

I didn't apply the position/rotation on hips, because the avatar is moved anywhere. Next I'll add hand pose.

Can you point me in the right direction ? I feel close to something that would correctly do the job.

https://github.com/yeemachine/kalidokit/blob/c720d483df867996be96e2c3218e963da9190ecf/src/FaceSolver/calcEyes.ts#L244-L277

I have written a simple python version of your calEyes.ts and combine it with mediapipe's output directly. However, here in L260, the ratio is calculated as:browRatio = browDistance / maxBrowRatio - 1;. And browDistance is always smaller than maxBrowRatio, which means thatbrowRatiowill always be a negative number. This will cause that browRaiseRatio will always be 0 after clamp.

Is this a bug?

Update: I found out the this was due to different video input. When my face is close to the camera, the above problem will happen. However when I get to a middle distance (half body in the image). This will not happen. I don't know what caused this.

So the problem now is: browDistance will change over different video input. Do you have any clues on how to solve thiS? It is very weird that browDistance = eyeLidAvg / eyeWidth will have huge changes because the ratio should be the same for the same person. Should I use a cropped image for facial landmark extraction?

Hi.

I am trying to run kalidokit\docs\script.js in VSCode. Running into the following error after "npm install kalidokit" Any advice on this?

[Running] node "c:\Users\USER\kalidokit\docs\script.js" (node:11004) Warning: To load an ES module, set "type": "module" in the package.json or use the .mjs extension. (Usenode --trace-warnings ...` to show where the warning was created) c:\Users\USER\kalidokit\docs\script.js:1 import * as Kalidokit from "../dist"; ^^^^^^

SyntaxError: Cannot use import statement outside a module at Object.compileFunction (node:vm:360:18) at wrapSafe (node:internal/modules/cjs/loader:1084:15) at Module._compile (node:internal/modules/cjs/loader:1119:27) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1209:10) at Module.load (node:internal/modules/cjs/loader:1033:32) at Function.Module._load (node:internal/modules/cjs/loader:868:12) at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:81:12) at node:internal/main/run_main_module:22:47`

is there any way to load bvh or fbx file ? and , is there some middlewear to save the face & body animation and drive the vrm asynchronously