Zoom , GoogleMeets에서 Vtuber 데뷔하기

Overview

EasyVtuber

  • Facial landmark와 GAN을 이용한 Character Face Generation
  • Google Meets, Zoom 등에서 자신만의 웹툰, 만화 캐릭터로 대화해보세요!
  • 악세사리는 어느정도 추가해도 잘 작동해요!
  • 안타깝게도 RTX 2070 미만에서는 실시간으로 잘 작동하지 않을 수도 있어요 ㅠㅠ



Demo



Requirements

  • Python >= 3.8
  • Pytorch >= 1.7
  • pyvirtualcam
  • mediapipe
  • opencv-python



Quick Start

  • ※ 이 프로젝트는 사용 전 OBS 설치가 필수입니다
  • 아래 설치 순서를 지켜주세요!
  1. OBS studio 설치

    • OBS virtualcam을 사용하기 위해서 먼저 OBS Studio를 설치해야합니다
  2. pip install -r requirements.txt

    • OBS virtualcam을 설치되어있어야 requirements에 포함된 pyvirtualcam이 정상적으로 설치되어 사용할 수 있습니다
  3. pretrianed model download

    • 아래 파일들을 pretrained folder에 넣어주세요
      • combiner.pt
      • eyebrow_decomposer.pt
      • eyebrow_morphing_combiner.pt
      • face_morpher.pt
      • two_algo_face_rotator.pt
  4. character image를 character folder에 넣어주세요

    • character image 파일은 다음의 조건을 충족해야합니다
      • alpha 채널을 포함할 것(png 확장자)
      • 1명의 인간형 캐릭터일 것
      • 캐릭터가 정면을 볼 것
      • 캐릭터의 머리가 128 x 128 pixel 내에 들어올 것 (기본적으로 256 x 256으로 resize되기 때문에 256 x 256 기준 128x128 안에 들어와야함)

    Example image is refenced by TalkingHeadAnime2

5.python main.py --webcam_output

  • 실제 facial feature가 어떻게 잡히는지 보고 싶다면 --debug 옵션을 추가하여 실행해주세요



How to make Custom Character

  1. 네이버, 구글 등에서 본인이 원하는 캐릭터를 찾으세요!
    • 되도록이면 위의 4가지 조건을 맞춰주세요! google search

  2. 찾은 이미지에서 캐릭터 얼굴이 중앙으로 가도록 가로세로 1:1 비율로 이미지를 잘라주세요!
  3. 이미지 배경을 제거해서 alpha 채널을 만들어 주세요!
  4. 완성!
    • character folder에 이미지를 넣고 python main.py --output_webcam --character (.png_제외한_캐릭터파일_이름) 실행!



Folder Structure

      │
      ├── character/ - character images 
      ├── pretrained/ - save pretrained models 
      ├── tha2/ - Talking Head Anime2 Library source files 
      ├── facial_points.py - facial feature point constants
      ├── main.py - main script to excute
      ├── models.py - GAN models defined
      ├── pose.py - process facial landmark to pose vector
      └── utils.py - util fuctions for pre/postprocessing image



Usage

webcam으로 송출 시

  • python main.py --output_webcam

캐릭터 지정

  • python main.py --character (character folder에 있는 .png를 제외한 캐릭터 파일 이름)

facial feature 확인 시

  • python main.py --debug

동영상 파일 inference

  • python main.py --input video파일_경로 --output_dir frame_저장할_디렉토리



TODOs

  • Add eyebrow feature
  • Parameter Controller GUI
  • Automation of Making Drivable Character



Thanks to



Acknowledgements

  • EasyVtuber는 TalkingHeadAnime2를 기반으로 제작되었습니다.
  • tha2 folder의 source와 pretrained model file은 원저작자 repo의 Liscense를 확인하고 사용하시기 바랍니다.
You might also like...
Very Simple Zoom Spam Pinger!

Very Simple Zoom Spam Pinger!

Analyse the limit order book in seconds. Zoom to tick level or get yourself an overview of the trading day.
Analyse the limit order book in seconds. Zoom to tick level or get yourself an overview of the trading day.

Analyse the limit order book in seconds. Zoom to tick level or get yourself an overview of the trading day. Correlate the market activity with the Apple Keynote presentations.

This application is the basic of automated online-class-joiner(for YıldızEdu) within the right time. Gets the ZOOM link by scheduled date and time.
This application is the basic of automated online-class-joiner(for YıldızEdu) within the right time. Gets the ZOOM link by scheduled date and time.

This application is the basic of automated online-class-joiner(for YıldızEdu) within the right time. Gets the ZOOM link by scheduled date and time.

Quick insights from Zoom meeting transcripts using Graph + NLP
Quick insights from Zoom meeting transcripts using Graph + NLP

Transcript Analysis - Graph + NLP This program extracts insights from Zoom Meeting Transcripts (.vtt) using TigerGraph and NLTK. In order to run this

A python script that automatically joins a zoom meeting based on your timetable.

Zoom Automation A python script that automatically joins a zoom meeting based on your timetable. What does it do? It performs the following processes:

Auto Join Zoom Meeting
Auto Join Zoom Meeting

Auto-Join-Zoom-Meeting Join a zoom meeting with out filling in meeting id's or passcodes, one button for it all! Setup See attached excel document. MA

This program automatically logs you into a Zoom session at your alloted time
This program automatically logs you into a Zoom session at your alloted time

This program automatically logs you into a Zoom session at your alloted time. Optionally you can choose to have end the session at your allotted time.

A python bot to move your mouse every few seconds to appear active on Skype, Teams or Zoom as you go AFK. 🐭 🤖
A python bot to move your mouse every few seconds to appear active on Skype, Teams or Zoom as you go AFK. 🐭 🤖

PyMouseBot If you're from GT and annoyed with SGVPN idle timeouts while working on development laptop, You might find this useful. A python cli bot to

Comments
  • 런타임에러 문의

    런타임에러 문의

    안녕하세요? 해당 프로젝트 다운받아 그대로 돌렸으나 런타임이 발생하여 문의 드립니다.

    RuntimeError: grid_sampler(): expected input and grid to have same dtype, but input has struct c10::Half and grid has float

    에러가 발생되는 부분은 main.py의 model 부분이고 특이한점은 카메라에 얼굴이 없다면 프로그램 자체가 멈춰버리는것으로 확인되고 있습니다. output_image = model(input_image, mouth_eye_vector, pose_vector)

    좋은 프로젝트 감사드리고 혹시 아시는것이 있다면 답장 부탁드리겠습니다 = ] 감사합니다.

    opened by jyoonab 3
  • Mediapipe is not working with python 3.8.2

    Mediapipe is not working with python 3.8.2

    Hi, I tried to install media pipe with pip install mediapipe with python 3.8.2.

    I got an error that " Could not find a version that satisfies the requirement mediapie (from versions: ) No matching distribution found for mediapie "

    Do you have any idea how to get it to install?

    BR, Than

    opened by thanisornsr 2
Owner
Gunwoo Han
EE student / Deep Learning / Computer Vision / Manager of 파이썬 처음처럼
Gunwoo Han
Virtual Zoom Gesture using OpenCV

Virtual_Zoom_Gesture I have created a virtual zoom gesture where we can Zoom in and Zoom out any image and even we can move that image anywhere on the

Mudit Sinha 2 Dec 26, 2021
A real-time dolly zoom camera effect

Dolly-Zoom I've always been amazed by the gradual perspective change of dolly zoom, and I have some experience in python and OpenCV, so I decided to c

Dylan Kai Lau 52 Dec 8, 2022
Avatarify Python - Avatars for Zoom, Skype and other video-conferencing apps.

Avatarify Python - Avatars for Zoom, Skype and other video-conferencing apps.

Ali Aliev 15.3k Jan 5, 2023
Materi workshop "Light up your Python!" Himpunan Mahasiswa Sistem Informasi Fakultas Ilmu Komputer Universitas Singaperbangsa Karawang, 4 September 2021 (Online via Zoom).

Workshop Python UNSIKA 2021 Materi workshop "Light up your Python!" Himpunan Mahasiswa Sistem Informasi Fakultas Ilmu Komputer Universitas Singaperban

Eka Putra 20 Mar 24, 2022
This is a project to detect gestures to zoom in or out, using the real-time distance between the index finger and the thumb. It's based on OpenCV and Mediapipe.

Pinch-zoom This is a python project based on real-time hand-gesture detection, to zoom in or out, using the distance between the index finger and the

Harshit Bhalla 6 Jul 11, 2022
Virtual Zoom Gesture using OpenCV

Virtual_Zoom_Gesture I have created a virtual zoom gesture where we can Zoom in and Zoom out any image and even we can move that image anywhere on the

Mudit Sinha 2 Dec 26, 2021
Python program to start your zoom meetings

zoomstarter Python programm to start your zoom meetings More about Initially this was a bash script for starting zoom meetings, but as i started devel

Viktor Cvetanovic 2 Nov 24, 2021
Clip Bing Maps backgound as RGB geotif image using center-point from vector data of a shapefile and Bing Maps zoom

Clip Bing Maps backgound as RGB geotif image using center-point from vector data of a shapefile and Bing Maps zoom. Also, rasterize shapefile vectors as corresponding label image.

Gounari Olympia 2 Nov 22, 2021
Herramienta para poder automatizar reuniones en Zoom.

Crear Reunión Zoom con Python Herramienta para poder automatizar reuniones en Zoom. Librerías Requeridas Nombre Comando PyAutoGui pip install pyautogu

JkDev 3 Nov 12, 2022