An executor that wraps 3D mesh models and encodes 3D content documents to d-dimension vector.

Related tags

Image Processing executor-3d-encoder
Overview

3D Mesh Encoder

An Executor that receives Documents containing point sets data in its blob attribute, with shape (N, 3) and encodes it to embeddings of shape (D,). Now, the following pretrained models are ready to be used to create embeddings:

  • PointConv-Shapenet-d512: A PointConv model resulted in 512 dimension of embeddings, which is finetuned based on ShapeNet dataset.
  • PointConv-Shapenet-d1024: A PointConv model resulted in 1024 dimension of embeddings, which is finetuned based on ShapeNet dataset.

Usage

via Docker image (recommended)

from jina import Flow

f = Flow().add(uses='jinahub+docker://3DMeshEncoder', \
               uses_with={'pretrained_model': 'PointConv-Shapenet-d512'})

via source code

from jina import Flow

f = Flow().add(uses='jinahub://3DMeshEncoder', \
               uses_with={'pretrained_model': 'PointConv-Shapenet-d512'})

This Executor offers a GPU tag to speed up encoding. For more information on how to run the executor on GPU, check out the documentation.

How to finetune pretrained-model?

install finetuner

$ pip install finetuner

prepare dataset

TBD...

finetuning model with labeled dataset

$ python finetune.py --help

$ python finetune.py --model_name pointconv \
    --train_dataset /path/to/train.bin \
    --eval_dataset /path/to/eval.bin \
    --batch_size 128 \
    --epochs 50

finetuning model with unlabeled dataset

$ python finetune.py --model_name pointconv \
    --train_dataset /path/to/unlabeled_data.bin \
    --interactive

References

  • PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation
  • PointConv: Deep Convolutional Networks on 3D Point Clouds
Comments
  • which version should the `finetuner` module be?

    which version should the `finetuner` module be?

    I was preparing the environment of this repo. However, I found it confused that the python import code in finetune.py gives error message:

    from finetuner.tuner.callback import BestModelCheckpoint
from finetuner.tuner.pytorch.losses import TripletLoss
from finetuner.tuner.pytorch.miner import TripletEasyHardMiner

    It seems that there's no finetuner.tuner in finetuner module but finetuner.finetuner. Besides, callback/pytorch is not in finetuner.finetuner now.

    Now my finetuner module installed by running pip install -U finetuner is 0.5.0. Is there any solution? Maybe you can freeze the version in requirements.txt.

    opened by elenacliu 4
  • fix: fix previous structure

    fix: fix previous structure

    I got pretrain and finetune process mixed up, and previously I pretrained the models using finetune model structure, which might lead to worse results. The fix code is in process and may be finished in a few days.

    opened by elenacliu 2
  • Add dataset, preprocess and pointnet2

    Add dataset, preprocess and pointnet2

    Here is the modification of the structure. There are still some bugs in my pointnet2 codes. I will solve the possible problems after you comment.

    @numb3r3 @jemmyshin

    opened by elenacliu 1
  • In `farthest_point_sample` function, the first centroid sampled is not random.

    In `farthest_point_sample` function, the first centroid sampled is not random.

    I'm reading your codes.

    https://github.com/jina-ai/executor-3d-encoder/blob/main/executor/models/modules.py#L75

    The randint line is commented, but in the original implementation and the algorithm itself, it should be random? Maybe we can provide a fixed seed if we want to get the same result?

    opened by elenacliu 1
  • Provide diffusion encoder

    Provide diffusion encoder

    https://github.com/nmwsharp/diffusion-net

    I'm unfamiliar with the jina ecosystem. So this is my first issue.

    Final goal is to have dalle generate 3d digital human meshes like https://github.com/jina-ai/dalle-flow

    opened by fire 2
Releases(v0.3.0)
Owner
Jina AI
A Neural Search Company. We provide the cloud-native neural search solution powered by state-of-the-art AI technology.
Jina AI
GitHub
Python framework for creating and scaling up production of vector graphics assets.

Board Game Factory Contributors are welcome here! See the end of readme. This is a vector-graphics framework intended for creating and scaling up prod

Adam Volný 5 Jul 13, 2022
Clip Bing Maps backgound as RGB geotif image using center-point from vector data of a shapefile and Bing Maps zoom

Clip Bing Maps backgound as RGB geotif image using center-point from vector data of a shapefile and Bing Maps zoom. Also, rasterize shapefile vectors as corresponding label image.

Gounari Olympia 2 Nov 22, 2021
sK1 2.0 cross-platform vector graphics editor

sK1 2.0 sK1 2.0 is a cross-platform open source vector graphics editor similar to CorelDRAW, Adobe Illustrator, or Freehand. sK1 is oriented for prepr

sK1 Project 238 Dec 4, 2022
Python script to generate vector graphics of an oriented lattice unit cell

unitcell Python script to generate vector graphics of an oriented lattice unit cell Examples unitcell --type hexagonal --eulers 12 23 34 --axes --crys

Philip Eisenlohr 2 Dec 10, 2021
Create a 2D mesh for an airfoil in GMSH using python.

GMSHFoil A simple class to create a 2D mesh for an airfoil in GMSH using python. Requirements pip install airfoils

Charilaos Mylonas 1 May 16, 2022
A minimal, standalone viewer for 3D animations stored as stop-motion sequences of individual .obj mesh files.

ObjSequenceViewer V0.5 A minimal, standalone viewer for 3D animations stored as stop-motion sequences of individual .obj mesh files. Installation: pip

csmailis 2 Aug 4, 2022
An API that renders HTML/CSS content to PNG using Chromium

html_png An API that renders HTML/CSS content to PNG using Chromium Disclaimer I am not responsible if you happen to make your own instance of this AP

null 10 Aug 8, 2022
Blender addon to generate better building models from satellite imagery.

Blender addon to generate better building models from satellite imagery.

Ivan Ereshchenko 24 Apr 14, 2022
Seaborn-image is a Python image visualization library based on matplotlib and provides a high-level API to draw attractive and informative images quickly and effectively.

seaborn-image: image data visualization Description Seaborn-image is a Python image visualization library based on matplotlib and provides a high-leve

null 48 Jan 5, 2023
This is a new web-based photo management application. Run it on your home server and it will let you find the right photo from your collection on any device. Smart filtering is made possible by object recognition, location awareness, color analysis and other ML algorithms.

Photonix Photo Manager This is a photo management application based on web technologies. Run it on your home server and it will let you find what you

Damian Moore 1.5k Jan 1, 2023
HtmlWebShot - A python3 package which Can Create Images From url, Html-CSS, Svg and from any readable file and texts with many setup features.

A python3 package which Can Create Images From url, Html-CSS, Svg and from any readable file and texts with many setup features

Danish 24 Dec 14, 2022
Django helper application to easily and non-destructively crop arbitrarily large images in admin and frontend.

django-image-cropping django-image-cropping is an app for cropping uploaded images via Django's admin backend using Jcrop. Screenshot: django-image-cr

Jonas und der Wolf GmbH 546 Jan 3, 2023
Fast batch image resizer and rotator for JPEG and PNG images.

imgp is a command line image resizer and rotator for JPEG and PNG images.

Terminator X 921 Dec 25, 2022
This is an app that allows users to upload photos and display and store the photos in a file until the user deletes them.

Qt Photo App This is an app that allows users to upload photos and display and store the photos in a file until the user deletes them. Setup python3 -

Kathy Yang 5 Jan 22, 2022
Snowfall - helpful image handling utils - abstracts various file and opencv and pil features into result oriented functions

snowfall helpful image handling utils - abstracts various file and opencv and pil features into result oriented functions usage examples: from image_h

Less Wright 2 Jan 9, 2022
Png-to-stl - Converts PNG and text to SVG, and then extrudes that based on parameters

have ansible installed locally run ansible-playbook setup_application.yml this sets up directories, installs system packages, and sets up python envir

null 1 Jan 3, 2022
Validate arbitrary image uploads from incoming data urls while preserving file integrity but removing EXIF and unwanted artifacts and RCE exploit potential

Validate arbitrary base64-encoded image uploads as incoming data urls while preserving image integrity but removing EXIF and unwanted artifacts and mitigating RCE-exploit potential.

A3R0 1 Jan 10, 2022
3D Model files and source code for rotating turntable. Raspberry Pi, DC servo and PWM modulator required.

3DSimpleTurntable 3D Model files and source code for rotating turntable. Raspberry Pi, DC servo and PWM modulator required. Preview Construction Print

Thomas Boyle 1 Feb 13, 2022
🎨 Generate and change color-schemes on the fly.

Generate and change color-schemes on the fly. Pywal is a tool that generates a color palette from the dominant colors in an image. It then applies the

dylan 6.9k Jan 3, 2023