VR-Caps: A Virtual Environment for Active Capsule Endoscopy


VR-Caps: A Virtual Environment for Capsule Endoscopy


We introduce a virtual active capsule endoscopy environment developed in Unity that provides a simulation platform to generate synthetic data as well as a test bed to develop and test algorithms. Using that environment, we perform various evaluations for common robotics and computer vision tasks of active capsule endoscopy such as classification, pose and depth estimation, area coverage, autonomous navigation, learning control of endoscopic capsule robot with magnetic field inside GI-tract organs, super-resolution, etc. The demonstration of our virtual environment is available on YouTube.

Our main contributions are as follows:

  • We propose synthetic data generating tool for creating fully labeled data.
  • Using our simulation environment, we provide a platform for testing numerous highly realistic scenarios.

See Summary of our work for details and our Paper.

Getting Started

1. Installation

The VR-Caps contains several components:

  • Unity
  • ML-Agents
  • SOFA
  • MagnetoDynamics
  • SC-SfMLearner

Consequently, to install and use the VR-Caps you will need to:

Clone the VR-Caps Repository

Now that you have installed Unity and Python, you can now clone this repository.

git clone https://github.com/CapsuleEndoscope/VirtualCapsuleEndoscopy.git

Now, you will need open Unity Hub and simply create a new Unity project by adding VR-Caps-Unity. Then, simply open the Unity by clicking on the project you just added. Please note that we have tested the environment on Unity Version 2019.3.3f1.

The opening scene Clinic Setup is our default scene. You can navigate other scenes by Scenes.

2. Creating Synthetic Data

For data creation tool, please open Record Collect scene from Scenes.

This will open a scene where there is one of our GI system models is already placed and a capsule with a mono camera and a light source is attached on it.

You will need Unity Recorder which can be installed using Unity Package Manager (see image)

After installing Unity Recorder, navigate to Recorder Window and open the Recorder panel.

On the panel, click Add New Recorders and then select Image Sequence and AOV Image Sequence for RGB image recording and depth recording respctively.

Adjust image resolutions from Capture Output Resolution and FPS(Frame per second) from Target Value.

Importing new models to the scene

You can import other models from GI-Organs folder by simply dragging the model to the scene. You will notice that the imported model has no texture.

In order to add texture, you need to navigate Organs folder and simply drag material files (.mat) to corresponding 3D organs (For example, Colon Material.mat to Colon that can be selected in the Hierarchy window under the Prefab.)

Generating 3D organs from scratch

One can also generate 3D organs from different patients using the publicly available Cancer Imaging Archive. Please select a CT data in the DICOM format among the dataset for colon or stomach (see below image). Please note that the DICOM images consist of two sets, one taken in the supine position and the other in the prone position. The supine position DICOM images were used since that is the patient’s position during the capsule endoscopy session

After downlading DICOM data, use InVesalius or any similar softwarer to convert the DICOM images to 3D objects. The software provides an automatic selection of the regions desired to be converted, which in our case is the Soft Tissue. Then a surface will be created on the selected regions constructing the corresponding 3D model, which is exported as a Wavefront (.obj) file.

The 3D model is then imported into Blender for further processing which includes removal of bones, fat, skin, and other artifacts that the imported model has so that only the geometries of the colon, small intestines and stomach remain. Please note that not all converted 3D models includes the whole colon and intestines, these models should be discarded.

As some models consist of a large number of mesh which makes it hard to process, we reduced the number of mesh by using another software called MeshLab, using an algorithm called Quadric Edge Collapse Decimation for mesh simplification. It reduces the face number of a mesh while preserving its boundaries and normals.

Please note that, due to some imperfections on the CT data, you may need to fill the gaps and fix the topology of the organs. We used Blender for this operation. Please make sure that there is no missing parts in the 3D organs and the connections and the openings between the stomach and small intestines, and the small intestines and colon are all set.

Generating Disease Classes

We create pipeline to mimic 3 classes of diseases in our environment (Polpys with various shapes and sizes, Ulcerative Collitis and Hemorrage for 3 and 4 different amount and severity levels respectively) that can be used to train/test disease classification algorithms.


In the Cancer Imaging Archive, you can also find different models of the organs with the cancerous lumps that can be used to mimic real shaped polyps with realistic locations of occurence. Firts, navigate to relavant class in the archive and download the corresponding DICOM format. Then, by following the same steps explained above, you can create 3D organ with polyps. In order to attain the texture particularly generated for polyps, you should use Blender or a similar software to manually depart meshes for the regions of polyp occurences and save them as different models. Then in Unity you can attain polyps texture Polyps.mat in the where other organ textures are located.

Ulcertive Collitis and Hemorrage

Unlike Polyps, Ulcer and Hemorrage do not have differ in the topology of 3D organs but in texture. Therefore, we generate specific textures for these classes. In order to create organs with these diseases, please select and attain a texture from the textures folder where other .mat files exist.

Various camera designs

As there are commercially available options in capsule camera designs in the wireless capsule endoscopy, in our environment, we extend the standard mono camera capsule to different designs such as stereo, dual and 360° camera). You can select these options from the Capsules folder under this folder

Adjusting camera parameters and post processing effects

Adjusting camera parameters can be used for both mimicing real endoscopy cameras and augmenting the data.

You can use the camera intrinsic parameters that we get by calibrating MiroCam and PillCam capsule endoscope cameras or play with them to generate augmented data.

To adjust Unity Camera, use parameters on Inspector window (e.g,, Field of View, Sensor Size, Focal Length etc.) Set the average of and for the focal length, 2x optical center ( and ) for sensor size X and Y.

Please note that image resolution is adjusted on Recorder.

Specular reflection which occurs on the surface of organs due to interaction of light source can also be adjusted by the Coat Mask parameter on Unity's Inspector window.

Post-processing effects that HDRP (High Definition Rendering Pipeline) provides (specular reflection, vignette, lens distortion, chromatic aberration and depth of field) can also be adjusted with relevant parameters.

Movement of the capsule

For the actuation of the capsule, we have placed a cylinder magnet inside the capsule and a ball magnet attached to the robot arm. Simulation of the magnetic field is modeled as dipole-dipole interactions by using (MagnetoDynamics).

The default Scene has two infinitesimal dipoles (MagneticDipole prefabs) embedded in Rigidbodies of the DiscMagnet(child object of Capsule) and BallMagnet objects. In Unity’s Scene and Hierarchy views, you can see that MagneticDipoles are attached to the them. Please note that every Scene that uses Magnetodynamics must contain an ElectromagneticFieldController that can be found inside the Magnetodynamics folder. Just drag it to anywhere in the scene to activate magnetic field.

If the InverseKinematic.cs script is activated, the robotic arm will also be moving as you move the ball magnet (either by a script or manually).

It is also possible to move the capsule directly without any electro magnetic force on it. To do that, add the CapsuleMovement.cs script to the capsule and control it by keyboard arrows.

The capsule camera can also be controlled if you add MouseCameraController.cs script to the camera. In that way, the capsule camera will look to the direction pointed by the mouse.

3. Tasks

3.1. Area Coverage

We use Unity's ML-Agents Toolkit to train a Deep Reinforcement Learning (DRL) based active control method that has a goal of learning a maximum coverage policy for human organ monitoring within a minimal operation time. We create a different project for the area coverage task (VR-Caps-Unity-RL) . To reproduce results or train you own control policy please follow the instructions provided here

3.2. Pose and Depth Estimation

To illustrate the effectiveness of VR-Caps environment in terms of neural network training for pose and depth estimation, we trained a state-of-the-art method, SC-SfMLearner algorithm, using synthetic data created on VR-Caps. The results showed on the paper can be reproduced by using the models given in the drive. Virtual Pre Training folder is for the model trained only with the synthetic data. Model 1 corresponds to the case when there is only real data is used (without virtual pretraining) and Model 2 is the case where we use synthetic data for pre-training and then fine-tune with the real data from EndoSLAM dataset. For the pretraining, we used the data on drive. The test sets for colon are Colon_Traj5_HighCam and Colon_Traj5_LowCam and for small instesine SmallInstesine_Traj1_HighCam and SmallInstesine_Traj4_HighCam.

For pose estimation, ATE and RPE calculations can be done by using this script To extend the test cases, you can generate new data as explained above and train new SC-SfM networks and test on both real or synthetic data.

For depth estimation, we test on both virtual and real endoscopy data (Kvasir and Redlesion datasets).

3.3. 3D Reconstruction

In this work, we propose and evaluate a hybrid 3D reconstruction technique. To exemplify the effectiveness of Unity data, we compare the results of reconstructions both on real and synthetic data.

3.4. Disease Classification

We mimic the 3 diseases (i.e., Polyps, Haemorrhage and Ulcerative Collitis) in our simulation environment. Hemorrage and Ulcerative Collitis are created based on the real endoscopy images from Kvasir dataset mimicking the abnormal mucosa texture. As polyps are not only distintive in texture but also in topology, we use CT scans from patients who have polyps and use this 3D morphological information to reconstruct 3D organs inside our environment. instances with different severities ranging from grade 1 to grade 4, three different grades of ulcerative colitis, and different polyps instances with various shapes and sizes.

3.5. Super Resolution

We benchmarked the effectivity of the Unity environment using Deep Super-Resolution for Capsule Endoscopy (EndoL2H) network based on the dilemma of high camera resolution coming with increasing the size of the optics and the sensor array.


Visual demonstration of all tasks done on this work and their results are as follows: For more details, please visit the article.

Frequently Asked Questions



If you find our work useful in your research or if you use parts of this code please consider citing our paper:

      title={VR-Caps: A Virtual Environment for Capsule Endoscopy}, 
      author={Kagan Incetan and Ibrahim Omer Celik and Abdulhamid Obeid and Guliz Irem Gokceler and Kutsev Bengisu Ozyoruk and Yasin Almalioglu and Richard J. Chen and Faisal Mahmood and Hunter Gilbert and Nicholas J. Durr and Mehmet Turan},
  • UnityCam - normalization and unit measurements of depth map

    UnityCam - normalization and unit measurements of depth map

    Hi, I'm writing to you to request more details about the normalization, unit measurements, and range of the depth ground truths, specifically of the UnityCAM images. I read the articles related to the library and database, but I didn't notice such information.

    How should I interpret the relationship between the gray level values and the depth? Does each level correspond to 1 millimeter (I guess it is this option), 1 centimeter?

    Or these depth maps have been normalized to some minimum and maximum depth values?

    Thank you for your time, and I will be pending for your answer.

    opened by jaruanob 3
  • VR-Caps-Unity-RL


    I installed VR-Caps-Unity-RL as shown in README.md. Two errors come out,

    Rebuilding Library because the metadata folder could not be found!
    Assets\Scripts\CoverageMagnetAgent.cs(98,26): error CS0115: 'CoverageMagnetAgent.OnActionReceived(float[])': no suitable method found to override

    Would you help me to solve the issue? This doesn't occur in the ml-agent example project - ball3d https://github.com/Unity-Technologies/ml-agents/tree/main/Project/Assets/ML-Agents/Examples/3DBall

    Environment: Windows 10; Unity 2020.3.13f1; ML-Agent 18; C#: ASP.NET Core 5.0 Thank you.

    opened by davidchun847 3
  • How to obtain depth images ground truth?

    How to obtain depth images ground truth?

    Thank you very much for the Unity project. It would be helpful for my future research.

    I am trying to collected my dataset to finish depth estimation task. However, I was confused about how to obtain depth images ground truth in your project for evaluation.

    Could you give me a tutorial? Thank you very much!

    opened by wangliangyu-code 2
  • Unity recorder failed recording depth sequences

    Unity recorder failed recording depth sequences

    Hi! I was trying to record depth image sequences as described in README.md, but they came out as RGB rendered images. Here are how I set properties in unity AOV recorder and images coming out: 1 2 What did I do wrong? Thank you for your time!

    opened by TianCuteQY 2
  • Missing scripts of creating images of depth

    Missing scripts of creating images of depth

    I couldn't find any scripts about creating images of depth, found only the pose recorder script. If there are, would you let me know how to take the images? Environment: Windows 10; Unity 2020.3.13f1; ML-Agent 18; C#: ASP.NET Core 5.0

    opened by davidchun847 1
  • Magnetodynamics & sofaapapi-unity

    Magnetodynamics & sofaapapi-unity

    Hi, Could you please tell me where to install the Magnetodynamics module ? I can't find it in the file you released. Also, I'd like to know if there are any free resources or alternative plugins for SofaAPAPI-Unity3D to share.

    opened by zmyl18641515513 1
  • How to add the depth channel into observation?

    How to add the depth channel into observation?

    Hi. As I carefully read about your paper and examples, I found the visual signal are not used in the given MLAgents. In addition, I found your settings for obtaining the depth channel are offline. Do you have any good solution about the online obtaining of depth channel?

    opened by howardchina 0
Camera-caps - Examine the camera capabilities for V4l2 cameras

camera-caps This is a graphical user interface over the v4l2-ctl command line to

Jetsonhacks 25 Dec 26, 2022
AI Virtual Calculator: This is a simple virtual calculator based on Artificial intelligence.

AI Virtual Calculator: This is a simple virtual calculator that works with gestures using OpenCV. We will use our hand in the air to click on the calc

Md. Rakibul Islam 1 Jan 13, 2022
A PyTorch implementation of "Capsule Graph Neural Network" (ICLR 2019).

CapsGNN ⠀⠀ A PyTorch implementation of Capsule Graph Neural Network (ICLR 2019). Abstract The high-quality node embeddings learned from the Graph Neur

Benedek Rozemberczki 1.2k Jan 2, 2023
GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks

GPU-accelerated PyTorch implementation of Zero-shot User Intent Detection via Capsule Neural Networks This repository implements a capsule model Inten

Joel Huang 15 Dec 24, 2022
paper: Hyperspectral Remote Sensing Image Classification Using Deep Convolutional Capsule Network

DC-CapsNet This is a tensorflow and keras based implementation of DC-CapsNet for HSI in the Remote Sensing Letters R. Lei et al., "Hyperspectral Remot

LEI 7 Nov 29, 2022
Re-implementation of the vector capsule with dynamic routing

VectorCapsule Re-implementation of the vector capsule with dynamic routing We implement the vector capsule and dynamic routing via graph neural networ

ZhenchaoTang 10 Feb 10, 2022
A modular active learning framework for Python

Modular Active Learning framework for Python3 Page contents Introduction Active learning from bird's-eye view modAL in action From zero to one in a fe

modAL 1.9k Dec 31, 2022
Vowpal Wabbit is a machine learning system which pushes the frontier of machine learning with techniques such as online, hashing, allreduce, reductions, learning2search, active, and interactive learning.

This is the Vowpal Wabbit fast online learning code. Why Vowpal Wabbit? Vowpal Wabbit is a machine learning system which pushes the frontier of machin

Vowpal Wabbit 8.1k Jan 6, 2023
Code for Multiple Instance Active Learning for Object Detection, CVPR 2021

Language: 简体中文 | English Introduction This is the code for Multiple Instance Active Learning for Object Detection, CVPR 2021. Installation A Linux pla

Tianning Yuan 269 Dec 21, 2022
Code for Multiple Instance Active Learning for Object Detection, CVPR 2021

MI-AOD Language: 简体中文 | English Introduction This is the code for Multiple Instance Active Learning for Object Detection (The PDF is not available tem

Tianning Yuan 269 Dec 21, 2022
Code for the Active Speakers in Context Paper (CVPR2020)

Active Speakers in Context This repo contains the official code and models for the "Active Speakers in Context" CVPR 2020 paper. Before Training The c

null 43 Oct 14, 2022
Tutorial on active learning with the Nvidia Transfer Learning Toolkit (TLT).

Active Learning with the Nvidia TLT Tutorial on active learning with the Nvidia Transfer Learning Toolkit (TLT). In this tutorial, we will show you ho

Lightly 25 Dec 3, 2022
Official PyTorch code for CVPR 2020 paper "Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision"

Deep Active Learning for Biased Datasets via Fisher Kernel Self-Supervision https://arxiv.org/abs/2003.00393 Abstract Active learning (AL) aims to min

Denis 29 Nov 21, 2022
gym-anm is a framework for designing reinforcement learning (RL) environments that model Active Network Management (ANM) tasks in electricity distribution networks.

gym-anm is a framework for designing reinforcement learning (RL) environments that model Active Network Management (ANM) tasks in electricity distribution networks. It is built on top of the OpenAI Gym toolkit.

Robin Henry 99 Dec 12, 2022
Active and Sample-Efficient Model Evaluation

Active Testing: Sample-Efficient Model Evaluation Hi, good to see you here! ?? This is code for "Active Testing: Sample-Efficient Model Evaluation". P

Jannik Kossen 19 Oct 30, 2022
Look Who’s Talking: Active Speaker Detection in the Wild

Look Who's Talking: Active Speaker Detection in the Wild Dependencies pip install -r requirements.txt In addition to the Python dependencies, ffmpeg

Clova AI Research 60 Dec 8, 2022
Submodular Subset Selection for Active Domain Adaptation (ICCV 2021)

S3VAADA: Submodular Subset Selection for Virtual Adversarial Active Domain Adaptation ICCV 2021 Harsh Rangwani, Arihant Jain*, Sumukh K Aithal*, R. Ve

Video Analytics Lab -- IISc 13 Dec 28, 2022
PyTorch implementation of "A Simple Baseline for Low-Budget Active Learning".

A Simple Baseline for Low-Budget Active Learning This repository is the implementation of A Simple Baseline for Low-Budget Active Learning. In this pa

null 10 Nov 14, 2022
Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks (MAPDN)

Multi-Agent Reinforcement Learning for Active Voltage Control on Power Distribution Networks (MAPDN) This is the implementation of the paper Multi-Age

Future Power Networks 83 Jan 6, 2023