Note: I uncommented the following code: https://github.com/facebookresearch/tacto/blob/master/tacto/renderer.py#L21 which is equivalent to your suggestion on using EGL Mode

Below is the error I get when attempting to run tacto headless:

Traceback (most recent call last):
  File "demo_pybullet_digit.py", line 23, in main
    digits = tacto.Sensor(**cfg.tacto, background=bg)
  File "/viscam/u/small02/tacto/tacto/sensor.py", line 80, in __init__
    self.renderer = Renderer(width, height, background, config_path)
  File "/viscam/u/small02/tacto/tacto/renderer.py", line 77, in __init__
    self._init_pyrender()
  File "/viscam/u/small02/tacto/tacto/renderer.py", line 115, in _init_pyrender
    self.r = pyrender.OffscreenRenderer(self.width, self.height)
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/pyrender/offscreen.py", line 31, in __init__
    self._create()
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/pyrender/offscreen.py", line 149, in _create
    self._platform.init_context()
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/pyrender/platforms/egl.py", line 177, in init_context
    assert eglInitialize(self._egl_display, major, minor)
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/OpenGL/platform/baseplatform.py", line 402, in __call__
    return self( *args, **named )
  File "/sailhome/small02/miniconda2/envs/tacto/lib/python3.8/site-packages/OpenGL/error.py", line 228, in glCheckError
    raise GLError(
OpenGL.error.GLError: GLError(
	err = 12289,
	baseOperation = eglInitialize,
	cArguments = (
		<OpenGL._opaque.EGLDisplay_pointer object at 0x7f00379bbdc0>,
		c_long(0),
		c_long(0),
	),
	result = 0
)

Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Hi, I am trying to use tacto for a door opening task, but I don't get a tactile feedback when grasping the handle.

For other objects like this banana it works well:

Do you have any idea what could cause this problem? I made sure that the door object is added to the digit sensor. This is the mesh of the door:

I took the digit fingers from demo_pybullet_grasp.py and added them to our panda gripper.

Thank you!

Hi,

I have object meshes from .obj files that I add to my pybullet simulator using p.createMultiBody.

What would be the best steps to follow in order to add this object to the Tacto simulator? I am not sure if there's something similar to add_object(): https://github.com/facebookresearch/tacto/blob/a21d0c4626d74a546d94859226c2fea348babb6a/tacto/sensor.py#L127

Thank you, Oliver

Hello,

I have a few digit related questions: 1/ I understand that digit is just another mesh object, that is loaded into bullet like any other collision object. but wondering if there are any units/size/mass of the default loaded digit object? and any function to change these default values?

2/ Let's say I have a heavy mass object B and I am using digit to generate touch forces on it. I do get some force values, but these don't seem realistic when comparing the relative mass sizes of the default-loaded digit and object B. I'm using globalScaling parameter into loadURDF while loading digit currently to do some relative scaling. But, how do you recommend scaling digit relative to object B to generate realistic touch forces? is it necessary? or do you recommend using multiple digits?

Please let me know if my question is unclear. Thanks.

Hi, I read your paper and was impressed by the realistic readings generated by Tacto. You mentioned in the paper that shadows can be easily enabled in Tacto, so I tried with changing the 515th line in renderer.py: color, depth = self.r.render(self.scene, flags=RenderFlags.SHADOWS_ALL) However, pyrender 0.1.45 seems not to support shadows for point lights. An exception is thrown:

NotImplementedError: Shadows not implemented for point lights

Could you help me find the way to enable shadows in Tacto? Thank you!

Hi, everyone. I am trying to create a GelSight in a cylinder. I try to write a sensor configuration file according to the OmniContact example. But the result turns out to be a blank image. Here is the link to my sensor configuration file.

|A simple demonstration to the sensor|Gelsight image| |-|-| |||

What could lead to that error? Any information will be highly appreciated!

Hi all, the currently release pypi Tacto package dates December 16, 2020. Would it be possible to build and upload a newer version of Tacto to Pypi to have included the fixes that have been intorduced since then? Best, Oier

I ran setup.py and installed all additional requirements before running each package, but getting this error. I tried doing pip install tables which did not fix the issue. both in draw.py and above file, deepdish is used which depends on pytables

  File "/Users/shivanimall/tacto/experiments/grasp_stability/draw.py", line 62, in <module>
    mean, std = load(field, Ns, epoch=epoch)
  File "/Users/shivanimall/tacto/experiments/grasp_stability/draw.py", line 27, in load
    log = dd.io.load(fn)
  File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/deepdish/io/__init__.py", line 14, in _f
    raise ImportError("You need PyTables for this function")
ImportError: You need PyTables for this function

Hello,

wanted to share some background I am using tacto for and hoping to get your thoughts/suggestions on the best ways about this:

I modified the robot in https://github.com/facebookresearch/tacto/blob/master/experiments/grasp_stability/robot.py to have only one finger tip (i.e deleted the left finger in setup/sawyer_wsg50.urdf) and instead of grasp or reach as in https://github.com/facebookresearch/tacto/blob/master/experiments/grasp_stability/grasp_data_collection.py#L246, trying to perform a touch on an object(is static) using digit but with very close accuracy for the touch location/position on the object.

My end goal is to make the finger_right_tip on end effector reach a desired base position with close accuracy( at least upto one decimal) on the static object so that I can obtain rgb, depth map touch readings for the desired touch/collision point. Currently using go function in robot.py does not get me this accuracy which is central to my use case. another caveat I found -- using the robot arm requires accounting for exact rotation/euler values to reach a desired location.

Some thoughts I have are to 1/ change/simplify robot structure 2/ use a different type of end effector 3/ currently I keep object static and move the digit. but maybe can do the reverse of this- control object movements (via grasp/lift) and bring in contact w/ a static digit touch sensor 4/ the surface area of digit tip also matters since that will determine the maximum covered region of the given touch location. (want it to be small for accuracy)

maybe physics related, but is there a simple way/api in tacto to perform touch on different desired points on an object?

Please let me know if my question is unclear. Thank you so much!

Currently the Sensor class doesn't accept a cid as input parameter [0]. For cases where one wants to have multiple pybullet instances / threads, using the cid would be a simple way to allow it.

[0] https://github.com/facebookresearch/tacto/blob/master/tacto/sensor.py#L52

Hi, if I understand correctly, the rgb images of tacto are being normalized with mean=0.5, std=0.5 https://github.com/facebookresearch/tacto/blob/main/experiments/grasp_stability/train.py#L236 However, since a pretrained resnet18 is used, I think the imagenet statistics should be used for normalization. transforms.Normalize(mean=[0.485, 0.456, 0.406], std=[0.229, 0.224, 0.225])

Hi authors, thank you for sharing this awesome repo!

I'm currently using GelSight Mini from GelSight Inc., and am wondering if I can somehow use this simulator to simulate deformations of the GelSight Mini sensor. For that, I'd like to ask two questions:

1. Do you have any plan to support the GelSight Mini sensor?
2. How can I simulate the GelSight Mini-like sensor by modifying the current codes?
   • It might not be as accurate as Digit sensor, but I imagine this simulator has some parameters that enable it to simulate another gel-based sensor, such as parameters that define sensor size, geometries, pose, etc. I want to know if there are some workarounds.

Thanks!

Thanks you for making this wonderful simulator!

I'm just wondering if it is possible to support franka panda arm smulation? I'm using panda instead of sawyer. I would really want to use panda for simulation. Thanks!

Hi, thanks for the fantastic work:

However, when I use synthetic images to train an image to depth neural network, it works very bad on real-world images.

I found the synthetic images (as shown below) are significantly different from the real image, is this the reason leading to depth calculation failure? When I test the neural network on the synthetic images, it works fine. However, when I test real-world images it just predicts them as no-depth images (a full black image).

As shown in the synthetic images, the contact image of synthetic images looks more three-dimensional.

I am new to the image process area, therefore, I may miss some key pre-treatment processes for the real-world images to make them more like synthetic images.

Is my digit sensor not fabricated well?

Or is any pre-treatment needed for the real-world images to make them more like synthetic images?

Any advice from the community will be extremely helpful to me.

Hello! When I render color and depth with egl, some images have white pixels as shown below. This didn't happen when using OpenGL. Does anyone happen to know why this occurs?

Hi @wx405557858, my team and I have bought the DIGIT sensors (directly from GelSight. I have read in your paper about the grasping stability NN that you trained.

We intend to use the tactile feedback from the sensors to establish the stability of the grasp (we uese real-life YCB Dataset objects). We have some ideas in mind, but we would also test out the NN you trained in the real world. We would probably like to implement it within ROS, so getting anything from python should work. Could you kindly walk me through, at code level:

-How to run inference of the NN? -Would it be possible to have the script to run it and the weights? ( we would probably do a finetuning on top of it) -How/Where to provide the DIGIT tactile images as input? -What exactly is the provided output? Regards Roberto