it works on certain angles, but not all angles. need to test and verify this. this is how the could should work:

You will need to compute the viewmatrix, based on the index finger world transform matrix. You can get this using the pybullet.getLinkState API. Then extract the position and orientation form that, and build the view matrix using pybullet.calculateViewMatrix. It is a bit complex, the racecar example (https://github.com/bulletphysics/bullet3/blob/449c8afc118a7f3629bc940c304743a084dcfac6/examples/pybullet/gym/pybullet_envs/bullet/racecarZEDGymEnv.py#L95) computes the camera based on the chassis. The finger world transform 3x3 matrix is basically fwd, left, up vectors.

better error if path to dataset not found, this is current error:

python reinforce.py --data_path=../concave_objects --render --obj_type=obj pybullet build time: Nov 30 2017 10:16:07 Vendor: NVIDIA Corporation Renderer: NVIDIA GeForce GT 750M OpenGL Engine Version: 4.1 NVIDIA-10.4.2 310.41.35f01 GLSL: 4.10 b3Printf: Selected demo: Physics Server startThreads creating 1 threads. starting thread 0 started thread 0 MotionThreadFunc thread started Traceback (most recent call last): File "reinforce.py", line 192, in env = SenseEnv(vars(args)) File "../env.py", line 47, in init self.load_object() File "../env.py", line 78, in load_object stlfile = files[random.randrange(0,files.len())] File "/Users/jtoy/miniconda3/lib/python3.6/random.py", line 198, in randrange raise ValueError("empty range for randrange() (%d,%d, %d)" % (istart, istop, width)) ValueError: empty range for randrange() (0,0, 0) numActiveThreads = 0 stopping threads stopThreads: Thread 0 used: 1 Thread with taskId 0 exiting Thread TERMINATED destroy semaphore semaphore destroyed destroy main semaphore main semaphore destroyed

our collisions use convex hull shapes. we want to model more accurately with concave shapes. we need to use VHACD on our objects. see this ticket for more information: https://github.com/bulletphysics/bullet3/issues/1324

we can't use resetBasePositionAndOrientation because that breaks the physics simulation. don't use "resetBasePositionAndOrientation" while simulating, it should be only if you 'reset' the simulation at the start of the simulation.

Instead, create a 'fixed' constraint, and set its transform. See for example vrhand.py ( https://github.com/bulletphysics/bullet3/blob/master/examples/pybullet/examples/vrhand.py )

hand_cid = p.createConstraint(hand,-1,-1,-1,p.JOINT_FIXED,[0,0,0],[0.1,0,0],[0.500000,0.300006,0.700000],ho)

Then instead of using resetBasePositionAndOrientation, you use

p.changeConstraint(hand_cid,e[POSITION],e[ORIENTATION], maxForce=50)

(pick your max force as suitable)

While the wand is in contact with the object the movement slows down, giving the appearance that the wand is stuck on the object. This is noticeable during the rendering process.

Wand should not show an appreciable slowdown while in contact with the object.

Related to the use of the computeProjectionMatrixFOV and getCameraImage functions. These introduce overhead that causes the time between successive calls to _step, thus giving the appearance of a slowdown.

In the TouchWandEnv it takes the wand approximately 30,000 moves to touch the object.

In the HandEnv it takes the hand only hundreds of moves to touch the object.

I suspect this is related to the interplay between the maxForce (in the changeConstraint), the mass of the agent, and the self.move parameter

These numbers of actions to touch the object should be standardized to enable the use of a universal max_steps parameter

https://gist.github.com/karpathy/a4166c7fe253700972fcbc77e4ea32c5

this is a pure numpy example. port it to work on sensenet just like our reinforce example: https://github.com/jtoy/sensenet/blob/master/agents/reinforce.py

need to get a logo designed soon. we need to get some concept art designed for this. probably an image with some combination of touch / senses and brains/ artificial intelligence.