Objectives:
- Create a two-way connection between a virtual environment (mediated through a head-mounted display) and a physical environment (mediated through a robotic manipulator).
- Assess baseline performance of initial system to understand applicability and future development.
Motivation:
The creation of haptic displays with increased immersive capabilities will produce more realistic renderings of reality within a virtual experience to enhance outcomes such as teaching and entertainment.
System:
- Virtual
- Visualization of virtual environment and hand-tracking through the Meta Quest Pro head-mounted display
- Unity 3D game engine (2022.3.17f1)
- D-Link DQA-F18 VR Air Bridge
- Physical
- Kuka LBR iiwa 7 R800
- Custom end-effector
- Communication
- UDP over ethernet serialized using Google’s Protocol Buffer framework
Teleoperation Task:
As a first demonstration of our system setup, we conceived a teleoperation task, as where the robot’s end-effector follows the movement of a user’s hand as tracked
by the headset.
Initial Performance Analysis:
To quantify latency, we devised an experimental setup where a replica of a human hand was attached as the end effector of the robot, and the headset was affixed near the base of the robot as shown below. The phase difference between the headset’s reported motion of the hand and the manipulator’s sinusoidal motion of the first joint was found to determine the hand tracking latency of the system.
| Mean Latency | Standard Deviation | |
| Cold-Start | 177.4 ms | 78.0 ms |
| Constant Motion | 27.8 ms | 13.7 ms |
A motion capture system consisting of 12 Vicon Vero cameras (Tracker 3.9.0) was used to provide ground-truth reference to determine hand-tracking positional error.
| Mean Error | Standard Deviation | |
| Static Position | 1.30 cm | 1.48 cm |
Future Work:
- Further analysis of latency and positional accuracy effects.
- Development of physical interaction using haptic proxies within a virtual experience.
Preprint Paper: