Stereolabs brings position tracking to mobile VR using its Zed camera pair

Adding to the cost and complexity of VR has been the need for fixed external sensors to enable positional tracking. Oculus Rift uses one small sensor for human-scale tracking, while the Vive relies on two to provide room-scale VR. 3D imaging pioneer Stereolabs is aiming to change that, by enabling full position tracking — at human-scale, room-scale, and beyond — using its ZED stereo camera and a lot of Nvidia GPU horsepower. With the right GPU solution, this approach can even be made to work with mobile VR headsets like the Samsung Gear VR.

I got to try out a mobile rig with this capability on a visit to Stereolabs' San Francisco office. I was able to take advantage of the large open area of the Upload VR collective to roam around in both fully virtual game environments, and in one the company had created using its camera to create a 3D model of a real-world scene.

Tracking location and motion through imaging isn't new. The technique, called SLAM, has been around for a long time — and is in use in other products like Lenovo's upcoming Tango-based phone. But having it work accurately in real time continues to be challenging. The tracking provided by the Stereolabs system was accurate in my experience, and felt similar to when I've used a Vive or Rift. There was a bit of a lag, which Stereolabs attributes to the status of the demos as essentially thrown-together prototypes to illustrate the capabilities of their developer kit. Especially when used with a mobile headset, I can see that completely-eliminating lag will be an ongoing issue, as the camera data needs to be sent to an off-board GPU (either on a PC or a Jetson-embedded unit), and then back to the headset itself for incorporation into the running game engine code.

Why position tracking matters

Position tracking is important because it allows you to physically move around in a virtual world. Without it, you can look around (typically tracked by the headset's sensors), but motion in a line isn't tracked — so you can't walk, jump, or dodge, for example. For 360-degree video consumption, this may not matter much, as often those videos only support a single viewpoint at any given point in the video. But for more immersive games and experiences, and many commercial and industrial applications, moving around in a scene is critical.

The Zed depth-sensing camera is stylish, but of course doesn't look all that great glued to the top of a headset. Stereolabs is planning to create future versions that are easier to integrate with headsets. It's pure speculation on my part, but with the rumors of a stereo camera in the iPhone 7, I can imagine a time when phone-powered headsets will have enough imaging and processing capability to run a system like this completely natively. Similarly, adding a Zed camera to an existing headset is actually more expensive than coupling it with a simple location-sensing system. So, for the Stereolabs solution to become practical for consumers, it will need to be integrated in a nicer and significantly less expensive form factor.

Anyone with a ZED camera (priced at $449) can download the updated SDK that includes position tracking. As with the current version, it requires an Nvidia CUDA-capable GPU to run.

Source: Stereolabs brings position tracking to mobile VR using its Zed camera pair