The stand-alone Oculus Quest was launched this week. Facebook’s Facebook director of technology, Mike Shrepfer, revealed that the company initially had a “vigorous debate” about whether the product would even exist:
“I remember a vigorous debate about whether to create such a product at all. Built-in tracking was in the research stage, and the idea of making a device on a mobile chip seemed impossible to many. Now we see a working product. ”
Schrepfer reports directly to Mark Zuckerberg. Even VR’s vice-president for VR / AR, Andrew Bosworth, tells Schöpfer all the information, not Zuckerberg directly.
This quote hints at the long cycle of creating Quest, a product that has been developed for many years, based on breakthrough technologies that did not exist when it was conceived:
“Oculus Quest has been in the process of being created for more than three years, but a product like it has been Oculus’s overall goal for much longer.”
In a video interview with Kotaku in 2012 (even before formally joining Oculus), CTO John Carmack described a headset that would use “mobile phone equipment” and cameras for positional tracking, without wires. In 2013, in an interview with Endaget, now at Oculus, Carmack even described this ideal headset as an Android version.
When the Rift and Touch equipment was completed in 2015, Oculus began an ambitious project. The goal was to bring the same Rift + Touch experience, minus some graphical accuracy, to a fully autonomous headset. The project was named “Project Santa Cruz”. As Shrepfer said this week, and as Shaun Liu stated on Oculus Connect 4, many on Facebook thought it was impossible.
Inline Tracking: Research Problem
One of the biggest challenges of the Santa Cruz project was getting high-precision built-in tracking for the headset and two controllers running on mobile equipment. In fact, running it altogether was an unsolved problem right from the start.
While today there are other autonomous headsets with integrated head tracking (besides Quest), which also use cameras to track controllers. HTC Vive Focus Plus is equipped with 6DoF controllers, but they are tracked using ultrasound equipment, not computer vision software. Like the Lenovo Mirage Solo, the cameras on the Focus Plus are used only for the headset itself.
Microsoft introduced built-in tracking to the VR market with Windows MR at the end of 2017, but it depended on the computing power of the PC.
Even when the Santa Cruz project began to demonstrate high-quality built-in tracking, then it was considered a field of research, not a ready-made technology. To create a team to solve it, Oculus has acquired a number of computer vision startups, including the 13th Lab and Surreal Vision.
Built-in tracking: Is it possible on a mobile phone?
Shortly before Oculus Connect 2 in 2015, Oculus founder Palmer Lucky stated:
“Inline tracking currently does not work on mobile devices.”
During the conference, Carmack elaborated on the complexity of this problem and criticized the company’s research focus on optimizing the underlying technology for mobile equipment:
“It does not look good to create an integrated tracking system that does not consume much battery power. We have 30 Oculus computer vision experts from various companies that we have acquired, and none of them wants to solve this problem. They are all working on their esoteric, sort of research things, while this is a problem that I want to solve right now. It is a pity that no one has spent on it all last year. ”
Tracking on a stand-alone headset was useless if it discharged the battery in minutes. But Carmack and others at Oculus continued to work, and by February 2016, the technical director was spending most of his time on positional tracking.
Hardware acceleration, active cooling
At the time, at the beginning of 2016, Carmack’s goal to track inside out seemed to be to bring it to Gear VR, a mobile VR solution, during the creation of which Oculus collaborated with Samsung. New hypothetical future products are one thing, but the Gear VR experience can be radically improved with positional tracking.
Unfortunately, position tracking never came to Gear VR (at least officially). Primarily due to the problems that Carmack identified in Oculus Connect 2. However, cooling is another problem.
The main disadvantage of Gear VR was that it often overheated during gameplay, causing the system to shut down. This can happen very quickly, depending on how much the game has strained the system. The launch of the VR game, cameras and advanced computer vision algorithms will immediately make this problem even more global.
Quest uses two methods to overcome these problems.
Thanks to a close partnership with Qualcomm, created for Gear VR, and then Oculus Go, Oculus was able to code tracking to work on Hexagon DSP, and not on the CPU. Smartphones mainly use DSP to enhance the photos taken by their cameras, and Quest uses it to track in VR. Oculus confirmed that positional tracking “does not affect” the performance of VR applications.
Quest uses an active cooling fan to solve the overheating problem. This is rarely done with ARM processors. Apple TV 4K and HTC Vive Focus are the only items in the consumer market that we know about (and the latter is technically for the enterprise).
While the issue of mobile tracking was being resolved, others at Oculus were developing hardware. The first task was to find out what equipment would be required. How many cameras and in what position? One of the first functional prototypes was named Monarch, which was referred to in the conversation in OC5. Monarch was proof of the concept that was achieved with large computer vision cameras.
The team then worked on miniaturizing this and finding the optimal placement of the camera. This led to a configuration seen today.
Era Santa Cruz
At Oculus Connect 3 at the end of 2016, Oculus revealed the existence of the Santa Cruz project and demonstrated this headset to press members.
The front of the headset was essentially just a Rift with added cameras. The battery, processor and all other mobile equipment were behind, behind the head. It is unclear why Oculus decided to put the equipment in front, but this is probably due to the cost and the inability to lean back.
Oculus Connect 4: Controllers
On Oculus Connect 4 at the end of 2017, Oculus showed that the last major milestone in the development of the Santa Cruz project was crossed – now the virtual reality helmet had positionally tracked controllers.
The controllers resembled Rift touch controllers, but with a tracking ring facing up, not down. Like the original touchscreen, they were tracked not only using IR LEDs under the tracking ring, but also using cameras on the headset.
Oddly, Oculus chose to replace the buttons and thumb with the trackpad. This decision was criticized by both developers and fans. Thumbs were one of Touch’s favorite features, and developers have already built their control schemes around this element. But, if this decision were made, they would have to develop separate control schemes for Rift and Santa Cruz.
At GDC 2018, Oculus announced that this decision was reversed. Santa Cruz will now have the same input elements as the Rift.
Seven years after John Carmack first laid out the goal, and almost three years since the controller-free headset was first shown, the Oculus Quest is now a $ 399 product that you can buy at Best Buy or Amazon.
Efforts to develop this product have created a new category of VR devices, redefining the VR market, which previously was between non-interactive mobile and interactive PCs. How well Quest will spread among consumers remains to be seen, but it seems certain that a standalone helmet will become an important part of the future VR.
Stay up to date with the latest news with VRcue.