Extended horizons: Cantaloupe-flavored multifocal, retina-resolution multifocal prototype headset from Meta

在SIGGRAPH 2023上，我试用了Meta的一款研究原型，它具有近视网膜角分辨率、动态焦距和动态畸变校正的特点。

Butterscotch Varifocal is based on the original Butterscotch prototype that Meta revealed last year. The Butterscotch prototype has an angular resolution of 56 pixels per degree of field of view (PPD), slightly lower than the 60 pixels per degree that the human eye is generally thought to be able to discern. That's almost 3x the central angular resolution of the Quest Pro, and 2x that of the Bigscreen Beyond.

However, the researchers at Meta did not achieve this angular resolution through any groundbreaking or specialized techniques. Butterscotch headsets use off-the-shelf 2880×2880 LCD displays that can provide about 30 PPD resolution in normal VR headsets, but they’re paired with lenses that have about half the field of view. The intent here is to demonstrate what retina resolution feels like, in order to ultimately inform future product prioritization and trade-off decisions. This prototype is not meant to propose any particular new technology to achieve retinal resolution with an acceptable field of view.

Varjo achieves retinal resolution in a very small field of view and works in conjunction with peripheral displays with a lower field of view.

As the name suggests, Butterscotch Varifocal isn't just a demonstration of retina resolution. It also includes zoom technology from Meta's 2018 Half-Dome prototype.

In virtual reality, each eye gets an independent perspective for stereoscopic differences, but that's just one cue the brain uses to determine depth. All headsets on the market today are equipped with fixed focal length lenses. The image is focused at a fixed distance, usually a few meters. Your eyes will point at the virtual object (converge or diverge), but can't actually make real focus adjustments to it. Known as vergence paradox, this can cause strain on the eyes and make virtual objects that are close together appear blurry.

The Half-Dome prototype proposes a solution: track where your eyes are pointing and quickly and mechanically move the display panel to dynamically adjust focus. The Butterscotch Varifocal includes the same eye-tracking and mechanical actuators to achieve the same effect.

The result of this combination of near-view resolution and dynamic focus adjustment is the ability to view virtual worlds with no visible pixelation or aliasing, I can see the tiniest details of even the smallest objects and read anything I can read in the real world to the size of the text. Not even the virtual tablet with the compositor layers that John Carmack repeatedly tells developers is required in current headsets and the phone that displays the fine print. The pixel density here is so high that no tricks like that are needed anymore.

Resolution, it's not the headset's display system that's the limiting factor in resolving detail, it's my eyesight. It's the exact opposite of today's VR headsets, which are mesmerizing and showcase the visual quality that consumer VR might offer in the future.

This is the first time I've tried a zoom headset - something few people outside of Meta have tried - and I can toggle the switch with the push of a button. With the switch on, even the smallest objects remained sharp when virtual objects were held close to my eyes, down to a minimum distance of 20cm. This has practical advantages, but I noticed something more subtle when the zoom was turned on. Getting the focal length right makes the virtual world and the objects within it suddenly look and feel more "real". In fact, some of the demo objects are so detailed that I daresay they feel completely real.

I asked Douglas Lanman, Meta's head of display systems research, for his opinion on this. He told me that while this effect is one that Meta researchers are aware of and discusses, subjectively perceived visual realism is more difficult to quantify and assess than other aspects of displays.

There's a small delay between looking at an object and the display moving to adjust focus, but as with the retina resolution aspect, the purpose of this prototype is to demonstrate what zooming feels like, not to claim specific technology to make it happen. Butterscotch Varifocal actually uses the actuator of the original Half-Dome. At the end of 2019, Meta also announced the Half-Dome 2 with faster and more reliable actuators and the Half-Dome 3 without mechanical parts. Change to adjustable lenses.

Dynamic Distortion Correction five times doesn't get much attention in Butterscotch Varifocal, but is also very important for realistic VR. In addition to the impressive angular resolution and dynamic focus, I also noticed that the Butterscotch Varifocal has excellent optical fundamentals, with no pupil swimming or other geometric distortions. After trying it out, I found out it's because it has dynamic distortion correction.

The lenses in modern VR headsets magnify the display to a relatively wide field of view, but this causes geometric distortion. A key innovation of Palmer Luckey's original prototype was to correct this problem in software by outputting distortion in the output image opposite to that of the display. However, optical distortion changes slightly based on where your eye is relative to the lens, and this software distortion correction only applies to the center position. Dynamic distortion correction means that the system generates a new correction every frame based on your eye position, which requires eye tracking support.

I've been told it's a computationally cheap technology, so I asked Lanman why the Quest Pro didn't use it, since it also has eye tracking. While he didn't answer the question directly, he talked about the importance of eye-tracking hardware for dynamic distortion correction to work well, since it requires measuring exactly where your pupils are in 3D space, not just the direction you're looking. It's worth noting that the Quest Pro has only one tracking camera per eye, while the Butterscotch Varifocal has two per eye, just like the Apple Vision Pro. VisionPro一样。

Meta described retina resolution as "our product roadmap" last year, so when will we see these technologies in actual products?

Lanman, of course, won't answer the question, since he's just a "researcher," but he's skeptical of the complexities of multi-monitor approaches like Varjo's. Achieving central-view retinal resolution in a headset with a standard field-of-view would require about a 5K display per eye, assuming retinal resolution will be achieved through raw pixel density. As the field of view widens, this need becomes even higher – over 10K per eye for full human field of view, and 16K per eye for retinal resolution across the entire field of view rather than just the center.

The Apple Vision Pro is expected to launch next year with an OLED microdisplay of about 3.5K per eye, although such microdisplays are reportedly very difficult to manufacture and low yields are a major reason for its $3,500 price. However, market demand for such high-resolution microdisplays is just beginning to emerge, and as display technology companies compete to find better manufacturing techniques, it is reasonable to expect prices to fall, yields to increase, and resolution to change over time. get higher.

So what about zoom? In a presentation in early 2020, Lanman described Half-Dome 3's electronic zoom approach as "ready for the market" and at a higher "technology maturity level" than any previous prototype. Last year, Mark Zuckerberg hinted that Zoom could arrive in the "second half of the decade," between 2026 and 2029.

Demonstrating the original Butterscotch and Starburst (which we tried), Lanman described his team's goal as a future display system that would pass the "Visual Turing Test," meaning it would look like a clear glass mask, while the Not the display.

The Butterscotch zoom doesn't quite pass such a test. Unlike the Starburst, its traditional LCD display doesn't offer real-world brightness, dynamic range, or contrast. But it delivers detail and clarity that alone are stunning. It's another reminder that today's headsets are just the beginning for virtual reality, which has a long and promising future.