"See through the Meta prototype without projection"

在 2023 年的 SIGGRAPH 上，我体验了Meta的研究原型产品Flamera，它是一种无重投影透视增强现实技术。

Quest Pro, Apple VisionPro 和 Quest 3 等不透明头戴设备使用前置高分辨率摄像头显示真实世界，但由于这些摄像头位于不同的位置，因此使用图像处理算法对摄像头视图进行重投影，以显示您的眼睛本应看到的图像。

This reprojection is an imperfect process that increases latency and causes image processing artifacts, although it's getting better as camera and processing chip components improve.

Flamera aims to skip the reprojection approach entirely, instead employing a hardware design built from the ground up for penetration design.

It has 53 lenses in front of each eye, each with an aperture that allows a specific angle of light to enter it. The light from these lenses is then focused onto a conventional image sensor, where it appears as 53 circles. The software then "rearranges the pixels and estimates a coarse depth map to achieve depth-dependent reconstruction," a process Meta says is significantly less computationally expensive than reprojection methods, resulting in lower latency and virtually no visual artifacts.

So how was the experience?

The weird thing about Meta's Flamera headset is that it uses a pretty crappy display system. It's a 60Hz, full-persistence display commonly used in transparent augmented reality systems, with a field of view only about half that of the Quest. This is somewhat ironic given that both the Flamera and Butterscotch Varifocal were launched by Meta's Display Systems Research team.

But Flamera's display system was chosen as a solution to achieve minimal headset thickness using off-the-shelf components. And the focus of the display is not on the display system, but on this novel capture system.

What I see is a see-through view of the real world without any distortion or offset, and the geometry of the object I see through the display is exactly the same as the object I see when I look up. Flamera certainly delivers on its promise.

Still, the poor display distracts from the true merits of the Flamera. For example, the refresh rate means latency is inherently higher, and the resolution and color reproduction result in blurry and washed-out images.

In the future, I'd like to see Meta combine Flamera's capture system with the "Holocake" lenses announced last year to see what's really possible with this capture method when combined with the right display system.

But is it practical?

Meta's Display Systems Research (DSR) team appears to be focused on two separate missions: studying the feel of specific long-term XR display features, and occasionally figuring out the specific technologies that enable those features.

Butterscotch Varifocal, another headset I tried at SIGGRAPH this year, obviously falls into the former category. It's not considered an actual practical way to achieve zoom or retina resolution.

But Flamera seems both an investigation of the sense of perspective penetration without reprojection and a proposal of a specific method. So this begs the question: will we see it in product?

I posed this question to Flamera project lead (and creative mind) Grace Kuo. I learned that the only inherently expensive part of the Flamera approach is the extremely large image sensor required to capture all angles, which also generates a lot of heat – you can see the heatsinks on the sides of the headset.

But there are some potential solutions, such as using a separate smaller image sensor for each lens, or at least using a separate image sensor for a group of lenses.

Another potential future approach could be a compromise between reprojection and Flamera, using fewer lenses to reduce complexity and component cost, and applying a milder reprojection than currently used, thus producing some visual artifacts, although Much less severe than traditional camera methods. This fusion of hardware and software is an area that Mea is interested in researching.

Meta acknowledged that the technology in Flamera "may never make it into a consumer-facing product." But even so, it still demonstrates the spirit of innovation within the company's research team and hints that there may be many non-common solutions to the problems of current VR headsets in the coming decades.