Microsoft patent shares how to adjust the optical path performance of AR/VR headsets

CheckCitation/SourcePlease click:XR Navigation Network

Adjusting the performance of an illumination beam path comprising a plurality of light sources and an optical imaging path comprising a selective reflection imaging device

(XR Navigation Network November 21, 2023) LEDs are very sensitive to temperature. Unless a corrective feedback system is implemented, the output power may drift over time and the output wavelength may drift with temperature and current settings. The reflectivity of spatial light modulators is also prone to drift due to temperature, resulting in suboptimal performance characteristics.

If a user wearing a near-eye display system moves from one ambient temperature to another, or if the display system becomes warm during operation time, or if the display system ages, or if the battery power drops, maintaining the same performance profile may result in a change in the output image.

所以在名为“Methods for adjusting display engine performance profiles”的专利申请中，MicrosoftAn accurate LED power feedback method that does not add a lot of calibration time to the manufacturing process is introduced. Specifically, the company proposes methods for adjusting the performance of an illumination beam path that includes multiple light sources and an optical imaging path that includes a selective reflective imaging device.

One or more photodiodes may be positioned to capture light reflected from the selective reflective imaging device. By commanding the selective reflective imaging device to operate at a predetermined reflectivity and commanding each light source to emit a pulse of light, the output power of the light source and the reflectivity of the selective reflective imaging device can be determined based on the photodiode readout. The performance profiles of the light source and selective reflective imaging device can then be adjusted accordingly.

2 illustrates an example near-eye display system 200 including a display engine 201 and projection optics 202. Display engine 201 includes illumination beam path 205 and optical imaging path 210 . Controller 215 may operate active components coupled to illumination beam path 205 and optical imaging path 210 . For example, the controller 215 can provide appropriate control signals to form a desired display image.

Illumination beam path 205 may include a light source cluster 225 containing one or more light sources. Illumination beam path 205 may further include optical element 235 , which may be configured to produce uniform illumination of selective reflective imaging device 240 included in optical imaging path 210 .

Reflective and/or refractive optical elements may be used to allow illumination beam path 205 and optical imaging path 210 to be positioned in a non-telecentric manner. This both reduces problems due to reflections and glare and enables a functional configuration that accommodates other components.

Optical imaging path 210 is configured to form a display image and release the display image through exit pupil 245 . Selective reflective imaging device 240 may be a reflective LCOS or DMD device, or other suitable device that can selectively adjust the reflectivity angle.

As an example, in the fully open state of the pixel, the reflective element may be deflected such that light incident on the element is reflected into the exit pupil 245 . In the fully closed state, the reflective element may be deflected such that incident light is reflected out of the exit pupil 245, such as back into the illumination beam path 205.

In one embodiment, an LCOS array may be utilized in which polarization-rotated liquid crystals are positioned over a rectangular array of passivated highly reflective pixel elements. . The LCOS array is a regular two-dimensional liquid crystal element array. Elements can share an optically transparent front electrode while providing each element with an individually addressable reflective rear electrode.

In other examples, two electrodes are addressable for each pixel. Electrical bias applied to the elements of the array changes the alignment of the liquid crystals therein, enabling the elements to act as polarizing filters for illumination reflected from the rear electrode.

In this way, a controlled polarization state of light emerges from each pixel. The unwanted polarization component can be removed as it passes through a common front polarizer array for all elements. This action converts the encoded polarization state of light from each element into the corresponding reflected intensity from the element. A suitable array driver provides control data to determine the bias level of each element, thereby defining the image reflected from the array.

In embodiments implementing a DMD array, an individually deflectable mirror element may be provided for each pixel of the display image. In other embodiments, the selective reflective imaging device 240 may take the form of a FLCOS array or a holographic spatial light modulator SLM with reduced polarization state switching delay.

Optical imaging path 210 may be offset from the user's field of view of near-eye display system 200 . Figure 3 shows display engine 300, which includes illumination beam path 301 and optical imaging path 303.

Illumination beam path 301 includes light source cluster 305 including green 306, blue 307 and red 308 light sources controlled by controller 309. Green light from green light source 306 passes through optics 310, 311, and 312 to beam combiner 314.

Likewise, light from blue light source 307 and red light source 308 passes through optics 316, 318, 320 to beam combiner 314. Beam combiner 314 combines the red, green, and blue light into a beam and directs the beam to microlens array MLA 322.

Microlens array 322 homogenizes a single beam and at least partially shapes the beam to provide illumination to downstream optical elements more uniformly and to remove artifacts. Microlens array 322 outputs the light to prism 324, which directs the light to lens 326.

Prism 324 allows non-telecentric illumination of optical imaging path 303, tilting the output of the illumination source. Prepolarizer 327 may be optically located in front of polarizing beam splitter PBS 328 of optical imaging path 303 such that light entering PBS 328 is polarized. PBS 328 then directs the light toward selective reflective imaging device 332, which is controlled by controller 309 to form an image.

A lens may optionally be located between PBS 328 and selective reflective imaging device 332. The image light from the selective reflection imaging device 332 returns through the lens 330 and the polarizing beam splitter 328 , is reflected from the mirror 334 , and then emerges from the quarter wave plate 336 . The light directed from quarter wave plate 336 then passes through lens 340 toward other projection optics 342 for display.

An additional sensor, such as a photodiode, may be included in the illumination beam path 301 to characterize the current aspect of one or more optical elements included in the display engine 300 . As an example, PD 350 is positioned to capture light reflected from selective reflective imaging device 332 back into illumination beam path 301 .

For example, when selective reflective imaging device 332 is turned off, a significant amount of light returns to illumination beam path 301 because the light is not polarized in a manner that passes through polarizing beam splitter 328 . When selective reflective imaging device 332 is turned on, since the effectiveness of selective reflective imaging device 332 is less than 1001 TP3T, most of the reflected light passes through polarizing beam splitter 328, but not all.

Photodiode PD 351 is positioned to capture the residual light from beam combiner 314, which makes it very sensitive to wavelength shifts due to its coating.

In one embodiment, the one or more photodiodes include two or more photodiodes with different wavelength filters. Therefore, the controller can be configured to determine the wavelength of each light source based on the readout of two or more photodiodes with different wavelength filters when commanding each light source to emit a pulse of light.

One or more temperature sensors may be located in light source cluster 305. Therefore, the controller can be configured to determine the wavelength of each light source based on readings from one or more temperature sensors and LED current settings.

FIG. 4 shows an alternative configuration for display engine 400 that includes illumination beam path 401 offset from optical imaging path 403 . The illumination beam path 401 includes a light source cluster 405 that includes a green light source 406, a blue light source 407, and a red light source 408, and is controlled by a controller 409.

Green light from green light source 406 reaches beam combiner 414 through optics 410, 411, and 412, while light from blue light source 407 and red light source 408 reaches beam combiner 414 through optics 416, 418, 420.

Beam combiner 414 combines the red, green and blue light into one beam and directs the beam to folding mirror 421 which reflects the single beam to MLA 422. MLA 422 outputs light to lens 426. Lens 426 then directs the light into optical imaging path 403 .

By using folding mirrors, the optical imaging path 403 can be offset from the illumination beam path 401. Prepolarizer 427 is located optically in front of polarizing beam splitter 428 in optical imaging path 303 to polarize light entering PBS 328. PBS 428 of optical imaging path 403 directs the light through optional lens 430 to selective reflective imaging device 432, which is controlled by controller 409 to form an image.

According to the optical imaging path 303, the optical imaging path 403 further includes a mirror 434, a quarter wave plate 436 and a lens 340, which focus the light onto the projection optical element 442 for display.

Photodiodes may be included in the illumination beam path 401 to characterize the current aspect of one or more optical elements included in the display engine 300 .

As an example, photodiode PD 450 is positioned to capture light reflected back from selective reflective imaging device 332 to illumination beam path 301 . Additional photodiodes 452 may be optically positioned behind the folding mirror 421 . The output of photodiode 452 may be sensitive to the coating of folding mirror 421 and to any shift in the wavelength of the light output by light sources 406, 407, and 408.

By positioning photodiode PD 452 toward lens 426, light from the light source and light reflected from selective reflective imaging device 432 can be captured. One or more temperature sensors may be located in light source cluster 405.

The photodiodes and temperature sensors may be used to characterize and adjust the performance characteristics of the display engine's light source and selective reflective imaging device.

Figure 5 illustrates an example method 500 for calibrating a near-eye display device.

At 510, method 500 includes commanding the selective reflective imaging device to operate at a predetermined reflectivity. The predetermined reflectivity may be a reflectance relative to one or more photodiodes. In other words, based on the relative positioning of each photodiode, the first predetermined reflectivity of the selective reflective imaging device may be commanded to reflect light out of the photodiode.

At 520, method 500 includes commanding the light source to emit light pulses while the selective reflective imaging device is operating at a predetermined reflectivity. At 530, one or more photodiodes are read while the selective reflective imaging device is operating at a predetermined reflectivity.

At 540, the performance profile of the one or more light sources and the selective reflection imaging device based on the readout of the one or more photodiodes is adjusted. For example, the driving power of the light source can be adjusted. In this way, the light source can be calibrated to a specified wavelength and intensity. Additionally or alternatively, the drive signal and/or one or more additional operating parameters of the selectively reflective image forming device may be adjusted.

Optionally at 550, when each light source is commanded to emit a pulse of light, the wavelength of each light source is determined based on readings from one or more sensors. For example, one or more temperature sensors may be located within or proximal to the light source cluster. The controller may be further configured to estimate, infer, and/or determine the wavelength of each light source based on readings from the one or more temperature sensors when commanding each light source to emit a pulse of light.

Figure 7 schematically illustrates a workflow 700 for determining an LED wavelength based at least on photodiode readings.

White balance model 705 may include an LED voltage predictor 720 configured to approximate the voltage of an LED based on LED drive current 710 and light source cluster temperature 712 .

Photodiode reader 708 may be used as the input of the photodiode count to optical power calculator 723. 723 The optical power can be determined directly from the photodiode readout 708 by any suitable method. The calculated optical power can then be output as LED optical power714.

The LED optical power, the determined LED electrical power, and the light source cluster temperature 712 may then be used as inputs to an LED efficiency calculator 724 . LED optical power may allow the LED efficiency calculator 724 to be determined accurately rather than by an estimate based on previous calculations because thermal power equals electrical power minus optical power. LED efficiency can then be used to determine LED thermal power and LED optical power.

Can output certain optical power. The determined LED thermal power and light source cluster temperature 712 may then be used as inputs to the LED housing temperature predictor 726 . The case temperature can be determined by any suitable method, such as using the Forster thermal model to simulate heat transfer through the semiconductor.

The predicted LED case temperature and LED thermal power can then be used as inputs to junction temperature calculator 728 . This junction temperature can be determined empirically for each LED type and is included in the LED's specification data sheet. The junction temperature can be stored in a lookup table. The determined LED junction temperature can then be used as an input to the dominant wavelength shift calculator 730.

Next, the dominant wavelength shift calculator 730 may determine the dominant wavelength of the LED, such as by performing regression on previously measured data. The determined wavelength can be output as the dominant LED wavelength 716.

Figure 8 illustrates a method 800 for calibrating the optical power of one or more LEDs. By precisely calibrating each LED, more accurate colors can be displayed on the monitor.

Method 800 may be performed by a controller. Method 800 may be performed in response to an instruction to perform a first calibration step. For example, the instruction to perform the first calibration step may include an instruction to perform the first calibration step during a warm-up phase of the near-eye display device. Alternatively, the instruction to perform the first calibration step may include an instruction to perform the first calibration step in response to a threshold change in ambient temperature during an operating phase of the near-eye display device.

Additionally, the instructions to perform the first calibration step include instructions to perform the first calibration step in response to exceeding a threshold operating time during operation of the near-eye display device. In this way, the LED power supply can be calibrated due to changes in external temperature, ambient temperature, battery charging, regular maintenance and other factors.

At 810, the LCOS panel is commanded to display a black frame, the LCOS panel being within the optical imaging path. For example, an LCOS panel might power off for a predetermined duration, such as a single frame of displayed content. In a closed configuration, the LCOS panel can reflect most of the light back into the illumination beam path, with very little light directed toward the near-eye display's projection optics.

Therefore, when the user is viewing a virtual image, the LCOS panel can be commanded to display black frames with little impact on the user experience. Because image light can be integrated over several frames, users may only perceive a brief, slight dimming of the display, rather than a long period of black image.

At 820, one or more LEDs are commanded to emit light pulses while the LCOS panel is commanded to display a black frame, the LEDs being positioned within an illumination beam path configured to produce uniform illumination of the LCOS panel. In this way, the LED power reflected from the LCOS panel back into the illumination beam path can be directly measured, thereby measuring the amount of light incident on the LCOS panel.

At 830, one or more photodiodes are read out to capture light reflected from the LCOS panel. As the LED junction temperature changes with ambient temperature, LED current, or LED duty cycle, the wavelength of the LED can shift by a few nanometers in any direction. However, the photodiode response can be set to be effectively independent of the LED output wavelength within a few nanometer shifts. Therefore, photodiodes can measure actual optical power with high accuracy.

At 840, when the LCOS panel is commanded to display a black frame, the optical power of the one or more LEDs is determined based on the readout of the one or more photodiodes. Optical power can be determined empirically and/or through lookup tables of photodiode readings and environmental conditions. This method for LED power determination is directly based on the amount of incident light reflected off the LCOS panel and is therefore highly relevant to display calibration.

At 850, method 800 includes adjusting the output power of the LED based on the determined optical power. For example, the output power of one or more LEDs can be increased or decreased to balance the output of the light group to produce the desired brightness and coloration of the displayed image.

The calibration procedure can be used for different LED duty cycle settings, such as a range or duty cycle setting or across two or more discrete duty cycle settings. For example, a light source can be commanded to emit pulses of light on a duty cycle pattern.

In duty cycle mode, one or more photodiodes can be read out per duty cycle. The output power of the light source can be adjusted based on the readout of one or more photodiodes per duty cycle in duty cycle mode. Between calibration phases, the output power of one or more LEDs can be adjusted using gradients of output power and duty cycle.

Figure 9 illustrates an example scenario 900 for display engine 300 when selective reflective imaging device 332 is turned off.

Light emitted from one or more light sources is combined in a beam combiner 314 and then directed through a microlens array 322 to a prism 324 . As shown by arrow 902, the light emitted by prism 324 is directed to polarizing beam splitter 336 through lens 326 and pre-polarizer 327, and pre-polarizer 327 is reflected to selective reflection imaging device 332, as shown by arrow 904.

When selective reflective imaging device 332 is turned off, light does not repolarize. Instead, it reflects back to polarizing beam splitter 336, as indicated by arrow 906. Instead of passing through polarizing beam splitter 336, the light is reflected back into the illumination beam path, as indicated by arrow 908. Therefore, photodiode 350 is positioned to detect this reflected light. The photodiode 350 readings can then be used to evaluate optical power, reflectivity and/or other parameters.

As discussed above, measurements taken when the selective reflection imaging device is powered off can be combined with measurements taken when the selective reflection imaging device is powered on to determine a more complete picture showing engine performance.

Figure 10 illustrates an example method 1000 for calibrating reflectance of a selectively reflective image forming device. Method 1000 may be performed in response to an instruction to perform a second calibration step. For example, the second calibration step may be performed after the first calibration step, such as method 800. The instruction to perform the second calibration step includes an instruction to perform the second calibration step in response to a near-eye display device warm-up period, in response to an ambient temperature change, in response to device temperature or operating time, or the like.

At 1010, method 1000 includes commanding the LCOS panel to display a white frame, the LCOS panel being located within the optical imaging path. When the LCOS is powered on and displays a white frame, light is sent to the user through the projection optics. However, instead of the light from the 100% being sent to the user, a considerable amount of light is reflected back to the lighting module.

At 1020, one or more LEDs are commanded to emit light pulses while the LCOS panel is commanded to display a white frame, the LEDs located within the path of the illumination beam being configured to produce uniform illumination of the LCOS panel. Since the light pulses will be directed into the projection optics, the user will most likely notice this calibration step.

Accordingly, method 1000 may be performed during warm-up and/or after providing a calibration warning to the user. However, if the calibration is performed as a single frame, the user may only experience a brief increase in background brightness, so advance warning may not be needed.

At 1030, one or more photodiodes are read out to capture light reflected from the LCOS panel. In an ideal world, when the LCOS is turned on, all light enters the exit pupil and no light is reflected back into the illumination beam path. In practical applications, LCOS cannot achieve perfect reflection, and a certain amount of light will be incident on the photodiode.

When powered, approximately 10-50% of incident light is expected to be reflected back to the lighting module due to the non-perfect reflectivity of the LCOS. When the LCOS is turned off, approximately 90% or more light is reflected back into the illumination beam path and incident on the photodiode.

At 1040, when the LCOS panel is commanded to display a white frame, the reflectivity of the LCOS panel is determined based on the readout of the one or more photodiodes.

At 1050, the driving parameters of the LCOS panel are adjusted according to the determined reflectivity. Determined reflectivity can allow LCOS driving parameters to be adjusted to maximize display efficiency, e.g. optimally reflecting light without wasting additional battery power.

Related patents:Microsoft Patent | Methods for adjusting display engine performance profiles

Named "Methods for adjusting display engine performance profiles"Microsoft patentThe application was originally submitted in April 2022 and was recently published by the US Patent and Trademark Office.