Meta Patent Introduces AR/VR In-Ear Optical Brain Imaging Device to Assess User Cognitive Load

CheckCitation/SourcePlease click:XR Navigation Network

(XR Navigation Network 2023年12月18日）功能性近红外光谱（Functional Near-Infrared Spectroscopy/fNIRS）是一种光学脑成像技术。它的原理是将光照射到用户的头部，然后通过比尔-朗伯定律原理比较不同波长的光吸收，从而估计大脑皮层的血流动力学变化。

与头部的其他组织不同，神经组织中的氧合血红蛋白（HbO）和脱氧血红蛋白（HbR）的血流动力学变化属于时间反相关。所以，fNIRS可用于从HbO和/或HbR痕迹估计神经脑组织的反应性。

In other words, because oxygenation levels change as brain regions become more active, brain activity can be recognized in real time by detecting changes in blood oxygenation using the fNIRS device.

然而，传统的fNIRS设备体积庞大，不适合用于便携式可穿戴设备设置。所以在名为“In-ear functional near-infrared spectroscopy for cognitive load estimation”的专利申请中，MetaAn in-ear fNIRS device that can be used for XR and is primarily used to assess the cognitive load of the user is presented.

The system may further include an IED with an in-ear device. one or more of the IEDs may include one or more fNIRS optoelectronic devices, each optoelectronic device including one or more light emitting light sources and one or more detectors. the fNIRS optoelectronic devices may be used to capture fNIRS signaling data representative of changes in hemodynamics in the user's brain.

In one embodiment, the set of fNIRS optoelectronic devices may include a set of mutually inverted fNIRS optoelectronic devices, capturing the fNIRS signals in each other's (bi-directional) rather than uni-directional directions and correcting for measurement errors by comparing the bi-directional fNIRS signal data and subtracting noise from the true neural signal.

In one embodiment, the system may further include one or more electrodes for measuring electrical signals from the brain and correcting the fNIRS signal data by filtering out signals with systematic sources from fNIRS signals that have neural origins and represent real brain activity.

FIG. 1 is a schematic diagram 100 illustrating an in-ear fNIRS signal data measurement technique. schematic diagram 100 illustrates a set of fNIRS photodiodes 106 of an in-ear fNIRS device placed within a user's ear canal 118 near a user's eardrum 120. a set of fNIRS photodiodes 106 may include at least one source photodiode that may be used to generate fNIRS signal data 106A and at least one detector photodiode 106B. said devices may provide continuous, unobtrusive monitoring of cognitive load.

As shown in FIG. 1, the source photodiode 106A and detector photodiode 106B of the FNIRS photodiode 106 define a curved optical path 107, and said optical path 107 captures brain activity 118 from within the ear canal.

Light travels along a curved optical path 107 from the light source photodiode 106A through the scalp and then through the skull and into the user's brain tissue. The light then travels back through the skull and scalp to the detector photodiode 106B.Absorption of this light by the detector photodiode 106B can be used to estimate HbO and HbR, and thus neural activity in the brain can be estimated.

More specifically, according to the principle of Beer-Lambert law, HbO and HbR absorb light differently as a function of wavelength. As light passes through tissue, saturated and unsaturated hemoglobins absorb light at different frequencies. For example, fully desaturated hemoglobin absorbs red light (e.g., 630 nm) and fully saturated hemoglobin absorbs infrared light (e.g., 940 nm).

Therefore, measurement of light absorption at two or more wavelengths can be used to estimate HbO and HbR concentrations. Based on the estimated HbO and HbR concentrations, neural activity in the brain can be estimated as NIRS signaling data.

The spacing between the source photodiode 106A and the detector photodiode 106B controls the recording depth. That is, the farther apart the source photodiode 106A is from the detector photodiode 106B, the deeper the penetration of the curved optical path 107 into the brain tissue and the deeper the depth of recording. In contrast, the farther apart the source photodiode 106A is from the detector photodiode 106B, the poorer the signal quality of the fNIRS signal captured at the detector photodiode 106B due to more light scattering along the optical path.

Each detector photodiode 106B may be a near-infrared light detector configured to detect near-infrared light. For example, the detector photodiode 106B may be a photodetector. Each detector photodiode 106B may be configured to detect one or more predetermined wavelengths of light in the near-infrared range. For example, the detector photodiode 106B may be configured to detect a first wavelength of light. Additionally or alternatively, the detector photovoltaic device 106B may be configured to detect a second wavelength of light.

The perspective schematic of FIGS. 2A-2C shows different exemplary configurations of groups of fNIRS optoelectronic devices embedded in the IED 200.

As shown in FIG. 2A, the IED 200A may include a set of near-infrared photovoltaic devices 106. the devices may include two source photodiodes 106A and a detector photodiode 106B. the IED 200A may unilaterally record near-infrared spectral signal data. That is, the IED 200A may record near-infrared signal data based on a curved optical path extending from the source photodiodes 106A to the detector photodiode 106B.

Said light source photodiode 106A is configured to emit two different wavelengths of near-infrared light (e.g., a first wavelength and a second wavelength), respectively, and said detector photodiode 106B is configured to detect both wavelengths of near-infrared light.

In another embodiment, the IED 200 may comprise a set of fNIRS photodiodes 106 comprising a source photodiode 106A and a detector photodiode 106B. said source photodiode 106A is configured to generate one or more wavelengths of near-infrared light and said detector photodiode 106B is configured to detect one or more wavelengths of near-infrared light.

In the example configuration of FIG. 2A, the source photodiode 106A may be time-multiplexed and time-synchronized with the detector photodiode 106B. That is, the photodiodes may be sequentially powered up to emit two different wavelengths of light and the detector photodiodes may be time-synchronized and configured to detect light of the corresponding wavelengths. The detected light may be used to generate near-infrared spectral signaling data.

As shown in FIG. 2B, the fNIRS signaling data may be recorded in both directions rather than unilaterally. In order to capture fNIRS signal data in both directions, the IED 200B may be embedded with a set of mutually inverted fNIRS photodiodes 106 ', the photodiodes 106 ' being mutually inverted with another set of fNIRS photodiodes 106 '.

That is, the IED 200B of FIG. 2B includes a set of reciprocally inverted fNIRS photodiodes 106 ' comprising a source photodiode 106A ' and a detector photodiode 106B '. The set of reciprocal fNIRS photodiodes 106 ' may be configured to capture the reciprocal fNIRS signal data in substantially the same region as the fNIRS signal data recorded by the first set of fNIRS photodiodes 106.

That is, the curved optical path of the first set of fNIRS photodiodes 106 covers substantially the same area as the curved optical path of the penultimate set of fNIRS photodiodes 106, but light flows through said area in the opposite direction.

The reciprocal group of fNIRS photodiodes 106 ' may be configured to capture reciprocal fNIRS signal data time interleaved with the data captured by the NIRS photodiodes 106 to increase the signal-to-noise ratio SNR of the captured fNIRS signal data. by embedding reciprocally inverted fNIRS photodiode groups 106 ', it is possible to dual feedback is achieved, the maximum SNR is realized using dual photodiode sets on each side of the in-ear module.

The first set of source photodiodes 106A and detector photodiodes 106B may be time-multiplexed or spectrally multiplexed and time-synchronized to generate NIRS signal data. Similarly, the source photodiode 106A ' and detector photodiode 106B ' of the mutually inverted set may be time multiplexed or spectrally multiplexed and time synchronized to generate mutually inverted fNIRS signal data. Said mutually inverted set may capture mutually inverted set fNIRS signal data time interleaved with said fNIRS signal data captured by said first set of photodiodes 106.

In one embodiment, more than two wavelengths may be emitted and captured to generate raw, unfiltered fNIRS signal data. A smaller number of wavelengths than the number of wavelengths used for the first set of raw, unfiltered fNIRS signal data may be used to generate the fNIRS signal data for the reciprocal set of fNIRS optoelectronic devices 106 '. For example, two different wavelengths may be used to generate the first set of raw, unfiltered fNIRS signal data, whereas the mutually inverted fNIRS signal data of the mutually inverted set of fNIRS optoelectronic devices 106 ' for noise estimation may be generated using only one of the two different wavelengths.

As shown in FIG. 2C, the IED 200C may include a plurality of groups of fNIRS optoelectronic devices 106 to generate filtered fNIRS signal data. For example, the plurality of fNIRS optoelectronic devices 106 may be arranged as an array of source optoelectronic devices 106A extending from one longitudinal end side of the IED 200C to the other, and an array of detector optoelectronic devices 106B arranged as an array of detector optoelectronic devices extending from one longitudinal end side of the IED 200C to the other.

The source photovoltaic device 106A and the detector photovoltaic device 106B may alternate along the length of the IED 200C, and the photovoltaic devices may be driven with time multiplexing or spectral multiplexing to generate fNIRS signal data for each of the plurality of fNIRS photovoltaic devices 106 that may have the same or different SDSs.

As mentioned previously, if the distance from the detector to the source decreases, the depth of penetration decreases. In one embodiment, brain activity can be captured at different depths in the temporal cortex when filtered fNIRS signal data is generated. Thus, the photodiodes of the IED 200C can be driven as virtual photodiodes in different combinations corresponding to multiple sets of fNIRS photodiodes 106 to generate fNIRS signal data corresponding to respective sets of different penetration depths.

FIG. 3 is a block diagram of a cognitive load estimation system 300. The cognitive load estimation system 300 may include an IED 301 and a cognitive load estimation device 350.

The IED 301 is mounted in the user's ear canal 118 proximate to the tympanic membrane 120 and captures various types of data from within the ear canal 118.

As shown in FIG. 3, the IED 301 may include an audio sensor 302, one or more EEG electrodes 304, a set of fNIRS optoelectronic devices including a source optoelectronic device 106A and a detector optoelectronic device 106B, an acoustic sensor 308, a motion sensor 310, a controller 312, a battery 314, a communication interface 316, and an acoustic sensor 324.

The audio transducer 302 is a speaker that generates sound from the audio data and outputs the sound into the ear canal 118. The audio transducer 302 may be used to present an audio signal to a user.

The one or more EEG electrodes 304 capture electrical charges generated by brain cell activity in the user's brain. The one or more EEG electrodes 304 may use the principle of differential amplification by recording voltage differences between different points, where the different points compare one active probe electrode location to another neighboring or distant reference electrode.

The electrical signals captured by the EEG electrodes 304 may be used to generate EEG signal data defining waveforms over time, and the waveforms represent electrical activity occurring within the user's brain.

The controller 312 may control one or more of the EEG electrodes 304 to receive electrical signals captured by the EEG electrodes 304. The controller 312 may include a differential amplifier to amplify differences between voltage signals detected at the EEG electrodes 304. The controller 312 may also include an analog-to-digital converter that converts the electrical signals from the EEG electrodes 304 into EEG signal data representing brain activity of a user.

As another example, the controller 312 may control the source photovoltaic device 106A and the detector photovoltaic device 106B of said photovoltaic device set to receive an electrical signal corresponding to the intensity of light detected by said detector photovoltaic device 106B. The analog-to-digital converter of the controller 312 may then convert the electrical signal corresponding to the intensity of the light detected by the detector photovoltaic device 106B into unfiltered fNIRS signal data.

EEG and near-infrared spectroscopic measurement techniques can be combined to subtract noise from true neural signals. In addition, by using this dual-mode (i.e., fNIRS+EEG) approach, it is possible to separate the portion of the fNIRS signal data that may have a neural origin from the portion that may have a systemic origin.

To implement this dual-mode approach, the controller 312 may be configured to time-synchronize the operation of the fNIRS optoelectronic device 106 of the system 300 to capture electrical signals representing fNIRS signal data with the operation of the EEG electrodes 304 of the system 300 to capture electrical signals representing EEG signal data.

The cognitive load estimation device 350 may estimate a cognitive load of the user. Said cognitive load estimation device may include a controller 360. in some embodiments, the cognitive load estimation device 350 may also include one or more fNIRS optoelectronic devices 106, one or more EEG electrodes 304, or some combination thereof.

In one embodiment, a cognitive load estimation device 350 may receive data from an IED 301 over a network 370. Said cognitive load estimation device 350 may further filter said fNIRS signaling data, estimate a cognitive load of said user based on said filtered data, and perform an action based on said estimated cognitive load of said user.

In one embodiment, the cognitive load estimation device 350 is a head-mounted display.

FIG. 4A is a headset 400 in the form of an eyewear device. said headset 400 is an example of said cognitive load estimation device 350. The frame 410 may include one or more biometric sensors. Said biometric sensors may include one or more fNIRS photoelectrodes 106, one or more EEG electrodes 304, or some combination thereof.

In one embodiment, the head-up display 400 may be configured to generate EEG signal data based on electrical signals captured by the EEG electrodes 304 in the IED 301 and based on electrical signals captured by the EEG electrodes 304 in the head-up display 400. The EEG electrodes 304 of the head-up display 400 may replace the EEG electrodes 304 of the IED 301 to capture electrical signals to generate EEG signal data.

FIG. 4A further illustrates two sets of fNIRS photodiodes 106 at the temple tips on either side of the user's head. each photodiode set may be mounted so as to be in contact with the user's anatomy when the user wears the head-mounted head unit 400.

In one embodiment, the head-up display 400 may be configured to generate additional fNIRS signal data based on the two sets of fNIRS optoelectronic devices 106, and the cognitive load estimation device 350 may be configured to generate filtered fNIRS signal data based on the fNIRS signal data generated by the two sets of fNIRS optoelectronic devices 106 configured to the IED 301. and additional fNIRS signal data generated based on the set of fNIRS optoelectronic devices 106.

FIG. 4B is a head-mounted device 405 in the form of a head-up display. the head-up display includes a display assembly, a DCA, an audio system, EEG electrodes, EOG electrodes, and a position sensor 490. fig. 4B illustrates an illuminator 440, a plurality of speakers 460, a plurality of imaging devices 430, a plurality of acoustic sensors 480, and a position sensor 490.

名为“In-ear functional near-infrared spectroscopy for cognitive load estimation”的Meta专利申请最初在2022年5月提交，并在日前由美国专利商标局公布。

Generally speaking, after a U.S. patent application is examined, it will be automatically published 18 months from the filing date or priority date, or it will be published within 18 months from the filing date at the request of the applicant. Note that publication of a patent application does not mean that the patent is approved. After a patent application is filed, the USPTO requires actual review, which can take anywhere from 1 to 3 years.