Microsoft XR patent proposes lightweight way to collect telemetry data and update 3D mapping

CheckCitation/SourcePlease click:XR Navigation Network

(XR Navigation Network 2023年12月13日)environmental3D映射是环境中的3D位置和与3D位置相关的捕获图像的集合。所述关联或者来自描述环境中3D位置的landmark的捕获图像，或者来自由具有环境中的3D位置和方向的捕获设备捕获的图像。绘制的环境是任何室内或室外环境，如家庭、办公室内部、花园、公共火车站或其他环境。

遥测数据可以用于映射优化。但由于各种原因，包括隐私、安全、缺乏计算资源、缺乏通信网络带宽等，目前难以获得合适的遥测数据。所以在名为“Collecting telemetry data for 3d map updates”的专利申请中，MicrosoftA way to collect telemetry data and update 3D mapping in a lightweight and privacy-enhancing manner is proposed.

The method includes performing a first relocalization event, wherein data from observations of the attitude tracker at time t1 is used to calculate a first 3D mapping pose of the attitude tracker, and receiving a first local pose of the attitude tracker at time t1 . A second relocalization event occurs in which a second 3D mapping pose of the pose tracker is calculated based on data from the then observation, and a second local pose of the pose tracker at time t2 is received.

A first relative pose between the first and second 3D mapping poses is calculated. A second relative pose between the first and second local poses is calculated. The residual, which is the difference between the first relative pose and the second relative pose, is stored as input to the process for updating the 3D map.

Residual measurements help improve the accuracy of 3D mapping. The residual can also be used to measure the mapping accuracy between two positioning events.

Figure 1 is a schematic diagram of the 3D mapping relocation service 122 for collecting telemetry data to update the 3D mapping. The map update function 118 is deployed using a distributed architecture that includes multiple worker nodes and management nodes that communicate with each other. In one example, worker nodes and management nodes are deployed in a data center, cluster of compute nodes, or other computing function.

The mapping update function receives sensor data captured from one or more sensors over the communications network 114 . The map update function may access one or more 3D maps 106 stored at a location accessible through the communications network 114 .

The 3D map 106 may be accessed through the communication network 114, such as through the hologram sharing and persistence service 120 and through the navigation service 108. In one example, the navigation service 108 can send a query to the 3D map 106 to find image data associated with a given 3D location, or to find the 3D location associated with a given image. In another example, the navigation service 108 can download a region of the 3D map or the entire 3D map to plan a route and provide directions to an end user, such as a user of a head-mounted augmented reality device.

Communication network 114 includes 3D mapping relocation service 122 . Queries can be sent to the 3D mapping relocalization service along with data from pose tracker observations. The data observed by the attitude tracker is the sensor data captured by the attitude tracker or data derived from the sensor data captured by the attitude tracker. Pose tracker observation data is used to query the 3D map to find the 3D map pose.

In a typical scenario, when a pose tracker is turned on and its local tracking technology is initialized from scratch, or when tracking fails for a short period of time, the 3D mapping relocation service 122 is used. Additionally, regular repositioning is necessary. In the examples described herein, 3D mapping positioning service 122 is used at additional times to facilitate calculation of telemetry data 132 .

Figure 2 is an example of 3D mapping. The map includes 3D locations of multiple landmarks. The map simultaneously includes the relative trajectories of the two capture devices. Solid black line 204 represents the trajectory of the first capture device. Solid black line 208 represents the trajectory of the second capture device. The solid points 202, 206 of each trajectory marker are the poses of the respective capture devices. The small black dots 210 shown in and around trajectories 204 and 208 are landmark locations in the environment to be optimized.

Figure 3 is a flowchart of a method of obtaining telemetry data for updating 3D mapping.

The method includes a first relocation event 300 and a second relocation event 306 . In the first repositioning event 300, the first 3D map pose (map_pose_1) of the pose tracker is calculated using the data observed by the pose tracker at time t1. Also in the first repositioning event 300, the first local pose of the pose tracker at time t1 (local_pose_1) is received.

In the second repositioning event 306, the second 3D mapping pose of the pose tracker (map_pose_2) is calculated using the data observed by the pose tracker at time t2. Also in the second repositioning event 306, the second local pose of the pose tracker at time t2 (local_pose_2) is received. The calculation formula of residual: residual = (local_pose_2−local_pose_1)−(map_pose_2−map_pose_1).

At 31, the residuals are stored as input to the process for updating the 3D map. Process 316 shown in Figure 3 is optionally repeated to accumulate multiple residuals, such as hundreds or thousands or more residuals.

Figures 4, 5, 6, and 7 are schematic diagrams of various methods of obtaining telemetry data to update 3D mapping. Figures 4 to 7 are shown as separate methods, they can be combined in any way.

Figure 4 is a schematic diagram of a method of obtaining telemetry data for updating 3D mapping. The pose tracker 400 is represented by a vertical line, as are the 3D mapping repositioning service 402 and the 3D mapping 404. The horizontal lines in Figure 4 represent messages sent between the entities represented by the vertical lines. The relative vertical position of the horizontal lines indicates the chronological order of the messages.

At 406, the observation data of the posture tracker at time t1 is sent from the posture tracker 400 to the 3D mapping repositioning service 402, and is sent together with the local pose (local_pose_1) of the posture tracker at time t1. The data observed by the attitude tracker are features obtained from images or other sensor data captured by the attitude tracker. The 3D mapping relocation service 402 has an associated 3D mapping 404 .

At 408, the 3D map is queried 404 using the attitude tracker's observation data at time t1. At 410, the position in the 3D map can be found in the 3D map repositioning service 402, and the position is map_pose_1 of the pose tracker. This is the first relocation event. The 3D mapped pose is optionally sent back to the pose tracker, where the 3D mapped pose can be used to reposition the pose tracker (.

A second relocation event occurs later. At 414, the pose tracker's observation data at time t2 is sent from the pose tracker 400 to the 3D mapping repositioning service 402, along with the pose tracker's local pose at time t2 (local_pose_2). Data from attitude tracker observations include features from images or other sensor data captured by the attitude tracker.

At 416, the 3D map is queried 404 using the attitude tracker's observation data at time t2. In 418, the position in the 3D map can be found in the 3D map repositioning service 402, and the position is map_pose_2 of the pose tracker.

At 422, the residual is calculated at the 3D relocation service 402 according to the formula (local_pose_2−local_pose_1)−(map_pose_2−map_pose_1). The remainder can be used to update the 3D map stored in the cloud 404. The residuals are optionally sent from the 3D mapping relocalization service 602 to the pose tracker.

Figure 5 is another method of obtaining telemetry data for updating 3D mapping. Figure 5 shows a gesture tracker 500 using vertical lines, a 3D mapping repositioning service 502 using vertical lines and a 3D mapping using vertical lines 504.

At 506, a query is sent from the posture tracker 500 to the 3D mapping relocation service 502 along with the location information associated with the posture tracker 500 at time t1. Based on the location data, 3D map relocation service 502 requests a portion of 3D map 504 from 3D map 504 . The mapping part is obtained by the 3D mapping relocalization service of step 510 and sent to the pose tracker of step 512. In the pose tracker, using the data associated with the pose tracker observation time t1, the position and orientation of the 3D map portion (map_pose_1) that matches the data is found at 514. The pose tracker also determines the local pose at time t1, which is local_pose_1.

At a later time t2, the pose tracker requests a different 3D mapping part. At 316, a query with location information is sent from pose tracker 500 to 3D mapping relocalization service 502. Another mapping part is requested at 518, obtained by the 3D mapping relocalization service at 520, and sent to the pose tracker at 528.

The second 3D mapping pose map_pose_2 is calculated at time 522 using the observation data of the pose tracker at time t2. The closest match is found in the appropriate 3D map section, i.e. the position and orientation in the 3D map section is found that matches the data. The pose tracker also determines the local pose at time t2, which is local_pose_2.

The residual is calculated at 534 and optionally sent to the 3D mapping relocation service at 526, with the residual being (local_pose_2−local_pose_1)−(map_pose_2−map_pose_1).

Figure 6 is another method of obtaining telemetry data for updating 3D mapping. In Figure 6, the pose tracker 600 is represented as a vertical line and the local 3D map 604 is represented as a vertical line and is local to the pose tracker. The 3D mapping relocation service 602 is represented as a vertical line. The 3D mapping pose (map_pose_1) in the 3D mapping used by the 3D mapping repositioning service 602 has data associated with it.

Data is sent to the pose tracker 600 from the 3D mapping repositioning service along with map_pose_1. At 608, 610, 612, the data is used to query the local 3D map 604, i.e., find the position and orientation in the local 3D map stored on the pose tracker that matches the derived features obtained from the 3D map repositioning service (local_pose_1 ). This is the first relocation event.

Later, the 3D mapping repositioning service 602 sends another 3D mapping pose (map_pose_2). At 616, 618, 620, the local 3D map 604 stored in the pose tracker 600 is queried to find the position and orientation matching the derived features (local_pose_2) in the local 3D map stored locally in the position tracker. This is the second relocation event.

The residual is calculated at 622, and the residual is (local_pose_2−local_pose_1)−(map_pose_2−map_pose_1).

Figure 7 is another method of obtaining telemetry data for updating 3D mapping. At 708, data from the pose tracker observation at time t1 is sent from the pose tracker 700 to the 3D mapping relocalization service 702. The pose tracker also uses tracking technology (local_pose_1) to determine the local pose of the pose tracker at t1.

Data from attitude tracker observations are features derived from images or other sensor data captured by attitude tracker 700. The 3D mapping relocation service 702 has an associated 3D mapping 704 in . At 710, 712, the 3D map in the cloud is queried 704 using data from the attitude tracker observation at time tl. At 714, the position and orientation map_pose_1 in the 3D map 704 are found at the 3D map relocation service 702. At 716, the 3D mapped pose is sent to pose tracker 700 and used to calculate the residual. This is the first relocation event.

A second relocation event occurs later. At 720 , the attitude tracker's observation data at time t2 is sent from the attitude tracker 700 to the 3D mapping relocalization service 702 . The pose tracker also uses tracking technology (local_pose_2) to determine the local pose of the pose tracker at t2.

At 722, 724, the 3D map 704 is queried using the attitude tracker observation data at time t2. The correlation data is used to find the closest match in the 3D map, i.e. the position and orientation in the 3D map that matches the derived feature. At 726, the position in the 3D map is found in the 3D map repositioning service 702 and the position is map_pose_2 of the pose tracker. The 3D map pose is sent to the pose tracker 700 at 728.

At 732, the residual is calculated according to the formula (local_pose_2−local_pose_1)−(map_pose_2−map_pose_1). Optionally, the residuals are sent to the 3D mapping relocalization service 702.

Residual measurement helps improve the accuracy of 3D mapping by leveraging local pose tracking and 3D mapping to relocalize the error characteristics of the pose. The invention describes a residual calculation of two relocation events that operates in an unconventional manner to enable efficient collection and calculation of telemetry data.

Named "Collecting telemetry data for 3d map updates"Microsoft patentThe application was originally submitted in April 2022 and was recently published by the US Patent and Trademark Office.

Generally speaking, after a U.S. patent application is examined, it will be automatically published 18 months from the filing date or priority date, or it will be published within 18 months from the filing date at the request of the applicant. Note that publication of a patent application does not mean that the patent is approved. After a patent application is filed, the USPTO requires actual review, which can take anywhere from 1 to 3 years.