WO2022186256A1 - マップ情報更新方法 - Google Patents

マップ情報更新方法 Download PDF

Info

Publication number
WO2022186256A1
WO2022186256A1 PCT/JP2022/008794 JP2022008794W WO2022186256A1 WO 2022186256 A1 WO2022186256 A1 WO 2022186256A1 JP 2022008794 W JP2022008794 W JP 2022008794W WO 2022186256 A1 WO2022186256 A1 WO 2022186256A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
map information
reprojection error
inference engine
error
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/008794
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
哲也 田中
幸宏 笹川
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Socionext Inc
Original Assignee
Socionext Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Socionext Inc filed Critical Socionext Inc
Priority to JP2023503896A priority Critical patent/JP7782544B2/ja
Publication of WO2022186256A1 publication Critical patent/WO2022186256A1/ja
Priority to US18/341,466 priority patent/US12567344B2/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B29/00—Maps; Plans; Charts; Diagrams, e.g. route diagram
    • G09B29/10—Map spot or coordinate position indicators; Map reading aids
    • G09B29/106—Map spot or coordinate position indicators; Map reading aids using electronic means
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G06T7/55—Depth or shape recovery from multiple images
    • G06T7/579—Depth or shape recovery from multiple images from motion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G09—EDUCATION; CRYPTOGRAPHY; DISPLAY; ADVERTISING; SEALS
    • G09B—EDUCATIONAL OR DEMONSTRATION APPLIANCES; APPLIANCES FOR TEACHING, OR COMMUNICATING WITH, THE BLIND, DEAF OR MUTE; MODELS; PLANETARIA; GLOBES; MAPS; DIAGRAMS
    • G09B29/00—Maps; Plans; Charts; Diagrams, e.g. route diagram

Definitions

  • This disclosure relates to a map information update method.
  • VSLAM Vehicle Simultaneous Localization and Mapping
  • the main processing in the VSLAM technology is the position of the captured landmark in the key frame, the assumed orientation information of the key frame (that is, the position and orientation of the camera), and the key frame calculated from the assumed landmark position. and the reprojection error, which is the error from the reprojection position, which is the position within the is to find the position of
  • map information is referred to as map information.
  • Non-Patent Document 1 Searching for map information that makes the reprojection error 0 is called bundle adjustment, and is generally classified as a nonlinear least-squares optimization problem. Therefore, in bundle adjustment, it is necessary to finely correct the map information so as to reduce the reprojection error, and repeat the correction until the value of the reprojection error converges.
  • Non-Patent Document 1 Non-Patent Document 2.
  • Algorithms using the gradient method are generally used as algorithms for convergence that are required during bundle adjustment.
  • an algorithm using the gradient method for example, an algorithm combining the steepest descent method and the Gauss-Newton method is known. In this algorithm, correction is performed using the steepest descent method until the reprojection error approaches the minimum value, and then correction is performed using the Gauss-Newton method after the reprojection error approaches the minimum value.
  • the present disclosure has been made to solve such problems, and aims to provide a map information update method that can reduce the amount of calculation.
  • a map information updating method provides one or more location information associated with one or more landmarks and one or more keyframes. and one or more attached pose information, wherein each of the one or more keyframes includes at least one of the one or more landmarks.
  • each of the one or more orientation information includes position and orientation information, and a projection relation obtaining step of obtaining one or more projection relations,
  • Each of the above projection relationships includes each of the one or more landmarks, each of the one or more keyframes, and each of the one or more landmarks on each of the one or more keyframes.
  • projection coordinate information corresponding to the coordinates of the projection point on the key frame when projected;
  • the position information the position information associated with one of the one or more landmarks that constitute the projection relationship, and of the one or more posture information, the one or more of the one or more that constitute the projection relationship.
  • a projection error information obtaining step for each of the one or more landmarks, generating a first group of reprojection error information aggregated from all the reprojection error information associated with the landmark; a landmark origin error aggregating step of obtaining a first total value based on all the reprojection error information included in the first reprojection error information group; generating a second group of reprojection error information aggregated from all the reprojection error information associated with the keyframes, based on all the reprojection error information included in the second group of reprojection error information; a keyframe starting point error aggregating step for obtaining a second total value; and for each of the one or more landmarks, position information of the landmark among the one or more position information from the first total value.
  • FIG. 1 is a flow chart showing the flow of the map information updating method according to the first embodiment.
  • FIG. 2 is a schematic graph showing the relationship between map information and reprojection error in bundle adjustment.
  • FIG. 3 is a schematic graph for explaining an outline of prediction map information according to Embodiment 1.
  • FIG. 4 is a flowchart showing a method of calculating prediction map information according to Embodiment 1.
  • FIG. 5 is a schematic diagram illustrating a projection relationship between keyframes and landmarks.
  • FIG. 6 is a schematic diagram for explaining the first reprojection error information group according to the first embodiment.
  • 7 is a schematic diagram for explaining a second reprojection error information group according to Embodiment 1.
  • FIG. 8 is a block diagram showing a functional configuration of the map information updating device according to Embodiment 1.
  • FIG. 9 is a diagram showing an example of the hardware configuration of a computer for executing the method according to each embodiment by software.
  • the map information update method is a method used in VSLAM technology that simultaneously estimates the position of the camera and the positions of surrounding landmarks from the information contained in the keyframes, which are the captured images. is.
  • the map information updating method according to the present embodiment includes one or more position information respectively associated with one or more landmarks and one or more orientation information each associated with one or more keyframes. and update the map information, including Each of the one or more keyframes is a captured image captured to include at least one of the one or more landmarks.
  • Each of the one or more pieces of orientation information includes information on the position and orientation of the camera used for shooting.
  • the first coordinate system is a coordinate system that is fixed with respect to the space in which cameras and the like are arranged, and is also called a world coordinate system.
  • a landmark is a three-dimensional point created in a first coordinate system.
  • a landmark is generated by triangulation based on corresponding feature points included in each of two keyframes. be.
  • map information updating method map information including the estimated positions of the camera and one or more landmarks in the first coordinate system is updated. Specifically, the map information according to the present embodiment is updated when the information of the keyframe captured by the camera is added to the map information and/or when the bundle adjustment is performed.
  • FIG. 1 is a flow chart showing the flow of the map information updating method according to this embodiment.
  • map information is acquired (S20).
  • the map information includes at least position information of one or more landmarks and orientation information of one or more keyframes.
  • projected coordinate information indicating the position of one or more landmarks in the second coordinate system in the keyframes captured by the camera is obtained (S30).
  • the second coordinate system is a coordinate system fixed with respect to the captured image, and is also called a keyframe coordinate system.
  • Projected coordinate information is the position in the second coordinate system of the feature point corresponding to the position of one or more landmarks.
  • the projection coordinate information is information corresponding to the coordinates of the projection points on the keyframes when each of the one or more landmarks is projected onto each of the one or more keyframes.
  • additional map information is generated by adding projection coordinate information to the map information acquired in step S20 (S40).
  • Information related to projection coordinate information may be added to the additional map information.
  • the information related to the projected coordinate information is the estimated positions of landmarks generated based on the feature points included in the keyframes.
  • the information related to the projected coordinate information may be information roughly estimated from the position of the camera or the like.
  • step S40 predictive map information is calculated based on the additional map information generated in step S40, and the map information updated in step S40 is updated to predictive map information (S50).
  • a method for calculating prediction map information will be described.
  • bundle adjustment is generally performed. That is, search is made for map information that makes the reprojection error zero.
  • the reprojection error is information obtained based on position information, orientation information, and projection coordinate information. More specifically, a function for calculating the error between the projected coordinate information and the re-projected position on the captured image corresponding to the projected coordinate information in the projected coordinates, which is calculated based on the map information.
  • the reprojection error is an error calculated using a reprojection error function for one or more landmarks included in the map information, and a reprojection error function for each of one or more keyframes. may include the sum of one or both of the errors calculated using
  • FIG. 2 is a schematic graph showing the relationship between map information and reprojection error in bundle adjustment.
  • the horizontal axis of FIG. 2 indicates the amount of the map information schematically expressed as one variable, and the vertical axis indicates the reprojection error for the map information.
  • bundle adjustment corrects the map information before bundle adjustment to map information that minimizes the reprojection error.
  • a gradient method such as an algorithm that combines the steepest descent method and the Gauss-Newton method
  • Map information that minimizes the reprojection error is searched for by repeating calculation of the reprojection error after correction.
  • the map information that minimizes the reprojection error is hereinafter also referred to as a map information solution.
  • the computational complexity of generating the Hessian matrix and calculating the correction amount by solving simultaneous equations using the nonlinear least squares method for each iteration of correction Includes a lot of processing.
  • iterative calculations need to be performed many times.
  • the present embodiment reduces the amount of computation by using an inference engine for at least part of the prediction map information calculation.
  • An outline of a method for calculating prediction map information according to this embodiment will be described with reference to FIG.
  • FIG. 3 is a schematic graph for explaining an outline of prediction map information according to the present embodiment.
  • map information with a reprojection error close to the minimum value is calculated as predicted map information based on map information using an inference engine.
  • a prediction neural network included in such an inference engine is a trained neural network that has been trained using learning map information as input and updated learning map information as teacher data.
  • a loss function based on the difference between the updated map information for learning and the map information output from the neural network is considered, and the learning proceeds so as to set the loss function to zero.
  • the learning map information is not particularly limited as long as it is the same information as the additional map information used in the map information updating method according to the present embodiment.
  • the updated map information for learning is map information that is generated based on the map information for learning and that reduces the reprojection error calculated using the reprojection error function. Therefore, the loss function in learning may be based on this reprojection error function. Since the reprojection error function can be obtained directly from the (learning) map information, learning in this case does not require updated learning map information as teacher data.
  • the reprojection error function is a function for calculating the error between the projection coordinate information and the reprojection position on the captured image corresponding to the projection coordinate information, which is calculated based on the map information. be.
  • the reprojection error function specifically, for example, a well-known function such as that described in Non-Patent Document 1 can be used.
  • the updated map information for learning can be obtained, for example, by actually performing bundle adjustment using the gradient method on the projected coordinate information for learning and the map information for learning.
  • the map information that reduces the reprojection error may be map information that minimizes the reprojection error, for example.
  • the map information that minimizes the reprojection error is not limited to map information that strictly minimizes the reprojection error, but includes map information that approximately minimizes the reprojection error.
  • the map information that minimizes the reprojection error includes map information in which the difference between the reprojection error of the map information and the minimum value of the reprojection error is 5% or less of the minimum value.
  • the prediction neural network learns the shape of the error function that indicates the relationship between the map information and the reprojection error.
  • Learning of a prediction neural network is a process corresponding to fitting to an error function.
  • the prediction neural network can predict the map information that minimizes the reprojection error.
  • Information such as the position of the camera included in the map information changes according to the map information, but the error function learned by the prediction neural network does not change.
  • the prediction map information may not be map information that minimizes the reprojection error.
  • the predicted map information calculated using the inference engine becomes farther from the solution of the map information (that is, the difference between the predicted map information and the solution of the map information is a difference between the additional map information and the solution of the map information) may be added.
  • an inference engine that predicts a correction direction that approaches the solution of the map information with respect to the additional map information is prepared in advance, and the inference engine determines whether or not the predicted map information is closer to the solution of the map information than the additional map information. You can judge.
  • At least part of the calculations such as solution calculation of simultaneous equations in the conventional gradient method can be replaced with inference using an inference engine. Therefore, in this embodiment, the amount of calculation can be reduced and the parallelism of operations can be increased as compared with the gradient method. Therefore, in this embodiment, effects such as speeding up map information updating and low power consumption can be obtained. Furthermore, inference using an inference engine may also reduce computational accuracy. Therefore, it is also possible to simplify the hardware configuration such as a computer for executing the map information updating method. The details of the calculation method of the prediction map information using the inference engine will be described later.
  • the updated map information is spatially geometrically calculated for the updated map information in step S50, and the updated map information is updated to the updated map information in step S50.
  • map information is updated using an algorithm that combines the steepest descent method and the Gauss-Newton method, for example, to bring the map information closer to the solution of the map information.
  • the reprojection error for the map information is calculated (S70). Specifically, the reprojection error for the map information is calculated using the reprojection error function described above.
  • the convergence of updating the map information updated in step S60 is determined based on the reprojection error calculated using the reprojection error function for the map information updated in step S60. Based on this, it is determined whether to return to the prediction step or update step, or to finish updating the map information updated in step S60 (S80). For example, if the amount of change from the previous determination of the reprojection error (at the time of the first determination, the amount of change from the reprojection error for the prediction map information) ⁇ E is smaller than the predetermined convergence threshold Sc (in S80 ⁇ E ⁇ Sc), it is determined that the map information solution has been obtained, and the update of the map information ends.
  • the process returns to step S50 to calculate the prediction map information again. Further, when the amount of change ⁇ E from the previous determination of the reprojection error is equal to or greater than the convergence threshold Sc and the reprojection error E is equal to or less than the upper limit Su ( ⁇ E ⁇ Sc, E ⁇ Su in S80), Returning to step S60, the map information is updated again using the gradient method.
  • map information update method By using the map information update method described above, the amount of calculation required to update map information can be reduced compared to the case of using conventional technology.
  • FIG. 4 is a flow chart showing a method of calculating prediction map information according to this embodiment.
  • FIG. 5 is a schematic diagram illustrating a projection relationship between keyframes and landmarks.
  • a projection relationship is one landmark, one keyframe, and projection coordinate information corresponding to the coordinates of the projection point on the keyframe when the one landmark is projected onto the one keyframe. relationship.
  • FIG. 5 shows two landmarks LM1, LM2 and two keyframes KF1, KF2.
  • FIG. 5 also shows projection coordinate information C11 and C12 obtained by projecting the landmark LM1 onto the key frames KF1 and KF2, respectively. Coordinate information C21 and C22 are shown.
  • the projection relationship for each of one or more landmarks and each of one or more keyframes is obtained.
  • a correspondence relationship between LM2, the key frame KF1, and the projection coordinate information C21, and a correspondence relationship between the landmark LM2, the key frame KF2, and the projection coordinate information C22 are included.
  • the projection coordinate information is the coordinates of the actual projection point on the keyframe of the landmark projected onto the keyframe, not the coordinates calculated from the map information.
  • reprojection error information is obtained, and the reprojection error information is associated with the projection relation (reprojection error information obtaining step S520).
  • the reprojection error information is inferred from one of the one or more position information, one of the one or more pose information, and the projection coordinate information corresponding to one projection relationship. Contains features indicating the result.
  • the reprojection error is an error spatially and geometrically calculated from the position information, the orientation information, and the projection coordinate information. Instead, a feature quantity indicating the result of inferring this reprojection error is obtained.
  • Feature quantities included in the reprojection error information include, for example, weight vectors and error vectors corresponding to projection relationships corresponding to landmarks and key frames. In other words, the feature quantity includes terms commonly included in blocks including diagonal elements of the Hessian matrix used to calculate the update value of the map information.
  • I ⁇ j is 0 or 1 and indicates the visibility of the ⁇ -th landmark from the j-th camera orientation.
  • R j is a rotation matrix indicating the pose of the j-th camera.
  • W ⁇ j is a 3 ⁇ 2 matrix calculated from the ⁇ -th landmark and the j-th camera pose.
  • e ⁇ j is the difference between the reprojected coordinates and the projection coordinate information (coordinates of the actual projection point), and is a two-dimensional coordinate vector.
  • the elements related to the diagonal blocks of the Hessian matrix and the diagonal blocks of the gradient vector are represented by the following equations (4a) to (5c), where ⁇ is an integer of 1 or more and M or less. be done.
  • x i is the position information of the i-th landmark.
  • t ⁇ is the translation vector of the ⁇ -th camera pose, which is the camera position.
  • the feature amount is calculated based on u ⁇ , R ⁇ W ⁇ , (x ⁇ ⁇ t ⁇ ).
  • u ⁇ represents the reprojected coordinates.
  • R ⁇ W ⁇ represents a weighting factor.
  • (x ⁇ ⁇ t ⁇ ) indicates the relative position of the landmark and the keyframe. Such a feature amount is obtained for each correspondence relationship.
  • the landmark originating point errors are aggregated (landmark originating point error aggregation step S530). That is, for each of one or more landmarks, a first reprojection error information group is generated in which all reprojection error information associated with the landmark is aggregated, and the first reprojection error information A first sum based on all reprojection error information included in the group is determined.
  • the first reprojection error information group will be explained using FIG.
  • FIG. 6 is a schematic diagram for explaining the first reprojection error information group according to this embodiment.
  • FIG. 6 shows an example configuration in which the landmark LM1 is projected onto only three keyframes KF1, KF2 and KF3. As shown in FIG.
  • reprojection error information is calculated from the position information of the landmark LM1, the orientation information of the keyframe KF1, and the projection coordinate information of the landmark LM1 onto the keyframe KF1.
  • Such calculation of reprojection error information is also performed for other keyframes KF2 and KF3, and these three reprojection errors are aggregated.
  • the reprojection errors are aggregated for each of the other landmarks as well.
  • the keyframe starting point errors are aggregated (keyframe starting point error aggregation step S540). That is, for each of one or more keyframes, a second reprojection error information group is generated in which all reprojection error information associated with the keyframe is aggregated, and the second reprojection error information A second sum based on all reprojection error information included in the group is determined.
  • the second reprojection error information group will be explained using FIG.
  • FIG. 7 is a schematic diagram for explaining the second reprojection error information group according to this embodiment.
  • FIG. 7 shows a configuration example in which only three landmarks LM1, LM2 and LM3 are projected onto the keyframe KF1.
  • reprojection error information is calculated from the orientation information of the keyframe KF1, the position information of the landmark LM1, and the projection coordinate information of the landmark LM1 onto the keyframe KF1.
  • Such reprojection error information calculation is also performed for the other landmarks LM2 and LM3, and these three reprojection errors are aggregated.
  • the reprojection error is aggregated for each of the other keyframes as well.
  • the feature values obtained for each correspondence relationship described above are totaled for each element.
  • the diagonal block of the Hessian matrix shown in the above formulas (4a) to (4c) and (5a) to (5c) and the gradient vector shown in the above formula (3b) The element associated with that block of the gradient vector is obtained.
  • the location information is updated (location information update step S550).
  • a position information update value which is an update value of the position information of the landmark, is inferred from the first total value calculated in the landmark origin error aggregation step S530. , update the position information of the landmark using the position information update value.
  • posture information update step S560 the posture information is updated (posture information update step S560).
  • an orientation information update value which is the orientation information update value of the keyframe, is inferred from the second sum calculated in the keyframe origin error aggregation step S540. , update the posture information of the key frame using the posture information update value.
  • the position information update value of each of one or more landmarks and the orientation information update value of one or more keyframes can be obtained.
  • Prediction map information can be obtained based on these position information update values and orientation information update values.
  • the position information update value and the orientation information update value are obtained by inferring them, so the amount of calculation can be significantly reduced compared to spatial geometric calculation. Further, in the present embodiment, by separating the inference of the position information and the inference of the orientation information, the degree of freedom of the solution in the inference can be reduced, so that more accurate inference can be achieved.
  • the degree of freedom of the solution of each inference engine is increased. Since it can be reduced, learning can be performed more reliably.
  • the learning method of each inference engine will be described later.
  • reprojection error information is also obtained by inference, so the amount of calculation can be further reduced.
  • FIG. 8 is a block diagram showing the functional configuration of the map information updating device 10 according to this embodiment. As shown in FIG. 8, the map information updating device 10 receives input information including position information, orientation information, and projection relationship, and outputs output information including position information update values and orientation information update values. is.
  • the map information updating device 10 includes an error inference engine 20, a first aggregator 21, a second aggregator 22, a position inference engine 23, and an orientation inference engine 24. .
  • the error inference engine 20 is an inference engine that executes the reprojection error information acquisition step of the map information update method.
  • the error inference engine 20 associates, for each of one or more projection relations, position information associated with one landmark that constitutes the projection relation with one key frame that constitutes the projection relation.
  • Re-projection error information is obtained based on the obtained orientation information and projection coordinate information forming the projection relationship, and the re-projection error information is associated with the projection relationship.
  • the reprojection error information indicates the result of inferring the reprojection error from one piece of position information, one piece of orientation information, and projection coordinate information corresponding to one or more projection relationships.
  • the reprojection error is an error spatially and geometrically calculated from the position information, the orientation information, and the projection coordinate information.
  • the first aggregating unit 21 is a processing unit that executes the landmark starting point error aggregating step of the map information updating method.
  • a first aggregating unit 21 generates, for each of one or more landmarks, a first reprojection error information group in which all reprojection error information associated with the landmark is aggregated; A first total value based on all reprojection error information included in one reprojection error information group is obtained.
  • the second aggregating unit 22 is a step that executes the keyframe starting point error aggregating step of the map information update method.
  • a second aggregating unit 22 generates, for each of one or more keyframes, a second reprojection error information group in which all reprojection error information associated with the keyframe is aggregated; A second total value based on all reprojection error information included in the second reprojection error information group is obtained.
  • the attitude inference engine 24 is an inference engine that executes the attitude information update step of the map information update method.
  • Pose inference engine 24 infers, for each of the one or more keyframes, a pose information update value that is an update value for pose information for that keyframe from the second sum, and uses the pose information update value to Update the posture information of the relevant keyframe.
  • the predictive map information calculating method of the map information updating method described above can be realized.
  • the position inference engine 23, attitude inference engine 24, and error inference engine 20 are engines that have learned using the sum of reprojection errors obtained spatially and geometrically based on updated values of map information as a loss function.
  • the update value of the map information is obtained by inference by the position inference engine 23 and the orientation inference engine 24 based on the reprojection error information inferred by the error inference engine 20 based on the map information.
  • position information and orientation information are inferred by separate inference engines.
  • the degree of freedom of inference can be greatly reduced. That is, the difference between the size determined by the entire Hessian matrix for calculating the update value of the entire map information, and the size of the submatrix corresponding to the position information and the size of the submatrix corresponding to the orientation information in the Hessian matrix. can reduce the degree of freedom of inference equivalent to . Therefore, the man-hours required for learning each inference engine can be greatly reduced.
  • the degree of freedom of inference can be reduced, the certainty of learning of the inference engine can be enhanced. This makes it possible to reduce the model scale (in other words, the amount of calculation) required to obtain the necessary inference accuracy.
  • map information update method in general, when a solution method using a Hessian matrix composed of arbitrary numerical combinations is performed by inference, a structure such as a simple perceptron or a convolutional neural network cannot be used.
  • the degree of freedom in obtaining a combination of the numerical value of and the solution becomes very large, and as a result, the scale of the inference model and the amount of calculation become large.
  • the scale and amount of calculation of the learning of the inference model also increase, so the man-hours and difficulty of learning increase.
  • the inference model is composed of the error inference engine 20 for inferring reprojection error information in one projection relationship, and the reprojection error information output from the error inference engine 20. are divided into a position inference engine 23 and an orientation inference engine 24 that respectively infer updated values of position information and orientation information from the sum of .
  • the change in the size of the Hessian matrix can be determined by the number of projection relationships (related to the number of times the error inference engine 20 is used) and the number of position information and orientation information (position inference engine 23 and orientation). related to the number of times the inference engine 24 is used). That is, the scale of the Hessian matrix is reduced to the number of times the same inference engine is used, and the scale of each divided inference engine can be constant and small. Therefore, learning of each divided inference engine can be facilitated.
  • the reprojection error in the output information could be reduced more than the reprojection error in the input information.
  • the amount of calculation can be reduced, so the time required for bundle adjustment can be reduced to 1/10 or less.
  • the average value of reprojection errors can be reduced more than the conventional map information updating method.
  • Embodiment 2 A map information updating method according to the second embodiment will be described.
  • the map information updating method according to the present embodiment differs from the map information updating method according to the first embodiment mainly in the reprojection error information obtaining step.
  • the map information updating method according to the present embodiment will be described below, focusing on differences from the map information updating method according to the first embodiment.
  • the reprojection error information calculated in the reprojection error information acquiring step of the map information updating method according to the present embodiment includes one piece of position information, one piece of orientation information, and a reprojection error spatially and geometrically calculated from projection coordinate information. That is, in this embodiment, unlike the map information updating method according to the first embodiment, the reprojection error is spatially and geometrically calculated instead of inferring the reprojection error.
  • the present embodiment it is possible to accurately calculate the reprojection error. Also in the present embodiment, since the position information update value and the orientation information update value are calculated by inference, the same effect as the map information update method according to the first embodiment can be obtained in this respect. .
  • the position information update value is inferred by the position inference engine
  • the orientation information update value is inferred by the orientation inference engine
  • the reprojection error is spatially geometrically calculated.
  • the position inference engine and the orientation inference engine use the sum of reprojection errors obtained spatially and geometrically based on updated values of map information as a loss function, It is a learned engine.
  • the update value of the map information is determined by reasoning by the position inference engine and the orientation inference engine based on the reprojection error spatially and geometrically determined based on the map information.
  • the position inference engine and attitude inference engine according to the present embodiment also have the same effects as the position inference engine 23 and attitude inference engine 24 according to the first embodiment.
  • FIG. 9 is a diagram showing an example of the hardware configuration of computer 1000 for executing the method according to each of the above embodiments by software.
  • the computer 1000 can realize a map information updating apparatus that executes each map information updating method according to the first and second embodiments.
  • the computer 1000 comprises an input device 1001, an output device 1002, a CPU 1003, a built-in storage 1004, a RAM 1005, a reading device 1007, a transmitting/receiving device 1008 and a bus 1009, as shown in FIG.
  • the input device 1001 , output device 1002 , CPU 1003 , internal storage 1004 , RAM 1005 , reading device 1007 and transmission/reception device 1008 are connected by a bus 1009 .
  • the input device 1001 is a user interface device such as a keyboard, mouse, input button, touch pad, touch panel display, etc., and receives user operations. Note that the input device 1001 may be configured to receive a user's contact operation, as well as a voice operation or a remote operation using a remote control or the like.
  • the output device 1002 is a device that outputs a signal from the computer 1000, and may be a device that serves as a user interface, such as a display and a speaker, in addition to a signal output terminal.
  • the internal storage 1004 is a flash memory or the like. Further, the built-in storage 1004 may store in advance a program or the like for executing the steps of each method according to the first and second embodiments.
  • the RAM 1005 is a random access memory, and is used to store data calculated when executing programs or applications.
  • a reading device 1007 reads information from a recording medium such as a USB (Universal Serial Bus) memory.
  • the reading device 1007 reads the programs and applications as described above from a recording medium in which the programs and applications are recorded, and stores them in the built-in storage 1004 .
  • the transmitting/receiving device 1008 is a communication circuit for wireless or wired communication.
  • the transmission/reception device 1008 communicates with, for example, a server device connected to a network, downloads the above-described programs and applications from the server device, and stores them in the built-in storage 1004 .
  • the CPU 1003 is a central processing unit, which copies programs, applications, etc. stored in the built-in storage 1004 to the RAM 1005, sequentially reads out instructions included in the copied programs, applications, etc. from the RAM 1005 and executes them. .
  • It may be a computer program for realizing each method according to the present disclosure by a computer, or it may be a digital signal composed of the computer program.
  • the present disclosure may be implemented as a non-transitory computer-readable recording medium such as a CD-ROM recording the computer program.
  • the present disclosure may also be a computer system comprising a microprocessor and memory, the memory storing the computer program, and the microprocessor operating according to the computer program.
  • the present disclosure can be used, for example, in VSLAM technology.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Mathematical Physics (AREA)
  • Business, Economics & Management (AREA)
  • Educational Administration (AREA)
  • Educational Technology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Navigation (AREA)
  • Processing Or Creating Images (AREA)
PCT/JP2022/008794 2021-03-04 2022-03-02 マップ情報更新方法 Ceased WO2022186256A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2023503896A JP7782544B2 (ja) 2021-03-04 2022-03-02 マップ情報更新方法
US18/341,466 US12567344B2 (en) 2021-03-04 2023-06-26 Map information update method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US202163156795P 2021-03-04 2021-03-04
US63/156,795 2021-03-04

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/341,466 Continuation US12567344B2 (en) 2021-03-04 2023-06-26 Map information update method

Publications (1)

Publication Number Publication Date
WO2022186256A1 true WO2022186256A1 (ja) 2022-09-09

Family

ID=83154606

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/008794 Ceased WO2022186256A1 (ja) 2021-03-04 2022-03-02 マップ情報更新方法

Country Status (3)

Country Link
US (1) US12567344B2 (https=)
JP (1) JP7782544B2 (https=)
WO (1) WO2022186256A1 (https=)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2018131165A1 (ja) * 2017-01-16 2018-07-19 富士通株式会社 情報処理プログラム、情報処理方法および情報処理装置
JP2020516853A (ja) * 2016-12-09 2020-06-11 トムトム グローバル コンテント ベスローテン フエンノートシャップ ビデオベースの位置決め及びマッピングの方法及びシステム

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
GB2506338A (en) * 2012-07-30 2014-04-02 Sony Comp Entertainment Europe A method of localisation and mapping
CN103914874B (zh) 2014-04-08 2017-02-01 中山大学 一种无特征提取的紧致sfm三维重建方法
GB2555788A (en) * 2016-11-08 2018-05-16 Nokia Technologies Oy An apparatus, a method and a computer program for video coding and decoding
JP6762913B2 (ja) * 2017-07-11 2020-09-30 キヤノン株式会社 情報処理装置、情報処理方法
WO2019164498A1 (en) * 2018-02-23 2019-08-29 Sony Mobile Communications Inc. Methods, devices and computer program products for global bundle adjustment of 3d images
JP7771952B2 (ja) 2020-05-21 2025-11-18 株式会社ソシオネクスト マップ情報更新方法
US11748954B2 (en) * 2020-06-01 2023-09-05 Snap Inc. Tracking an augmented reality device

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2020516853A (ja) * 2016-12-09 2020-06-11 トムトム グローバル コンテント ベスローテン フエンノートシャップ ビデオベースの位置決め及びマッピングの方法及びシステム
WO2018131165A1 (ja) * 2017-01-16 2018-07-19 富士通株式会社 情報処理プログラム、情報処理方法および情報処理装置

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
PETER W. BATTAGLIA; JESSICA B. HAMRICK; VICTOR BAPST; ALVARO SANCHEZ-GONZALEZ; VINICIUS ZAMBALDI; MATEUSZ MALINOWSKI; ANDREA TACCH: "Relational inductive biases, deep learning, and graph networks", ARXIV.ORG, CORNELL UNIVERSITY LIBRARY, 201 OLIN LIBRARY CORNELL UNIVERSITY ITHACA, NY 14853, 4 June 2018 (2018-06-04), 201 Olin Library Cornell University Ithaca, NY 14853 , XP080886921 *
SHINDOH, TOMONORI: "Google innovates with deep learning-based monocular SLAM technology, achieving self-position estimation accuracy that surpasses existing vSLAM", NIKKEI ROBOTICS, no. 48, 10 June 2019 (2019-06-10), pages 5 - 13, XP009539343, ISSN: 2189-5783 *

Also Published As

Publication number Publication date
US20230335016A1 (en) 2023-10-19
JPWO2022186256A1 (https=) 2022-09-09
JP7782544B2 (ja) 2025-12-09
US12567344B2 (en) 2026-03-03

Similar Documents

Publication Publication Date Title
JP7665839B2 (ja) Ｗｉｆｉ／ｇｐｓベースのマップマージを伴うクロスリアリティシステム
US11900547B2 (en) Cross reality system for large scale environments
JP7684321B2 (ja) 位置特定に関するジオロケーション情報の優先順位化を伴うクロスリアリティシステム
JP7653982B2 (ja) 無線フィンガプリントを用いたクロスリアリティシステム
JP7671769B2 (ja) 高速位置特定を伴うクロスリアリティシステム
US11443481B1 (en) Reconstructing three-dimensional scenes portrayed in digital images utilizing point cloud machine-learning models
CN115176285B (zh) 利用缓冲的交叉现实系统用于定位精度
JP2025013609A (ja) 位置特定サービスを伴うクロスリアリティシステム
CN115398314A (zh) 使用多分辨率帧描述符进行地图处理的交叉现实系统
US11417095B2 (en) Image recognition method and apparatus, electronic device, and readable storage medium using an update on body extraction parameter and alignment parameter
JP7711263B2 (ja) クロスリアリティシステム
JP2006260527A (ja) 画像マッチング方法およびこれを用いた画像補間方法
CN112183547B (zh) 基于多模态数据的多任务学习方法及系统
CN111862299A (zh) 人体三维模型构建方法、装置、机器人和存储介质
CN115349140A (zh) 基于多种特征类型的有效定位
WO2023185241A1 (zh) 数据处理方法、装置、设备以及介质
CN112233149A (zh) 场景流的确定方法及装置、存储介质、电子装置
CN114821659A (zh) 一种基于双流网络的人体动作识别方法及系统
CN120451448A (zh) 面向大场景空间的动态三维重建方法及系统
WO2022186256A1 (ja) マップ情報更新方法
CN117241065B (zh) 视频插帧图像生成方法、装置、计算机设备和存储介质
CN117455968B (zh) 一种坐标系变换方法及存储介质
CN116071487B (zh) 三维人脸模型的重建方法、装置、设备及存储介质
CN115082620A (zh) 稀疏地图优化方法、设备及存储介质
HK40097770B (zh) 视频插帧图像生成方法、装置、计算机设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22763310

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2023503896

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22763310

Country of ref document: EP

Kind code of ref document: A1