JP2018022247A - Information processing apparatus and control method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To calculate a position and attitude stably with high accuracy and robustness.SOLUTION: An information processing apparatus holds a three-dimensional map including three-dimensional information of features existing in an image and position attitude information of an imaging section obtained in capturing the image, inputs the image captured by the imaging section, determines consistency between the input image and the features recorded on the three-dimensional map, calculates an evaluation value indicating a result of evaluating a presence state of the features associated between the three-dimensional map and the input image, calculates weight for controlling the influence of the features in the three-dimensional map on calculation of the position and attitude of the imaging section, on the basis of the consistency and evaluation value, and calculates a position and attitude of the imaging section capturing the input image by use of the three-dimensional map and the weight.SELECTED DRAWING: Figure 1A

Description

  The present invention relates to an information processing apparatus that measures the position and orientation of an imaging apparatus and a control method thereof.

  A technique for calculating the position and orientation of an imaging device by comparing an image with a three-dimensional map of a real space is known. Such a technique is used for self-position estimation of robots and automobiles, alignment between a real space and a virtual object in augmented / mixed reality.

  Patent Document 1 discloses a method for calculating the position and orientation of a camera based on an image input by a camera that captures the scene in a scene where a moving object exists in the real space. In this method, it is determined whether or not the feature point on the three-dimensional map is a point on the moving object, and the position and orientation of the camera are calculated using the feature point excluding the feature point on the moving object.

JP 2012-221042 A

G. Klein and D. Murray, “Parallel tracking and mapping for small AR workspaces,” International Symposium on Mixed and Augmented Reality, pp. 225-234, 2007 Z. Zhang, “A flexible new technique for camera calibration,” IEEE Trans. On Pattern Analysis and Machine Intelligence, vol.22, no.11, pp.1330-1334, 2000. H. Kato and M. Billinghurst, “Marker tracking and hmd calibration for a video-based augmented reality conferencing system,” International Workshop on Augmented Reality, 1999 RY Tsai, “A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses,” IEEE Journal of Robotics and Automation, vol.3, no.4, pp.323-344, 1987. R. I. Hartley, “Self-calibration from multiple views with a rotating camera,” European Conference on Computer Vision, pp.471-478, 1994. C. Pirchheim, D. Schmalstieg, G. Reitmayr, “Handling pure camera rotation in keyframe-based slam,” International Symposium on Mixed and Augmented Reality, pp.229-238, 2013. J. Engel, T. Schops, D. Cremers, “LSD-SLAM: Large-Scale Direct Monocular SLAM,” European Conference on Computer Vision, pp.834-849, 2014.

  However, in Patent Document 1, when it is determined that a feature point on the three-dimensional map is a point on a moving object, the feature point is excluded. Therefore, in Patent Document 1, there is a problem that the accuracy and robustness of the position and orientation of the camera are lowered when the number of feature points used for position and orientation calculation is small or when the feature distribution is biased.

  Therefore, an object of the present invention is to make it possible to calculate a position and an attitude stably, highly accurately and robustly.

In order to achieve the above object, an information processing apparatus according to an aspect of the present invention has the following arrangement. That is,
Holding means for holding a three-dimensional map in which three-dimensional information of features existing in the image and position and orientation information of the imaging unit at the time of capturing the image are recorded;
Input means for inputting an image captured by the imaging unit;
Determining means for determining consistency with the input image for the features recorded in the three-dimensional map;
Evaluation means for calculating an evaluation value indicating a result of evaluation of the existence state of the feature associated between the three-dimensional map and the input image;
Weight calculation means for calculating a weight for controlling the influence of the characteristics of the three-dimensional map on the calculation of the position and orientation of the imaging unit based on the consistency and the evaluation value;
Calculating means for calculating the position and orientation of the imaging unit when the input image is captured using the three-dimensional map and the weight.

  According to the present invention, the position and orientation can be calculated stably, with high accuracy and with high robustness.

The block diagram which shows the function structural example of the information processing apparatus in 1st Embodiment. The block diagram which shows the hardware structural example of information processing apparatus. The flowchart which shows the process sequence in 1st Embodiment. The figure which shows an example of the list structure which the characteristic on a three-dimensional map has. 6 is a flowchart showing a procedure for determining consistency in the first embodiment. The block diagram which shows the function structural example of the information processing apparatus in 4th Embodiment. The flowchart which shows the process sequence in 4th Embodiment. The figure of the graph which shows the change of the relationship between consistency and weight by the level of an evaluation value.

  Embodiments of the present invention will be described below with reference to the accompanying drawings.

<First Embodiment>
Hereinafter, an information processing apparatus that measures the position and orientation of the imaging unit in the real space based on a three-dimensional map that holds the three-dimensional coordinates of the feature points existing in the real space will be described. The information processing apparatus according to the present embodiment calculates the position and orientation with high accuracy and high robustness even when the number of feature points used for position and orientation calculation is small and there is a moving object in the real space. It is possible.

  In the first embodiment, the image input from the imaging unit and the 3D map are collated, and the position and orientation of the imaging unit are calculated. At this time, the feature point on the moving object lowers the magnitude of the influence (hereinafter referred to as the weight) when calculating the position and orientation. However, as the number of feature points used for position and orientation calculation increases, a weight is given to a feature point that is less likely to be on a moving object, and a feature point that is more likely to be on a moving object is assigned to a smaller number of feature points. . In this way, the position and orientation are calculated with high accuracy and high robustness by adjusting the weight according to the number of feature points used for the position and orientation calculation.

  FIG. 1A is a block diagram illustrating a functional configuration example of the information processing apparatus 1 according to the first embodiment. The information processing apparatus 1 includes a three-dimensional information holding unit 110, an image input unit 120, a consistency determination unit 130, a feature evaluation unit 140, a weight calculation unit 150, and a position / orientation calculation unit 160 as functional units. The three-dimensional information holding unit 110 holds a three-dimensional map used by the consistency determining unit 130 and the position / orientation calculating unit 160. The image input unit 120 is connected to the image capturing unit 180 and inputs an image captured by the image capturing unit 180 to the consistency determining unit 130 and the position / orientation calculating unit 160. In the first embodiment, one color camera is used as the imaging unit 180. The initialization unit 170 sets various pieces of initial information when the information processing apparatus 1 starts calculating the position and orientation of the imaging unit 180.

  The consistency determination unit 130 calculates the position and orientation based on the three-dimensional map held by the three-dimensional information holding unit 110, the image input by the image input unit 120, and the position and orientation calculated by the position and orientation calculation unit 160. The consistency of the feature point group used for is determined. Here, the consistency is a value indicating the possibility that the feature point on the three-dimensional map is not on the moving object. Details of the consistency will be described later. The feature evaluation unit 140 calculates an evaluation value related to the distribution of the feature point group used in the position and orientation calculation extracted by the consistency determination unit 130. Based on the consistency determined by the consistency determination unit 130 and the evaluation value that is the result of the evaluation by the feature evaluation unit 140, the weight calculation unit 150 includes each feature included in the feature point group used for position and orientation calculation. Calculate point weights. The position / orientation calculation unit 160 is based on the three-dimensional map held by the three-dimensional information holding unit 110, the image input by the image input unit 120, and the weight calculated by the weight calculation unit 150. Calculate the position and orientation at.

  FIG. 1B is a block diagram illustrating a hardware configuration example of the information processing apparatus 1 according to the present embodiment. The CPU 10 implements each functional unit described above by executing a program stored in the ROM 11 or the RAM 12. The ROM 11 stores programs executed by the CPU 10 and various data. The RAM 12 provides a work area when the CPU 10 executes various processes. Note that the program stored in the external storage device 13 is loaded into the RAM 12 and executed by the CPU 10. The external storage device 13 is composed of a hard disk, a flash memory, or the like, and holds various information. The keyboard 14 and the pointing device 15 are instruction input units for a user to give various instructions to the CPU 10. The interface 16 is connected to an external device, and realizes communication between the information processing device 1 and the external device. In the present embodiment, the imaging unit 180 is connected to the interface 16, and an image captured from the imaging unit 180 is input to the information processing apparatus via the interface 16. The bus 17 connects the above-described units so that they can communicate with each other.

  In addition, although each function part shown to FIG. 1A shall be implement | achieved by execution of the software by CPU10, it is not restricted to this. At least a part of the functional units shown in FIG. 1A may be realized by dedicated hardware. Each function may be realized by one CPU (processor) or a plurality of CPUs (processors).

  Next, a processing procedure for position and orientation calculation in the first embodiment will be described. FIG. 2 is a flowchart showing a processing procedure in the first embodiment.

  In step S <b> 101, the initialization unit 170 reads the initial three-dimensional map and stores it in the three-dimensional information holding unit 110. The initialization unit 170 reads the internal parameters of the imaging unit 180 and calculates the initial position and initial posture of the imaging unit 180. In the present embodiment, the three-dimensional map is held as a set of feature points on the key frame image. Here, the key frame image is an image to which the position and orientation in the world coordinate system of the imaging unit 180 when the image is captured are assigned as attributes.

  The feature point group of the three-dimensional map has a list structure as shown in FIG. 3 for each key frame image. FIG. 3 is an example of a list structure possessed by M feature points. In the three-dimensional map, three-dimensional information of features existing in the image and position / orientation information of the imaging unit at the time of capturing the image are recorded. The feature point ID in FIG. 3 is a number that can uniquely identify each feature point within one key frame image. The luminance value is a luminance value of each feature point obtained from the key frame image. The image coordinates are two-dimensional coordinates of each feature point in the key frame image coordinate system. The depth value is a depth value of each feature point based on the imaging unit coordinate system. The feature amount is a feature amount extracted at each feature point. The three-dimensional coordinates based on the world coordinate system of the feature points can be calculated using the image coordinates and depth values of the feature points, and the position and orientation given to the key frame image. Each three-dimensional map holds, as attributes, a key frame ID for identifying the key frame from which information has been extracted and the position and orientation (world coordinate system) of the imaging unit 180 when the key frame is imaged. ing.

  In addition, the initial three-dimensional map can be created, for example, by the method of Klein et al. (Non-Patent Document 1) that performs the three-dimensional map creation and the position and orientation measurement of the imaging unit 180 at the same time. In addition, the internal parameters of the imaging unit 180 are calibrated in advance by, for example, the Zhang method (Non-Patent Document 2) using an image obtained by capturing a planar pattern from multiple viewpoints. In addition, the initial position and initial posture of the imaging unit 180 are calculated by, for example, the method of Kato et al. (Non-Patent Document 3) using an artificial marker whose size is known. Note that when the position and orientation of the imaging unit 180 are calculated in S106, which will be described later, a new three-dimensional map is generated using the target input image as a key frame and additionally held. The additionally held three-dimensional map can be used for subsequent calculation of position and orientation. The storage of the three-dimensional map may be performed every time the position and orientation are calculated in S106, or may be performed once every n times (n> 1). Further, when the calculated position and orientation are similar to the position and orientation of the already stored three-dimensional map, it may not be stored as a three-dimensional map.

  Next, in step S <b> 102, the image input unit 120 inputs an image captured by the imaging unit 180 to the information processing apparatus 1. In step S103, the consistency determining unit 130 calculates the position and orientation based on the 3D map held by the 3D information holding unit 110, the image input in step S102, and the position and orientation of the frame immediately before the imaging unit 180. Determine the consistency of the feature points used for. In the present embodiment, the consistency is a value to be lowered when the feature point on the three-dimensional map is a point on a moving object or a point that is blocked by a moving object and is not correctly observed from an input image. More specific description will be given below.

  FIG. 4 is a flowchart showing the procedure of consistency determination processing in step S103. In the present embodiment, the consistency determining unit 130 obtains a difference between the 3D map and the input image with respect to the corresponding feature between the 3D map and the input image, and determines the consistency based on the difference. To do. In step S111, the consistency determining unit 130 first uses the position and orientation calculated in the previous frame as predicted values of the position and orientation of the current frame, and includes a position and orientation closest to the predicted value as attributes. Select the key frame image. Then, the feature on the input image is extracted, and the feature point group on the key frame image and the feature point group on the input image are associated by feature amount comparison. Hereinafter, the feature points on the key frame image corresponding to the feature points on the input image are referred to as selected feature points, and a collection of all the selected feature points in one key frame image is referred to as a selected feature point group. Call. Note that when there is no previous frame, the initial position and initial posture are used. In addition, for example, a template matching method can be used for the above-described feature amount comparison.

  In step S112, the consistency determining unit 130 selects a selected feature point whose consistency has not been determined from the selected feature point group. In step S113, the consistency determining unit 130 projects the selected feature point on the input image based on the predicted position and orientation values, and predicts the coordinates (position p) in the input image coordinate system. In step S114, the consistency determining unit 130 calculates the absolute value of the difference between the pixel value (luminance value) of the input image at the predicted coordinates (position p) and the luminance value of the selected feature point recorded in the three-dimensional map. To determine consistency. The larger the absolute value of the luminance value difference, the more likely it is that the selected feature point is a point on a moving object (moving object) or a point that is blocked by a moving object and is not correctly observed from the input image, Reduce the consistency value. This is because the pixel value of the place (background) where the feature point exists greatly changes as the moving body moves. In this embodiment, the absolute value of the difference obtained by comparing the pixel value (luminance value) of the projection position of the feature point in the input image with the attribute value (luminance value) of the feature point in the three-dimensional map is used. However, it is not limited to this. For example, a difference obtained by comparing the average pixel value of pixels within a predetermined range of the projection position of the feature point in the input image and the attribute value of the feature point in the three-dimensional map may be used.

但し、ｓ_ｔｈは整合性を０とする輝度値の差の絶対値の閾値とする。 Specifically, when the absolute value of the difference in luminance value is s, the consistency c is determined as shown in Equation 1.

However, s _th is a threshold value of the absolute value of the difference between the luminance values where the consistency is zero.

  In step S115, the consistency determining unit 130 determines whether the consistency of all selected feature points has been determined. If it is determined that the consistency of all selected feature points has been determined, the consistency determination process in step S103 is terminated. If there is an unprocessed selected feature point, the processing from step S112 is repeated.

Returning to FIG. 2, in step S104, the feature evaluation unit 140 evaluates the material state of the selected feature point obtained in step S103, and obtains an evaluation value. Hereinafter, the evaluation of the feature point existence state performed by the feature evaluation unit 140 in step S104 is also simply referred to as feature point evaluation. In the first embodiment, the feature evaluation unit 140 calculates an evaluation value based on the number of selected feature points. The feature evaluation unit 140 sets the evaluation value high so that only the feature points having higher consistency are used for position and orientation calculation as the number of selected feature points is larger, and the smaller the number of selected feature points is, the higher the matching value is. The evaluation value is set to be low so that feature points with low characteristics are also used for position and orientation calculations. In the first embodiment, when the number of selected feature points is N _A , the evaluation value E is calculated as shown in Equation 2.

Ｅ＜Ｅ_ｔｈのとき、

但し、Ｅ_ｔｈは評価値の高低を判断する閾値、ｃ_ｔｈ１、ｃ_ｔｈ２は重みを０とする整合性の閾値（ｃ_ｔｈ１＞ｃ_ｔｈ２）とする。 In step S105, the weight calculation unit 150 calculates a weight (w) based on the consistency (c) determined in step S103 and the evaluation value (E) calculated in step S104. The weight w is calculated individually for each selected feature point. As the evaluation value E is higher, a weight is given only to feature points having higher consistency, and as the evaluation value is lower, weights are given to feature points having lower consistency. Specifically, when the evaluation value is E and the consistency is c, the weight w is calculated as in Equations 3 and 4.
When E ≧ E _th

When E <E _th

However, E _th is a threshold value for judging the level of the evaluation value, and c _th1 and c _th2 are consistency threshold values with a weight of 0 (c _th1 > c _th2 ).

  In step S106, the position / orientation calculation unit 160 includes the 3D map stored in the 3D information storage unit 110, the image input in step S102, and the weight calculated in step S105. Calculate position and orientation. The position and orientation are calculated in consideration of the weight calculated in step S105 based on the Klein method (Non-Patent Document 1) used in step S101. Specifically, the position and orientation at which S in Equation 5 is minimized are calculated.

Ｓは、入力画像座標系における被選択特徴点の二次元座標ｍｉ’と、特徴量比較によってｍｉ’と対応付けた入力画像上の特徴点の二次元座標ｍｉとのユークリッド距離の二乗に重みｗｉを乗算したものを被選択特徴点ごとに計算し、それらを合計したものである。

S is a weight wi for the square of the Euclidean distance between the two-dimensional coordinate mi ′ of the selected feature point in the input image coordinate system and the two-dimensional coordinate mi of the feature point on the input image associated with mi ′ by the feature amount comparison. Is calculated for each selected feature point and summed up.

  In step S107, it is determined whether or not to end the entire process. When a command for ending the entire process is input from the user via a mouse or a keyboard, the entire process is ended. Otherwise, the process from step S102 is repeated to continue the process.

  As described above, in the first embodiment, as the number of feature points used for position and orientation calculation increases, weight is given to a feature point that is less likely to be on a moving object, and as the number of feature points decreases, the feature point may be on a moving object. Weights are also given to feature points having high characteristics. Thus, the position and orientation can be calculated with high accuracy and high robustness by adjusting the weight according to the number of feature points used for calculating the position and orientation.

Second Embodiment
In the first embodiment, the weight, which is the magnitude of the influence of each feature point on the position and orientation calculation, is calculated based on the number of feature points used for the position and orientation calculation. In the second embodiment, a weight that is the magnitude of the influence of each feature point on the position and orientation calculation is calculated based on the density of the distribution of the feature points used for the position and orientation calculation. In other words, in the second embodiment, weights are given to feature points that are less likely to be on moving objects as the distribution of feature points used for position and orientation calculation is denser, and the distribution points are located on moving objects. Give weights to highly likely feature points. In this way, the position and orientation are calculated with high accuracy and high robustness by preventing the deviation of the distribution of the feature points used for the position and orientation calculation.

  The configuration of the apparatus in the second embodiment is the same as that of the information processing apparatus 1 described in the first embodiment. The processing procedures for initialization, image input, consistency determination, weight calculation, position and orientation calculation in the second embodiment are the same as those in the first embodiment. The main part different between the first embodiment and the second embodiment is the evaluation of the feature existence state in the flowchart of FIG. 2 (step S104).

In step S <b> 104, the feature evaluation unit 140 calculates an evaluation value of the feature distribution based on the density distribution in the input image coordinate system of the selected feature point obtained in step S <b> 103. The density of the distribution of the selected feature points is determined by, for example, dividing the input image into four grids and calculating the number of selected feature points projected in each divided region. Specifically, if the number of selected feature points projected in the divided region D is N _D , the evaluation value E of the selected feature points projected in the divided region D is calculated as in Equation 6.

Then, the weight calculation unit 150 determines E by Equation 6 according to which divided region the projection position of the selected feature point onto the input image belongs, and uses _Equation 3 above when E ≧ E _th. The weight w is calculated, and when E < _Eth , the weight w is calculated using the above _equation 4. The position / orientation calculation unit 160 calculates the position and orientation of the imaging unit 180 using Equation 5 using the weights calculated by the weight calculation unit 150 for each feature point. However, the value of _Eth is different from that in the first embodiment. For example, since the image is divided into four in the second embodiment, it is set to 1/4 of the threshold value used in the first embodiment.

  As described above, in the second embodiment, weights are given to feature points that are less likely to be on moving objects as the distribution of feature points used for position and orientation calculation is denser, and objects that move as the distribution is coarser. Also weight the feature points that are likely to be above. Thus, the position and orientation are calculated with high accuracy and high robustness by preventing the bias of the distribution of features used for the position and orientation calculation.

<Third Embodiment>
In the first embodiment, an example is shown in which weights for controlling the influence of each feature point on the position and orientation calculation are calculated based on the number of feature points used in the position and orientation calculation. In the second embodiment, an example has been shown in which weights are calculated based on a density distribution in the input image coordinate system of feature points used for position and orientation calculations. In the third embodiment, a large number of feature points used for position and orientation calculation and a density distribution are used together. Therefore, in the third embodiment, as the number of feature points used for position and orientation calculation is larger, and as the distribution of feature points used for position and orientation calculation is denser, the feature points that are less likely to be on moving objects are weighted. give. Further, in the third embodiment, weights are given to feature points that are more likely to be on a moving object as the number of feature points is smaller and a region having a coarser distribution. In this way, the position and orientation are calculated with high accuracy and high robustness by adjusting the weight according to the number of feature points used for position and orientation calculation and preventing the bias of the distribution of features used for position and orientation calculation.

  The configuration of the apparatus in the third embodiment is the same as that of the information processing apparatus 1 described in the first embodiment. Also, the initialization, image input, consistency determination, weight calculation, position and orientation calculation processing procedures in the third embodiment are the same as those in the first embodiment. The main difference between the first embodiment and the third embodiment is the evaluation of the presence state of features in the flowchart of FIG. 2 (step S104).

In step S104, the feature evaluation unit 140 evaluates the presence state of the selected feature point based on the number of selected feature points obtained in step S103 and the density distribution of the selected feature points in the input image coordinate system. Calculate For example, as in the second embodiment, the density of the distribution of selected feature points is determined by dividing the input image into four grids and calculating the number of selected feature points projected in each divided region. . Specifically, if the normalized total number of selected feature points is N _A ′, and the normalized number of selected feature points projected in the divided region D is N _D ′, the divided region D The evaluation value E of the selected feature point projected inside is calculated as shown in Equation 7.

Then, the weight calculation unit 150 determines E by Equation 6 according to which divided region the projection position of the selected feature point onto the input image belongs, and uses _Equation 3 above when E ≧ E _th. The weight w is calculated, and when E < _Eth , the weight w is calculated using the above _equation 4. The position / orientation calculation unit 160 calculates the position and orientation of the imaging unit 180 using Equation 5 using the weights calculated by the weight calculation unit 150 for each feature point. However, a value suitable for the evaluation value E calculated by _Equation 7 is used as E _th .

  As described above, in the third embodiment, as the number of feature points used for position and orientation calculation is larger and the distribution of feature points used for position and orientation calculation is denser, the feature is less likely to be on a moving object. Controlled to give weights to points. On the other hand, the smaller the number of feature points and the rougher the distribution, the higher the possibility that the feature points on the moving object are weighted. In this way, the position and orientation are calculated with high accuracy and high robustness by adjusting the weight according to the number of feature points used for position and orientation calculation and preventing the bias of the distribution of features used for position and orientation calculation.

Note that the evaluation value may be calculated as long as the evaluation value is calculated based on the number of selected feature points and the density distribution of the selected feature points, and is not limited to Equation 7 described above. For example, if N _A 'and N _D ' are the same as those defined in Equation 7, the sum of N _A 'and N _D ' is used as the evaluation value, or either N _A 'or N _D ' is the threshold value. If it is below, the evaluation value is 0. Otherwise, the evaluation value may be the product of N _A ′ and N _D ′.

<Fourth embodiment>
In the first to third embodiments, since the consistency is determined independently for each frame, the consistency greatly changes between frames, and the position and orientation may become unstable. In the fourth embodiment, the position and orientation are made highly accurate and robust by suppressing a large change in consistency using past consistency (consistency history).

  FIG. 5 is a diagram showing the configuration of the apparatus in the fourth embodiment. The imaging unit 180, the three-dimensional information holding unit 110, the image input unit 120, the consistency determining unit 130, and the position / orientation calculating unit 160 in the fourth embodiment are the same as those of the information processing apparatus 1 (FIG. 1) described in the first embodiment. It is. The main difference between the first embodiment and the fourth embodiment is that the feature evaluation unit 140 in the information processing apparatus 1 is omitted and a weight calculation unit 150a. The weight calculation unit 150a calculates the weight of each feature point included in the feature point group used for position and orientation calculation based on the consistency determined by the consistency determination unit 130.

  Next, a processing procedure in the present embodiment will be described. FIG. 6 is a flowchart showing a processing procedure in the fourth embodiment. The processing procedures of initialization, image input, position and orientation calculation in the fourth embodiment are the same as those in the first embodiment (S101, S102, S106 in FIG. 2). The main difference between the first embodiment and the fourth embodiment is that the evaluation of feature points (S104) in the flowchart of FIG. 2 is omitted, consistency determination (S201), and weight calculation (S202). .

  In step S201, first, temporary consistency is calculated based on the absolute value s of the difference between the luminance value of the selected feature point and the luminance value of the input image, as in step S103 of the first embodiment. The calculated temporary consistency is retained as a history. Then, from the newest one including the temporary consistency, it extracts from the consistency history in which the consistency for n frames is maintained, and the median value of the extracted consistency group is set as the current consistency c.

但し、ｃ_ｔｈは重みが０となる整合性の閾値とする。 In step S202, a weight is calculated based on the consistency determined in step S201. Specifically, when the consistency is c, the weight w is calculated as in Expression 8.

Here, c _th is a consistency threshold value with a weight of 0.

  As described above, in the fourth embodiment, the position and orientation are made highly accurate and robust by suppressing a large change in consistency using past consistency.

  Note that the determination of consistency is not limited to the above-described method using the median value of past consistency as long as it can determine the plausible current consistency based on the past consistency. For example, an average value of past consistency may be set as the current consistency. Also, past consistency is approximated by a polynomial, and the average value of the current provisional consistency obtained from the approximated polynomial and the current provisional consistency obtained from the difference between the selected feature point and the luminance value of the input image. May be the current consistency.

Further, the weight calculation is not limited to the above-described formula 8, as long as the weight is calculated using consistency. For example, the consistency c may be used as it is as a weight. Alternatively, the weight may be calculated assuming that the consistency c and the weight w follow an exponential function such as Equation 9 or a sigmoid function.

  In the fourth embodiment, the configuration in which the evaluation value E is not used has been described. However, using the evaluation value based on the evaluation of the feature existence state as described in the first to third embodiments, Equation 3 and The weight may be calculated as shown in FIG. Note that the current consistency may be determined without using past consistency in the configuration of the apparatus according to the fourth embodiment. In this case, for example, the feature evaluation unit 140 and its processing (S104) are omitted in the first to third embodiments.

<Other embodiments>
The configurations shown in the first to fourth embodiments are merely examples, and the present invention is not limited to these configurations. Hereinafter, modified examples of the first to fourth embodiments will be described.

  The imaging unit 180 only needs to be able to determine the consistency between the three-dimensional map and the image captured by the imaging unit 180, and is not limited to the color camera described above. For example, a camera that picks up a gray image, a camera that picks up a distance image, or a camera that picks up a color image and a distance image at the same time may be used.

  The three-dimensional map held by the three-dimensional information holding unit 110 may be any map that can express the three-dimensional coordinates of the feature points in the real space, and is not limited to the one held as a set of the key frame images described above. For example, the key frame image may be held as a set of feature points without being held. Alternatively, it may be held as a set of all frame images.

  The data structure of the feature points held by the three-dimensional map is not limited to that shown in FIG. 3, and any data structure can be used as long as the position and orientation of the imaging unit can be calculated by collating with the input image. For example, three-dimensional coordinates based on the world coordinate system may be assigned as attributes instead of image coordinates and depth values. Also, the time registered in the three-dimensional map may be given as an attribute in order not to use a feature point after a certain period of time.

  The creation method of the initial three-dimensional map read at the time of initialization is not limited to the above-described Klein method, and any method can be used as long as it can create a three-dimensional coordinate of a feature point in the real space. For example, a method of creating a three-dimensional map by measuring a real space using a laser scanner may be used. When a 3D model of real space created by CAD or the like can be used, a method of extracting features from the 3D model and creating a 3D map may be used.

  Further, the method for calibrating the internal parameters of the image capturing unit read at the time of initialization is not limited to the Zhang method described above, and any method that can calibrate the internal parameters may be used. For example, the Tsai method using a point with known three-dimensional coordinates (Non-Patent Document 4) or the Hartley method using a camera that performs only a rotational motion (Non-Patent Document 5) may be used.

  The calculation method of the initial position and initial posture of the imaging unit 180 at the time of initialization is not limited to the above-described Kato method, and any method that can acquire the position and orientation of the imaging unit 180 at the start of processing may be used. . For example, a method may be used in which an initial position and an initial posture are designated in advance, and processing is started after the imaging unit 180 is fixed to the initial position and the initial posture. Alternatively, a method may be used in which an image of a specific pattern is pasted on the imaging unit 180, the pattern is recognized by another fixed camera whose position and orientation in the world coordinate system are known, and the position and orientation of the imaging unit are acquired.

  The case where the consistency is lowered is not limited to the case where the above-described feature point on the three-dimensional map is a point on a moving object or a point that is blocked by a moving object and is not observed from an input image. Any difference between the feature points on the map and the input image may be used. For example, the brightness value may change due to a change in the light source environment, or it may be blocked by a stationary object that is not registered in the three-dimensional map.

  The determination of the consistency (provisional consistency in the fourth embodiment) is not limited to the above-described method based on the difference in luminance value between the feature point on the 3D map and the input image. Any method may be used as long as it is based on the degree of matching. For example, a method may be used in which a hand or a person is detected from the input image and the consistency of feature points on a three-dimensional map projected in the detected hand or person region is reduced. Also, the smaller the color difference between each feature point on the 3D map and the input image (elements in the RGB color space and HSV color space), the higher the consistency, and the color between the feature point on the 3D map and the input image. The larger the difference, the lower the consistency. Also, when the input image is a distance image, the smaller the difference in the distance value between the feature point on the 3D map and the input image, the higher the consistency, and the difference in the distance value between the feature point on the 3D map and the input image. A method of lowering the consistency may be used as the value is larger. Further, the smaller the distance between the projected position of the selected feature point of the 3D map on the input image and the position of the feature point associated with the selected feature point on the input image, the higher the consistency. It may be. In addition, when multiple images of grayscale, color, and distance images are obtained as input, the difference in brightness value, color difference, and distance value between the feature points on the 3D map and the input image are combined. Thus, the consistency may be determined.

  In the first to third embodiments, consistency may be determined based on past consistency (history) as in the fourth embodiment. For example, the median value or average value of past consistency may be used as the current consistency. In addition, the past consistency is approximated by a polynomial, the current provisional consistency obtained from the approximated polynomial, and the average value of the current provisional consistency obtained from the difference between the feature points used for position and orientation calculation and the input image May be the current consistency.

  The evaluation of the number of feature points in the first to third embodiments is not limited to the above-described method for evaluating based on the number of selected feature points, but a method that can determine the number of feature points used for position and orientation calculation. Anything is fine. For example, the evaluation value may be increased as the number of feature points detected from the input image is increased, and the evaluation value may be decreased as the number of feature points detected from the input image is decreased. This is because it is assumed that the greater the number of feature points detected from the input image, the greater the number of selected feature points from the three-dimensional map. The evaluation value increases as the number of selected feature points and the number of feature points detected from the input image increase, and the evaluation value increases as the number of the selected feature points and the number of feature points detected from the input image decreases. It may be lowered.

  The evaluation of the density distribution of feature points in the second and third embodiments is not limited to the above-described method of calculating the number of feature points in each divided region by dividing the input image into four grids, and features in a certain range. Any method can be used as long as it can determine the density of the point distribution. For example, the input image is divided into concentric circles with different radii centered on the center of the input image, and the larger the number of feature points in each divided region, the higher the evaluation value of the feature points in that divided region. As the number of feature points decreases, the evaluation value of the feature points in the divided region may be increased. Also, the higher the number of feature points within a specific radius centered on a certain feature point, the higher the evaluation value of that feature point. The smaller the number of feature points within a specific radius centered on a certain feature point, the higher the evaluation value of that feature point. The value may be lowered.

The weight is not limited to the above-described continuous value, and may be expressed as a binary value or a quantized value. For example, if the consistency is c and the consistency thresholds at which the weights are changed are c _th , c _th1 , c _th2 (c _th1 > c _th2 ), the weight w may be calculated as in _Expression 10 or Expression 11.

The weight calculation in the first to third embodiments is not limited to the above-described Expression 3 and Expression 4, and may be anything as long as the weight is calculated using the evaluation value of the consistency and the existence state of the feature. For example, the sum or product of the consistency and the evaluation value may be used as the weight. Further, when the consistency is c and the normalized evaluation value is E ′, the weight w may be calculated by changing the consistency threshold at which the weight becomes 0 according to the evaluation value as shown in Equation 12.

  Further, the weight does not necessarily have to be 0 when the consistency is less than the threshold value, and the weight may be decreased as the consistency is lower. For example, as shown in FIG. 7, the higher the consistency is, the larger the weight is, and the lower the consistency is, the smaller the weight is. However, when the consistency is low, the lower the evaluation value is, the higher the evaluation value is. May be. That is, the lower the evaluation value, the smaller the change in weight with respect to the change in consistency.

  The position and orientation calculation is not limited to the above-described Klein method, and any method may be used as long as the position and orientation are calculated based on the three-dimensional map and the input image. For example, the method of Pirchheim et al. (Non-Patent Document 6) or the method of Engel et al. (Non-Patent Document 7) may be used. The technique of Pirchhem et al. Calculates the position and orientation that minimizes the Euclidean distance in the image coordinate system between the feature points on the three-dimensional map and the feature points on the input image that are associated by the feature amount comparison, as in the Klein method. To do. Therefore, when the present invention is implemented, the influence of the feature points on the three-dimensional map on the position and orientation calculation is controlled by multiplying the Euclidean distance by the weight as shown in Equation 5. The method of Engel et al. Determines the position and orientation that minimize the difference between the luminance value of the feature point on the three-dimensional map and the luminance value of the input image at the image coordinates when the feature point on the three-dimensional map is projected onto the input image. calculate. Therefore, when the present invention is implemented, the influence of the feature points on the three-dimensional map on the position and orientation calculation is controlled by multiplying the luminance value difference by the weight.

  The predicted value of the current frame position and orientation used when extracting the selected feature point group is not limited to the position and orientation of the previous frame, but anything that seems to be close to the position and orientation of the current frame. Good. For example, a motion model such as a constant velocity motion or a constant acceleration motion may be assumed, and the position and posture of the previous frame may be updated based on the motion model. Alternatively, a sensor for measuring the position and orientation may be attached to the imaging unit, and the predicted value of the position and orientation of the current frame may be obtained based on the sensor measurement values.

  The timing for performing each process is not limited to that in FIG. 2 or FIG. 6, and may be anything as long as the weight is reflected in the position and orientation calculation. For example, consistency determination (S103, S201), feature evaluation (S104), weight calculation (S105, S202) and position and orientation calculation (S106) may be processed in parallel. Further, the consistency determination may be performed after the feature evaluation.

  The feature held by the three-dimensional map is not limited to a point, and any feature can be used as long as the three-dimensional map and the real space can be collated. For example, the coordinates in the world coordinate system of both end points of the edge of the object may be held as features. In addition, the position and orientation of the marker in the world coordinate system and the pattern of the marker that can calculate the position and orientation of the imaging unit by drawing a specific pattern may be held as features.

Consistency is not limited to the above-described continuous values, and may be expressed by binary values or quantized values. For example, the absolute value of the difference between the luminance values of the feature point on the three-dimensional map and the feature point on the input image is s, and the threshold value of the absolute value of the difference between the luminance values whose consistency changes is s _th , s _th1 , s _th2 ( If s _th1 > s _th2 ), the consistency c may be determined as in _Equation 13 or Equation 14.

  Consistency is not limited to that which is lowered when there is a difference between the feature point on the three-dimensional map and the input image. For example, it may be increased when there is a difference between the feature point on the three-dimensional map and the input image. In this case, the higher the consistency, the smaller the weight.

  The evaluation value of the feature is limited to a value that is higher as the number of feature points described above is larger or higher in a region where the distribution of feature points is higher, and lower as the number of feature points is smaller or in a region where the distribution of feature points is coarser. It is not a thing. For example, the area may be lower as the number of feature points is larger or the distribution is denser, and may be higher as the number of feature points is smaller or the area is coarser. In this case, the higher the feature evaluation value is, the more weight is given to the feature points having lower consistency. However, in the case where the consistency is increased when a difference occurs between the feature point on the three-dimensional map and the input image, the higher the evaluation value of the feature, the higher the consistency is given to the feature point. Consistency evaluation is not limited to that performed on feature points on a three-dimensional map. When the 3D map is generated in a static environment, the feature points on the 3D map are not detected from moving objects. In that case, the consistency is evaluated for the feature points on the input image corresponding to the feature points of the three-dimensional map, and a weight is given.

<Effect>
According to the first embodiment, as the number of feature points used for position and orientation calculation increases, weights are given only to feature points that are less likely to be on the moving object, and as the number of feature points decreases, the feature points are on the moving object. A weight is given also to the feature point with high possibility. Thus, the position and orientation can be calculated with high accuracy and high robustness by adjusting the weight according to the number of feature points used for position and orientation calculation.

  According to the second embodiment, weights are given to feature points that are less likely to be on moving objects as the distribution of feature points used in position and orientation calculation is denser, and moving objects are located as the distribution is coarser. Weights are also given to feature points that are likely to be. In this way, it is possible to calculate the position and orientation with high accuracy and high robustness by preventing the deviation of the distribution of the features used for the position and orientation calculation.

  According to the third embodiment, as the number of feature points used for position and orientation calculation is larger, and the feature point distribution used for position and orientation calculation is denser, the feature points that are less likely to be on moving objects are weighted. Given. On the other hand, the smaller the number of feature points used for position and orientation calculation, and the more likely the feature points that are more likely to be on the moving object are the regions whose distribution is coarser. As described above, the position and orientation are calculated with high accuracy and high robustness by adjusting the weight according to the number of feature points used for the position and orientation calculation and preventing the bias of the distribution of the features used for the position and orientation calculation.

  Further, according to the fourth embodiment, it is possible to make the position and orientation highly accurate and robust by suppressing a large change in consistency using past consistency.

<Definition>
The three-dimensional information holding unit 110 is an example of a configuration that holds a three-dimensional map in which three-dimensional information of features existing in an image and position and orientation information of the imaging unit at the time of capturing the image are recorded. Any feature can be used as long as it retains a feature that can match the real space with the real space. For example, the coordinates in the world coordinate system of feature points such as both end points of the object edge may be held as features. In addition, the position and orientation of the marker in the world coordinate system and the pattern of the marker that can calculate the position and orientation of the imaging unit by drawing a specific pattern may be held as features.

  The consistency determining unit 130 is an example of a configuration that determines the consistency between the feature recorded in the three-dimensional map and the input image, and determines the degree of matching between the three-dimensional map and the real space (input image). Anything may be used as long as the consistency is determined based on it. For example, in the above embodiment, the smaller the difference in luminance value between the feature point on the three-dimensional map and the input image, the higher the consistency, and the larger the difference in luminance value between the feature point on the three-dimensional map and the input image. Although consistency was determined to be low, this is not restrictive. For example, the smaller the color difference between each feature point on the 3D map and the input image (elements in the RGB color space and HSV color space), the higher the consistency, and the color between the feature point on the 3D map and the input image. The greater the difference, the lower the consistency. Also, the smaller the difference in the distance value between the feature point on the 3D map and the input image, the higher the consistency, and the greater the difference in the distance value between the feature point on the 3D map and the input image, the lower the consistency. May be. Alternatively, a hand or a person may be detected from the input image, and the consistency of the feature points on the three-dimensional map projected in the detected hand or person region may be determined to be low. This is because the hand and person areas are moving objects.

  The feature evaluation unit 140 is an example of a configuration that calculates an evaluation value indicating a result of evaluating a presence state of a corresponding feature between a three-dimensional map and an input image. The feature evaluation unit 140 may be anything that evaluates feature points used for position and orientation calculation based on the number and distribution of features on the three-dimensional map and / or input image. As an example, in the first embodiment, the evaluation value is calculated to be higher as the number of feature points on the three-dimensional map and / or the input image increases, and the number of feature points on the three-dimensional map and / or the input image is calculated. The smaller the value, the lower the evaluation value is calculated. As an example, in the second embodiment, a higher evaluation value is calculated for a region having a dense distribution of feature points on the three-dimensional map and / or the input image, and the feature on the three-dimensional map and / or the input image is calculated. The lower the evaluation value is, the lower the distribution of points is.

  The weight calculation unit 150 is an example of a configuration that calculates a weight for controlling the influence of the features on the three-dimensional map on the calculation of the position and orientation of the imaging unit 180 based on the consistency and the evaluation value. The weight calculator 150 may be anything as long as it calculates weights using consistency and / or feature evaluation values. For example, the sum or product of the consistency and the evaluation value may be used as the weight. In addition, when the consistency is equal to or higher than a predetermined threshold, a value greater than 0 is given to the weight. When the consistency is lower than the predetermined threshold, the weight is set to 0, and the higher the evaluation value, the higher the predetermined threshold. But you can. Also, when the consistency is high, the weight is increased, when the consistency is low, the weight is decreased, and when the consistency is low, the lower the evaluation value is, the higher the evaluation value is compared with the higher evaluation value. It may be a thing.

  The position / orientation calculation unit may be anything as long as it calculates the position and orientation based on the three-dimensional map and the input image. For example, Klein's method, Pirchheim's method, or Engel's method may be used.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in the computer of the system or apparatus execute the program. It can also be realized by a process of reading and executing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

1: Information processing device, 110: Three-dimensional information holding unit, 120: Image input unit, 130: Consistency determining unit, 140: Feature evaluation unit, 150: Weight calculation unit, 160: Position and orientation calculation unit, 180: Imaging unit

Claims (15)

Holding means for holding a three-dimensional map in which three-dimensional information of features existing in the image and position and orientation information of the imaging unit at the time of capturing the image are recorded;
Input means for inputting an image captured by the imaging unit;
Determining means for determining consistency with the input image for the features recorded in the three-dimensional map;
Evaluation means for calculating an evaluation value indicating a result of evaluation of the existence state of the feature associated between the three-dimensional map and the input image;
Weight calculation means for calculating a weight for controlling the influence of the characteristics of the three-dimensional map on the calculation of the position and orientation of the imaging unit based on the consistency and the evaluation value;
An information processing apparatus comprising: calculation means for calculating the position and orientation of the imaging unit when the input image is captured using the three-dimensional map and the weight.   The information processing apparatus according to claim 1, wherein the evaluation unit calculates the evaluation value based on the number of the corresponding features.   The information processing apparatus according to claim 1, wherein the evaluation unit calculates the evaluation value based on a density distribution of the corresponding feature in the input image.   The information processing apparatus according to claim 1, wherein the evaluation unit calculates the evaluation value based on the number of the corresponding features and the density distribution of the corresponding features in the input image. .   The weight calculation means gives a value greater than 0 to the weight when the consistency is equal to or greater than a predetermined threshold, sets the weight to 0 when the consistency is less than the predetermined threshold, and further increases the evaluation value. The information processing apparatus according to claim 1, wherein the predetermined threshold is increased.   The weight calculation means increases the weight as the consistency is high, and increases the weight when the evaluation value is low as compared with the case where the evaluation value is high as the consistency is low. The information processing apparatus according to any one of claims 1 to 4.   The said determination means determines the said consistency based on the difference of the said three-dimensional map and the said input image regarding the said corresponding characteristic, The one of the Claims 1 thru | or 6 characterized by the above-mentioned. Information processing device.   The difference is at least one of a feature point of the three-dimensional map and a brightness value difference, a color difference, and a distance value difference at a projection position of the feature point on the input image. The information processing apparatus according to claim 7.   The information processing apparatus according to claim 1, wherein the determining unit holds the consistency history and determines the consistency based on the consistency history. Holding means for holding a three-dimensional map in which three-dimensional information of features existing in the image and position and orientation information of the imaging unit at the time of capturing the image are recorded;
An image input means for inputting an image captured by the imaging unit;
Consistency with the inputted image is acquired for the feature recorded in the three-dimensional map, and consistency of the feature is determined based on the acquired consistency and a history of consistency that is held. A determination means;
Weight calculation means for calculating a weight for controlling the influence of the characteristics of the three-dimensional map on the calculation of the position and orientation of the imaging unit based on the determined consistency;
An information processing apparatus comprising: calculation means for calculating the position and orientation of the imaging unit at the time of imaging the input image based on the three-dimensional map and the weight.   The information processing apparatus according to claim 1, wherein the weight calculation unit calculates the weight as a continuous value.   The three-dimensional map is a three-dimensional map in which a position and orientation closest to the position and orientation calculated immediately before for the imaging unit are recorded among a plurality of pre-registered three-dimensional maps. The information processing apparatus according to any one of claims 1 to 11. A control method of an information processing apparatus having a holding unit that holds a three-dimensional map in which three-dimensional information of features existing in an image and position and orientation information of an imaging unit at the time of imaging the image are recorded,
An input step of inputting an image captured by the imaging unit;
A determination step for determining consistency with the input image for features recorded in the three-dimensional map;
An evaluation step of calculating an evaluation value indicating a result of evaluation of the existence state of the feature associated between the three-dimensional map and the input image;
A weight calculation step of calculating a weight for controlling the influence of the characteristics of the three-dimensional map on the calculation of the position and orientation of the imaging unit based on the consistency and the evaluation value;
A control method for an information processing apparatus, comprising: a calculation step of calculating the position and orientation of the imaging unit at the time of imaging the input image using the three-dimensional map and the weight. A control method of an information processing apparatus having a holding unit that holds a three-dimensional map in which three-dimensional information of features existing in an image and position and orientation information of an imaging unit at the time of imaging the image are recorded
An image input step of inputting an image captured by the imaging unit;
Consistency with the inputted image is acquired for the feature recorded in the three-dimensional map, and consistency of the feature is determined based on the acquired consistency and a history of consistency that is held. A decision process;
A weight calculation step of calculating a weight for controlling the influence of the characteristics of the three-dimensional map on the calculation of the position and orientation of the imaging unit based on the determined consistency;
A control method for an information processing apparatus, comprising: a calculation step of calculating a position and a posture of the imaging unit at the time of imaging the input image based on the three-dimensional map and the weight.   The program for making a computer perform each process of the control method described in Claim 13 or 14.

Images