JP2014032623A - Image processor - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the accuracy of corresponding point detection between an observation object and the camera image thereof without increasing calculation cost.SOLUTION: An image processor includes a probability density DB 122b for learning local feature information and three-dimensional coordinates of a first feature point detected from an observation object, and giving probability density of the three-dimensional coordinates to optional local feature information, a normal management part for managing a normal of each first feature point, a feature point detection part 121 for detecting a second feature point from a camera image obtained by photographing the observation object, and extracting the local feature information thereof, a camera visual line estimation device 14 for estimating a visual line of the camera image, a matching part 122 for associating each second feature point with the first feature point having a higher matching score on the basis of the local feature information and the probability density database, and a second culling processing part 124 for comparing a normal of the associated first feature point with the visual line of the camera image, and eliminates correspondence point candidates whose angular difference exceeds a predetermined threshold.

Description

  The present invention relates to an image processing apparatus that associates three-dimensional coordinates of an observation object with two-dimensional coordinates of a camera image obtained by photographing the observation object with a camera, and in particular, without increasing the calculation cost and the amount of memory used. The present invention relates to an image processing apparatus that enables association with high accuracy.

  In recent years, AR (Augmented Reality) technology that adds real-time images by processing images in a real space has been realized on PCs connected to web cameras and mobile phone terminals with cameras. Yes. In the AR technology, it is necessary to estimate a camera posture (an external parameter of the camera) with respect to an object in a camera image, and a method using a sensor or a reference marker is used. In addition, a technique for estimating a camera posture using a three-dimensional object whose shape and image information are known as an object is being studied.

  In Patent Document 1, feature points detected from both registered images and input images of a plurality of objects are matched, a highly correlated registered image is specified, and the camera posture is determined from the combination of the feature points and the shape of the object. A technique for estimating the above is disclosed.

  Non-Patent Document 1 discloses a technique for increasing the accuracy of corresponding points (the proportion of points correctly associated) by setting a threshold value for the output of the classifier.

Patent 4715539

D. Wagner et al., "Real-Time Detection and Tracking for Augmented Reality on Mobile Phones," Visualization and Computer Graphics, IEEE Transactions on, vol. 16, no. 3, pp. 355 -367, may / june. 2010 .

  In Patent Document 1, in order to perform feature point matching between an input image and a registered image, there are few corresponding points when a registered image recorded in a state close to the posture of a three-dimensional object in the input image does not exist in the database. Thus, there is a problem that feature point matching becomes difficult.

  In addition, when the number of registered images is increased by shooting the object from various directions, the calculation cost is high for identifying highly correlated registered images, and the size of the classifier increases, so the amount of memory used by the computer increases. There was a problem to do.

  Non-Patent Document 1 has a problem that a high-accuracy corresponding point can be obtained when the object is a planar image, but sufficient accuracy cannot be obtained for a three-dimensional object.

  An object of the present invention is to solve the above technical problem and provide an image processing apparatus capable of obtaining a large number of corresponding points with high accuracy without increasing the calculation cost and the amount of memory used.

  In order to achieve the above object, the present invention is characterized in that the following configuration is provided in an image processing apparatus that associates the three-dimensional coordinates of an observation object with the feature points of a camera image obtained by photographing the observation object. There is.

  (1) A probability density database that learns local feature information and three-dimensional coordinates of a first feature point detected from an observation target and gives a probability density of the three-dimensional coordinates to arbitrary local feature information, and each first feature Means for detecting a normal of a point, feature point detecting means for detecting a second feature point from a camera image obtained by photographing an observation object, local feature information extracting means for extracting local feature information from each second feature point, Means for estimating the line of sight of the camera image; matching means for associating each second feature point with a first feature point having a higher matching score based on the local feature information and the probability density database; Corresponding point output means is provided for comparing the normal of one feature point with the line of sight of the camera image and outputting as a corresponding point a correspondence relationship in which the angle difference is below a predetermined threshold.

  (2) A gaze database that learns each local feature information and gaze of feature points detected from a large number of two-dimensional images projected from different gazes, and gives a gaze probability density to arbitrary local feature information Further, the means for estimating the line of sight of the camera image estimates the line of sight of the camera image based on the local feature information of the second feature point detected from the camera image and the probability density of the line of sight.

  (3) Based on the angle difference between the normal line of each first feature point and the line of sight of the camera image, means for generating a mask for excluding the first feature point whose angle difference exceeds a predetermined threshold from the matching target And the corresponding point output means performs matching only on the first feature point that is not masked, and the mask has an angular difference between the normal line and the line of sight of the camera image for each first feature point. A multi-value mask value corresponding to the multi-value mask value is set, and the corresponding point output means performs matching using the multi-value mask value as a weight.

  (4) Of the corresponding points output from the corresponding point output means, comprising camera posture estimation means for estimating the posture of the camera using only corresponding points satisfying the geometric constraint condition, the corresponding point output means, The threshold value is dynamically changed so that the threshold value becomes gentler as the number of corresponding points that can be used for camera posture estimation in the camera posture estimation means is smaller.

  According to the present invention, the following effects are achieved.

  (1) Corresponding points are not only the matching score between the feature points detected from the observation target and the local feature information of the feature points detected from the camera image, but also the direction of the observation target (the normal of the feature point) and the relevant point Since the detection is performed in consideration of the angle difference with the direction of the camera (camera line of sight) that captured the observation object, it is possible to detect corresponding points with high accuracy with a small amount of calculation.

  (2) A gaze database that learns each local feature information and gaze of feature points detected from a large number of two-dimensional images projected from different gazes, and gives a gaze probability density to arbitrary local feature information Since it is provided, it is possible to accurately estimate the line of sight of the camera image without providing a GPS function or a direction sensor.

  (3) The matching score between the feature points detected from the observation object and the local feature information of the feature points detected from the camera image, the direction of the observation object (the normal of the feature point) and the camera that captured the observation object If the weighting is performed based on the angle difference with respect to the direction (camera line of sight), for example, even if the angle difference is large, if the matching score is sufficiently high, the corresponding corresponding point can be detected flexibly.

  (4) Since the threshold value of the angle is dynamically controlled based on whether the output corresponding points satisfy the geometric constraint condition, the necessary and sufficient number of corresponding points can be obtained. Become.

1 is a block diagram of an AR system to which first and second embodiments of the present invention are applied. FIG. It is a block diagram of a 1st embodiment of the present invention. It is a figure for demonstrating the function of a 2nd culling process. It is the figure which showed the method of setting the threshold value (theta) ref of an angle difference dynamically. It is a block diagram of a 2nd embodiment of the present invention. It is the figure which showed the relationship between the number of corresponding points (horizontal axis) and matching accuracy (vertical axis) of the embodiment using no mask and the embodiment using a mask. It is a block diagram of an AR system to which the third and fourth embodiments of the present invention are applied. It is a block diagram of a 3rd embodiment of the present invention. It is a block diagram of a 4th embodiment of the present invention.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a block diagram showing a configuration of an AR system 1 to which the present invention is applied, and is used by being mounted on an information terminal such as a mobile phone, a smartphone, a PDA, or a notebook PC.

  The imaging device (camera) 10 is a camera module or a WEB camera device mounted on a mobile terminal or the like. The imaging device (camera) 10 captures the observation target 2 and outputs the camera image Ica to the display device 11 and the camera posture estimation device 12. .

  The camera line-of-sight estimation device 14 is based on, for example, its own position and orientation detected by an orientation sensor and a GPS function (both not shown) mounted on the information terminal, and the position of the observation object 2 given in advance. Thus, the relative line of sight (camera line of sight) of the camera 10 with respect to the observation object 2 is roughly estimated. In the present invention, as will be described in detail later, the accuracy of corresponding points is improved by narrowing down corresponding point candidates in advance based on the camera line of sight.

  The camera posture estimation device 12 performs feature point matching between the local feature amount extracted from each feature point Pm of the observation object 2 and the local feature amount extracted from each feature point Pc of the camera image Ica. A pair including a feature point Pm having a higher matching score and a smaller angle difference between the direction of the observation target (normal line of the feature point) and the direction of the camera that captured the observation target (camera line of sight). Among these corresponding points, the camera posture is estimated using only the truly correct corresponding points satisfying the geometric constraint condition.

  The camera pose is expressed in the form of a matrix called camera external parameters, and includes information on the position and orientation of the camera in the three-dimensional space. This and the camera's intrinsic parameters, called camera internal parameters, are the positions of the main axes. The appearance of the object in the screen is determined by a matrix including the information and other optical distortion parameters.

  In the present embodiment, it is assumed that internal parameters and distortion parameters are acquired in advance by camera calibration and the distortion is removed, and the camera posture estimated by the camera posture estimation device 12 is output to the display device 11. When the AR system 1 targets a plurality of types of three-dimensional objects, the corresponding types of observation targets 2 are also output to the display device 11 together with the camera posture.

  The display device 11 is a monitor device that can post a camera image Ica continuously acquired by the imaging device 10 to the user, and may be a display of a portable terminal. Further, a form such as a head-mounted display (HMD) may be used. In particular, in the case of a see-through type HMD, the camera image Ica is not displayed, and only the additional information can be superimposed and displayed in the field of view. When the display device 11 is a display, the additional information input from the additional information DB to the camera image Ica is superimposed and displayed at the position corrected by the camera posture input from the camera posture estimation device.

  The additional information database 13 is a storage device composed of a hard disk drive, a semiconductor memory module, or the like. When the AR system 1 recognizes the position of the observation target 2, the additional information database 13 is displayed on the display device 11 so as to be superimposed on the observation target 2. And the additional information regarding the observation object 2 corresponding to the camera posture estimated by the camera posture estimation device 12 is output to the display device 11.

  In FIG. 1, the globe is handled as an example of the observation object 2. However, even when the object has a primitive structure such as a rectangular parallelepiped shape, a cylindrical shape, a spherical shape, or a complicated structure, the 3D model is If given, a similar AR system can be constructed. And according to such an AR system, the surface of the earth is displayed in a superimposed manner so that the altitude is visually displayed, the superimposed display of past continental shapes, and the information updated when there is a change in borders or country names are displayed in a superimposed manner. An example of use in which a country name pointed to in combination with gesture recognition is displayed is assumed.

  In the camera posture estimation device 12, the feature point identification unit 12b detects a feature point using a feature point detector from the camera image Ica output from the imaging device 10, as described in detail later, and the feature point and its feature point Classify feature points based on local feature information in the vicinity. Then, the three-dimensional coordinates, local feature amounts and normals of each feature point learned from the observation object 2 in advance are compared with the two-dimensional coordinates, local feature amounts and line of sight of each feature point detected from the camera image Ica. , Output corresponding points with higher accuracy.

  The camera posture calculation unit 12a calculates a camera posture based on a plurality of corresponding points output from the feature point identification unit 12b, and outputs the calculation result to the display device 11. Methods for estimating camera posture (external camera parameters) from 2D and 3D coordinates of corresponding points have been studied, and the relationship between 3D and 2D coordinates is generally expressed by the following equation (1) It is represented by

  [u, v, 1] ^ T = sAW [X, Y, Z, 1] ^ T… (1)

  Here, [u, v] and [X, Y, Z] represent a two-dimensional pixel coordinate value and a three-dimensional coordinate value, respectively, and [·] ^ T represents a transposed matrix. A and W represent an internal parameter and an external parameter (camera posture) of the camera, respectively. The internal parameters of the camera are obtained in advance by camera calibration.

  Camera posture W = [R, t] = [r1, r2, r3, t], which is represented by a rotation matrix R and a translation vector t. The camera posture W can be estimated using the match between the three-dimensional coordinates [X, Y, Z, 1] ^ T and the two-dimensional coordinates [u, v, 1] ^ T and the internal parameters of the camera.

  Here, since the corresponding points of the input 2D coordinates and 3D coordinates include incorrect corresponding points, only correct corresponding points (inliers) are selected from the corresponding points input by a method such as the sampling method. It is common to extract and estimate the camera pose. Here, the camera posture calculation unit 12a can feed back the absolute number of inliers obtained by the sampling method to the feature point identification unit 12b.

[First embodiment]
FIG. 2 is a functional block diagram of the first embodiment of the feature point identification unit 12b. The feature point detection unit 121 detects the feature point Pc from the camera image Ica, and outputs the feature point Pc and its local feature information to the matching unit 122. The camera image Ica may be either a frame captured in real time or a recorded continuous frame, and may be a single photograph showing a pre-registered object without being continuous. .

  As the feature point detector, any kind of detector can be used as long as it can identify a two-dimensional coordinate having a feature, such as a Harris corner detector, a Hessian key point detector, or a FAST corner detector. In the present embodiment, the FAST corner detector employed in the method of Non-Patent Document 1 is used.

  The local feature information is information for identifying feature points such as a SIFT descriptor and a SURF descriptor. In the present invention, any kind of general local feature information can be used. In the present embodiment, a patch image adopted by the method of Non-Patent Document 1 is used as local feature information.

  Here, the patch image is an image cut out from the original image at a specific size, and in this embodiment, an arbitrary width and height (for example, a width of 32 pixels and a height of 32 pixels) centering on the feature point. It is the image of. When a feature point also has size (scale) information, the width and height of the patch image can be changed according to the size. The patch image is acquired at a very high speed because it is only necessary to cut out the image as compared with general local feature information.

  The matching unit 122 includes a classifier 122a and a probability density database 122b, and all feature points Pm detected from a large number of projection images obtained by projecting a three-dimensional model of the observation object 2 in two dimensions from various directions in advance. The local feature information (patch image) is applied to the classifier 122a. Then, by calculating the correspondence between the classification results of each local feature information and the three-dimensional coordinates, a probability density distribution that can centrally manage the relationship between the local feature information and the probability density of the three-dimensional coordinates is constructed. Is registered in the probability density database 122b as a learning model.

  As a classifier 122a that employs such a matching method, a Ferns classifier that processes patch images at high speed can be used. The Ferns classifier is composed of a plurality of decision trees (Fern), and each Fern is composed of a multi-stage branch point (node) having a decision rule for branching a patch image and a terminal end (leaf) of the node. The decision rule is to branch left and right depending on the magnitude relationship of the luminance of two pixels randomly selected from the patch image. The latter node further branches the patch image by another decision rule, but the nodes having the same number of steps have the same decision rule. Therefore, the node type is equal to the number of nodes. The node (leaf node) where the patch image finally arrives becomes the patch image classification result.

  Each leaf node holds a probability density of three-dimensional coordinates corresponding to a patch image acquired by learning in advance in the probability density DB 122b. Since the Ferns classifier has multiple Ferns, the patch image is input to each Fern, the probability density corresponding to each arrived leaf node is obtained, and the probability is multiplied using a naive Bayes classifier. The probability density of the three-dimensional coordinates corresponding to the final patch image is determined.

  The matching unit 122 obtains the probability density of the three-dimensional coordinates corresponding to the patch image by applying the patch image of the feature point Pc detected from the camera image Ica to the Ferns classifier. Then, the three-dimensional coordinates having the highest probability (likelihood) are associated with each patch image (feature point) as a corresponding point, and the likelihood is output to the first culling processing unit 123 as a matching score.

  The first culling processing unit 123 sets a certain threshold (score threshold) for the matching score, and removes pairs whose matching score is less than the predetermined threshold from the corresponding point candidates, while the second culling processing unit 124 Is output as a corresponding point candidate. In general, since the matching points that are mistakenly associated with each other do not have a very high matching score, it is possible to increase the proportion of correct corresponding points (inliers) in the corresponding point candidates by this first culling process.

  The second culling processing unit 124 performs a second culling process based on the camera line of sight given by the camera line-of-sight estimation device 14 for the corresponding point candidate that has undergone the first culling process, and the response after the second culling process Point candidates are output as corresponding points. Specifically, among the corresponding point candidates after the first culling processing, the corresponding point where the normal of the feature point Pm faces the front with respect to the camera 10, that is, the method of the feature point Pm detected from the observation object 2. The line and the line of sight Pc of the camera image Ica are compared, and only corresponding point candidates whose angle difference θ is less than a threshold θref (angle threshold) are output as corresponding points.

  FIG. 3 is a diagram for explaining the function of the second culling processing unit 124. In general, the way in which a region around a point on the surface of a three-dimensional object is reflected (projected) on the camera image is determined by the focus of the camera. As the angle difference θ of the surface normal to the straight line (line of sight) from the point to the point on the surface of the three-dimensional object increases, the deformation increases and the matching accuracy deteriorates. When the angle difference θ exceeds a right angle, the point (and the surrounding local region) is not reflected in the camera image and cannot be matched.

  That is, it is possible to predict the matching accuracy of the three-dimensional coordinates by obtaining information about the camera line of sight from the outside. Therefore, by performing the second culling process, it is possible to exclude corresponding point candidates that are estimated to be inaccurate and increase the ratio of inliers in the corresponding points.

  Here, when the observation object 2 is configured as a plane like a rectangular parallelepiped or has a shape including a plane, the angle difference θ between the line of sight of the camera image Ica and the normal of the feature point Pm is determined by each feature point. Similar values are expected when they are on the same plane. Therefore, the second culling processing unit 124 calculates the angle formed between the line of sight and the normal of each feature point between the normal of the center of the plane (polygon) constituting the observation object 2 and the line of sight of the camera image. The calculation cost can be reduced by calculating the angle difference θ and treating it as the angle difference θ with respect to the normal of all the feature points Pm on the polygon. For example, when the object is a rectangular parallelepiped (hexahedron) and there are 100 three-dimensional coordinates included in the corresponding points, this processing makes it possible to calculate the angle 100 times 6 times.

  In the above embodiment, the description has been made assuming that the threshold value θref of the angle difference θ set in the second culling processing unit 124 is one, but the present invention is not limited to this, and the camera 10 May be set to a different threshold value θref for each angle at which the observation object 2 is viewed or for each feature point Pm of the observation object 2.

  That is, when the observation object 2 is a globe, for example, the ocean area occupies most when viewed from one angle, while the land area occupies most when viewed from another angle. In addition, the texture tendency of the observation object 2 may vary greatly depending on the angle. In such a case, since the matching accuracy with respect to the angle difference between the normal line and the line of sight changes greatly, the threshold value θref may be made variable according to the angle at which the observation object 2 is expected.

  From the same point of view, the matching accuracy with respect to the angle difference between the normal and the line of sight largely depends on the texture of the feature point, and there are a texture part with a large change in local feature information and a small texture part with a change in the line of sight direction. There is a case. Therefore, for feature points whose local feature information is robust to the line of sight, the threshold value θref may be set larger than those of other feature points.

  FIG. 4 is a diagram showing a method for dynamically setting the threshold value θref as described above. The vertical axis represents the matching accuracy (%), and the horizontal axis represents the angle difference (θ) between the normal line and the line of sight. Represents.

  If the threshold value θref is set for each feature point, that is, for each three-dimensional coordinate of the feature point Pm detected from the observation object 2, matching is performed on test images obtained by photographing the observation object 2 at various angles. Then, the relationship between the matching accuracy and the angle difference θ is plotted for each three-dimensional coordinate x of the feature point. Then, the threshold θrefx (θref1, θref2...) Corresponding to the intersection of the curve connecting each plot and the target accuracy is obtained for each three-dimensional coordinate x, and the pair including the feature point corresponds to the three-dimensional coordinate x. The second culling process may be performed using the threshold value θrefx.

  At this time, a test image for evaluation may be artificially created using the 3D model of the observation object 2. When a test image is artificially created by 3D rendering, since the set camera parameter is known, the angle difference between the normal and the line of sight can be accurately obtained, so that each threshold value θrefx can be accurately calculated. Become.

  When obtaining the intersection between the curve connecting the plots and the target accuracy, the plot may be perturbed when the number of test images is small, and the position of the intersection may not be stable. In such a case, a straight line approximating the plot may be calculated by linear regression from a point group constituting a plot near the set accuracy, and the intersection of the approximate straight line and the target accuracy may be calculated.

  Regarding the relationship between the threshold value θref and the matching accuracy, the accuracy increases if only the feature point photographed from near the front is left by setting the threshold value of the angle difference θ small. However, since the corresponding points of the inliers may also be culled, the absolute number decreases instead of increasing the proportion of inliers included in the corresponding points to be output. When there is a certain number of inliers, the ratio increases to speed up the subsequent camera posture calculation process. However, when the absolute number of inliers decreases, the camera posture calculation itself cannot be performed.

  Therefore, the second culling processing unit 124 may receive the feedback of the absolute number of inliers used for calculation of the camera posture from the camera posture calculation unit 12a and adjust the threshold value.

  In general, the camera posture cannot be calculated from less than 4 corresponding points, and at least 8 to 10 corresponding points are required to obtain a stable calculation result. Therefore, the second culling processing unit 124 evaluates the absolute number of corresponding points for each frame, and when it becomes equal to or smaller than a certain number (for example, 10), the second culling processing unit 124 relaxes (increases) the threshold θref, thereby calculating the camera posture Can be ensured.

  Further, since the relationship between the threshold and the matching accuracy also changes depending on the threshold used in the first culling processing unit 123, the second culling processing unit 124 determines its own angle according to the threshold set in the first culling processing unit 123. The threshold value θref may be adjusted.

  Specifically, the relationship between the angle and the matching accuracy and the threshold value of the angle calculated therefrom are recorded for each threshold set by the first culling processing unit 123, and the setting of the angle threshold is set by the first culling processing unit 123. Perform according to the threshold.

[Second Embodiment]
FIG. 5 is a block diagram showing a configuration of the feature point identification unit 12b according to the second embodiment of the present invention, and the same reference numerals as those described above represent the same or equivalent parts. In the present embodiment, in comparison with the feature point identification unit 12b of the first embodiment described with reference to FIG. 2, the second culling processing unit 124 is omitted, and a mask generation unit 126 is added instead. There is a feature in the point.

  The mask generation unit 126 determines the angle difference θ between the camera line-of-sight estimated by the camera line-of-sight estimation device 14 and the normal line detected and registered for each three-dimensional coordinate of the feature point Pm detected from the observation object 2. Are all calculated. Then, a binary array whose length is the number of feature points of the observation object 2 is “1” for a three-dimensional coordinate having an angle difference θ exceeding a predetermined threshold θref and “0” for a three-dimensional coordinate not exceeding the threshold θref. Generate a mask for

  In the matching unit 122, when performing corresponding point matching between each feature point Pm of the observation target 2 and each feature point Pc of the camera image Ica, the feature point Pm of the observation target 2 corresponds to the three-dimensional coordinates in advance. By applying the mask value, it is determined whether each feature point Pm is included in the matching target.

  That is, the feature point Pm corresponding to the mask value “1” is excluded from the matching candidates, and the feature point Pm corresponding to the mask value “0” is set as the matching candidate. Therefore, if the mask values are all “0”, no mask processing is performed.

  The first culling processing unit 123 performs corresponding point matching only between the patch image extracted from the unmasked feature point Pm and the patch image extracted from each feature point Pc, and the matching score is predetermined. A combination exceeding the threshold is output as a corresponding point.

If the mask process is employed instead of the second culling process as in the present embodiment, there is a possibility that more corresponding points can be acquired as compared with the case where the second culling process is performed.
FIG. 6 is a diagram showing a case where the number of corresponding points can be increased by mask processing. As a result of detecting the feature point Pc from the camera image Ica and performing the matching with the three-dimensional coordinates by the matching unit 122, the feature point Pc has a pattern similar to the feature point Pm2 that should be matched if originally. The feature point Pm1 exists, the matching score between the incorrect corresponding Pc-Pm1 is the highest among all the feature points Pm, and the matching score between the correct corresponding Pc-Pm2 is the second.

  In the first embodiment, by performing the second culling process, the matching unit 122 first matches the feature point Pc with the three-dimensional coordinate of the feature point Pm1 having the highest matching score, so that the first corresponding point is obtained. Configure. Since the matching score of the first corresponding point satisfies the score threshold value, the first culling process is passed, and then the angle is evaluated by the second culling process. Here, since the angle (105 °) of the first corresponding point does not satisfy the angle threshold (70 °), the first corresponding point is excluded from the corresponding points, and Pc is not included in the corresponding points after the culling process.

  In contrast, when the mask process is first performed instead of the second culling process as in the present embodiment, the feature point Pm1 is masked and the feature point Pc is the most unmasked three-dimensional coordinate. A second corresponding point is configured by matching with the feature point Pm2 having a high matching score. Since the score (8 points) and the angle (0 °) of the second corresponding point both satisfy the score threshold value (7 points) and the angle threshold value (70 °), the corresponding point after the culling process has a feature point Pc. Is included.

  As described above, when the mask process is performed, more corresponding points can be obtained than when the second culling process is performed. Therefore, it is expected that the number of correct corresponding points increases. On the other hand, however, there is a possibility that erroneous corresponding points may increase, and there is generally a trade-off between the number of corresponding points and accuracy.

  In the above embodiment, the description has been given on the assumption that the mask value is binary. However, the present invention is not limited to this, and the smaller the angle difference θ, for example, according to the magnitude of the angle difference θ. You may employ | adopt the multi-value of 3 values or more used as a big value. The matching unit 122 uses this mask value as a weight value, and the matching score is increased and corrected as the mask value is increased, and decreased as the mask value is decreased. In this way, even a combination with a large angle difference θ is regarded as a corresponding point if the matching score is sufficiently high, so that flexible culling processing becomes possible.

  FIG. 7 is a block diagram showing the configuration of the AR system 1 to which the third and fourth embodiments of the present invention are applied. The same reference numerals as those described above represent the same or equivalent parts, and the explanation thereof is as follows. Omitted. Compared with the configuration of FIG. 1, the camera gaze estimation device 14 is omitted, and a feature equivalent to this is implemented in the feature point identification unit 12b.

[Third embodiment]
FIG. 8 is a functional block diagram showing the configuration of the main part of the third embodiment of the present invention, and shows the configuration of the feature point identification unit 12b. In this embodiment, the camera gaze estimation unit 125 that acquires local feature information of each feature point of the camera image from the feature point detection unit 121 and associates the local feature information with the gaze of the camera image (camera gaze). The camera gaze estimation device 14 is characterized in that it is provided in place of the camera gaze estimation device 14.

  The camera line-of-sight estimation unit 125 estimates the camera line-of-sight using a Ferns classifier as in the matching unit 122. Here, since the estimation target is not a three-dimensional coordinate but a camera line of sight, each leaf node of the Ferns classifier is detected from a large number of two-dimensional images that are projected by dividing the observation target 2 equally with various lines of sight. A probability density that learns each local feature information and line of sight of each feature point Pm and gives a probability density of the camera line of sight to arbitrary local feature information is held in advance.

  At runtime, the patch image cut out from each feature point Pc of the camera image is applied to the classifier to classify the patch image into the camera line of sight, and the most classified camera line of sight is the final camera for the patch image The line of sight estimation result is output to the second culling processing unit 124.

  At this time, the line-of-sight estimation unit 125 can reduce the data size necessary for line-of-sight estimation by sharing the classifier determination rule for line-of-sight estimation with the matching unit 122. In that case, the line-of-sight estimation unit 125 and the matching unit 122 use a classifier having two types of classes and probability densities and one type of decision tree group.

  According to the present embodiment, the camera line of sight can be estimated with high accuracy even in an information terminal that is not equipped with an orientation sensor, a GPS function, and the like. ) And a feature point detection that also considers the angle difference between the direction of the camera (camera line of sight) that captured the observation target.

[Fourth embodiment]
FIG. 9 is a functional block diagram showing the configuration of the main part of the fourth embodiment of the present invention, and shows the configuration of the feature point identification unit 12b. The present embodiment is characterized in that a mask for corresponding point matching is created based on the line-of-sight information estimated by the line-of-sight estimation unit 125 and a mask generation unit 126 is provided to provide the matching unit 122 with the mask.

  DESCRIPTION OF SYMBOLS 1 ... AR system, 2 ... Observation object, 10 ... Imaging apparatus, 12 ... Camera attitude estimation apparatus, 12a ... Camera attitude calculation part, 12b ... Feature point identification part, 13 ... Additional information database, 14 ... Camera gaze estimation apparatus

Claims (9)

In the image processing apparatus that associates the three-dimensional coordinates of the observation object with the feature points of the camera image obtained by photographing the observation object,
A probability density database that learns local feature information and three-dimensional coordinates of a first feature point detected from an observation target, and gives a probability density of the three-dimensional coordinates to arbitrary local feature information;
Means for detecting a normal of each first feature point;
Feature point detecting means for detecting a second feature point from a camera image obtained by photographing an observation object;
Local feature information extracting means for extracting local feature information from each second feature point;
Means for estimating the line of sight of the camera image;
Matching means for associating each second feature point with a first feature point having a higher matching score based on the local feature information and the probability density database;
And a corresponding point output unit that compares the normal line of the associated first feature point with the line of sight of the camera image and outputs a pair whose angle difference is below a predetermined threshold as a corresponding point. Image processing device. Further comprising means for excluding the matching score other than the pair whose matching score is higher than a predetermined threshold,
2. The image processing according to claim 1, wherein the corresponding point output unit outputs, as corresponding points, corresponding point candidates whose angle difference is lower than a predetermined threshold among the corresponding point candidates having a high matching score. apparatus. Based on the angle difference between the normal line of each first feature point and the line of sight of the camera image, the apparatus further comprises means for generating a mask for excluding the first feature point whose angle difference exceeds a predetermined threshold from the matching target. ,
The image processing apparatus according to claim 1, wherein the corresponding point output unit performs matching only on the first unmasked first feature point. In the mask, for each first feature point, a multi-value mask value is set according to the angle difference between the normal line and the line of sight of the camera image,
The image processing apparatus according to claim 3, wherein the corresponding point output unit performs matching using the multi-value mask value as a weight. Further comprising means for detecting the position and orientation of the device itself;
The means for estimating the line of sight of the camera image estimates the line of sight of the camera image based on the position and orientation of the device itself and the position of the observation target separately provided. An image processing apparatus according to claim 1. It further includes a gaze database that learns each local feature information and gaze of feature points detected from a number of two-dimensional images obtained by projecting the observation target with different gazes and gives a gaze probability density to arbitrary local feature information. ,
The means for estimating the line of sight of the camera image estimates the line of sight of the camera image based on the local feature information of the second feature point detected from the camera image and the probability density of the line of sight. 5. The image processing device according to any one of 4.   7. The normal line detection unit typifies a normal line of a plurality of feature points belonging to the same plane to be observed by a normal line of one of the feature points. Image processing apparatus.   8. The image processing apparatus according to claim 1, wherein in the corresponding point output unit, a threshold value related to the angle difference is set to a unique value for each first feature point. Of the corresponding points output from the corresponding point output means, comprising camera posture estimation means for estimating the camera posture using only corresponding points satisfying the geometric constraint condition,
The said corresponding point output means changes the said threshold value dynamically so that a threshold value may become loose, so that there are few corresponding points which could be utilized for the camera attitude estimation in the said camera attitude estimation means. The image processing apparatus according to any one of 8.