JP2014106641A - Image processing apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve accuracy and speed of feature point matching between an object and a camera image without increasing calculation cost or memory consumption.SOLUTION: An image processing apparatus for associating three-dimensional coordinates of first feature points detected from images obtained by observing an object M from a plurality of directions with second feature points detected from a camera image Ica obtained by capturing the object includes: a sorter 32a which sorts local feature quantity on the basis of a predetermined sorting rule; a database 32b of probability distribution for applying three-dimensional coordinates and view point probability distribution to a result of sorting the local feature quantity; an identification part 32d which identifies correspondence between the local feature quantity of the second feature point, the three-dimensional coordinate, and the view point, on the basis of a sorting result obtained by applying the local feature quantity of the second feature point to the sorter 32a and the probability distribution; and accuracy improving means 3b which improves accuracy of correspondence between the local feature quantity of the second feature point and the three-dimensional coordinate on the basis of the correspondence between the local feature quantity of the second feature point and the view point.

Description

  The present invention relates to an image processing apparatus that associates three-dimensional coordinates of an observation object with two-dimensional coordinates of a camera image obtained by photographing the observation object with a camera, and particularly uses it for estimation of a camera posture. The present invention relates to an image processing apparatus that enables high-speed and high-accuracy association without increasing costs and memory usage.

  In recent years, AR (Augmented Reality) technology that adds real-time images by processing images in a real space has been realized on PCs connected to web cameras and mobile phone terminals with cameras. Yes. In the AR technology, it is necessary to estimate a camera posture (an external parameter of the camera) with respect to an object in a camera image, and a method using a sensor or a reference marker is used. In addition, a technique for estimating a camera posture using a three-dimensional object whose shape and image information are known as an object is being studied.

  Patent Document 1 discloses a technique for preparing a stored image obtained by photographing a three-dimensional object from a plurality of viewpoints and specifying the most similar stored image (viewpoint) by feature point matching between the input image and the stored image. Yes. After the viewpoint is specified, corresponding points are acquired between the stored image and the input image, and the camera posture is calculated.

  Non-Patent Document 1 discloses a technique for estimating a viewpoint by classifying feature points into candidate viewpoints for an object by classifying feature points. Patent Document 2 discloses a technique for acquiring highly accurate corresponding points using viewpoint information on an object.

JP 2012-83855 A Japanese Patent Application No. 2012-174320

Yuki Hirai, Satoru Suzuki, Hironobu Fujiyoshi, "High-speed 3D object recognition by two-stage Randomized Trees", Image Sensing Symposium, IS1-15, 2011.

  In Patent Documents 1 and 2, since the accuracy of viewpoint estimation depends on the shooting angle of a stored image, there is a problem that sufficient accuracy of viewpoint estimation cannot be obtained for an object shot from a specific direction. It was.

  In Non-Patent Document 1, although sufficient viewpoint estimation accuracy is obtained for a three-dimensional object photographed in a limited posture range, the estimation accuracy deteriorates when the number of viewpoint candidates to be estimated increases. Therefore, there has been a problem that sufficient viewpoint estimation accuracy cannot be obtained for an input image obtained by photographing a target three-dimensional object from an arbitrary direction.

  Furthermore, when the conventional techniques of Patent Document 1 and Non-Patent Document 1 are applied to the camera posture estimation problem, it is necessary to perform two-stage matching processing of viewpoint estimation and corresponding point acquisition, so that the entire camera posture estimation processing is performed. There has been a problem that the amount of calculation increases.

  An object of the present invention is an image that solves the above-described technical problems and that can improve the accuracy and speed of feature point matching between an observation target and its camera image without increasing the calculation cost or the amount of memory used. It is to provide a processing apparatus.

  In order to achieve the above object, the present invention provides a three-dimensional coordinate of each first feature point detected from each image obtained by observing the observation object from a plurality of viewpoints and a camera image obtained by photographing the observation object. In the image processing apparatus for associating each second feature point detected from the classification means for classifying the local feature amount according to a predetermined classification rule, and each correspondence relationship between the three-dimensional coordinates and the viewpoint for the classification result of the local feature amount A local feature amount of the second feature point, a three-dimensional coordinate, and a probability distribution obtained by applying the local feature amount of the second feature point to the classification means and the probability distribution. Based on the identification means for identifying the correspondence between the viewpoint and the correspondence between the local feature of the second feature point and the viewpoint, the correspondence between the local feature of the second feature and the three-dimensional coordinate is highly accurate. The point which comprises There is a feature.

  According to the present invention, only by applying the local feature amount of the feature point detected from the camera image once to the classifier, both the 3D coordinates and the viewpoint corresponding to the local feature amount can be obtained simultaneously. It is possible to improve the accuracy and speed of feature point matching with a camera image without increasing the calculation cost and the amount of memory used.

It is a functional block diagram showing the configuration of an AR system to which the present invention is applied. It is the figure which showed the function of the camera viewpoint estimation part 3a typically. It is a functional block diagram of 1st Embodiment of the camera viewpoint estimation part 3a. It is the figure which showed the classification method of the patch image using the Ferns classifier in 1st Embodiment. It is a flowchart of a learning process. It is a schematic diagram of a learning process. It is the figure which showed the selection method of the feature point used for teacher data. It is the figure which showed the setting method of a viewpoint candidate. It is the figure which showed the calculation method of the focus position of a camera. It is a flowchart of an identification process. It is the figure which showed how to obtain | require 3D coordinates and a viewpoint. It is a functional block diagram of 2nd Embodiment of the camera viewpoint estimation part 3a. It is the figure which showed the classification method of the patch image using the Ferns classifier in 2nd Embodiment. It is the figure which showed the setting method of the viewpoint candidate corresponding to the discretization to a distance direction.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram illustrating a configuration of a main part of an AR system to which the present invention is applied, and is used by being mounted on an information terminal such as a mobile phone, a smartphone, a tablet terminal, a PDA, or a notebook PC.

  The imaging device (camera) 1 is a digital camera module or a WEB camera device mounted on a portable terminal or the like. The imaging device (camera) 1 captures the observation object M and outputs the camera image Ica to the display device 2 and the camera posture estimation device 3. To do. The camera posture estimation device 3 is based on a camera viewpoint estimation unit 3a that estimates a relative viewpoint (camera viewpoint) of the camera 1 with respect to the observation target M when the observation target M is photographed. Based on the correspondence between the three-dimensional (3D) coordinates of the observation object M and the two-dimensional (2D) coordinates of the camera image, which is highly accurate, and the highly accurate 2D-3D coordinates. The camera posture calculation unit 3c that calculates the camera posture is the main component.

  The additional information database (DB) 4 is a storage device configured by a hard disk drive, a semiconductor memory module, or the like, and is superimposed on the observation target M on the display device 2 when the AR system 1 recognizes the position of the observation target M. CG, two-dimensional image or text information to be displayed is held, and additional information of the observation target M corresponding to the camera posture estimated by the camera posture estimation device 3 is provided to the display device 2.

  The display device 2 is a monitor device that can post a camera image Ica continuously acquired by the camera 1 to the user, and may be a display of a portable terminal. Further, a form such as a head-mounted display (HMD) may be used. In particular, in the case of a see-through type HMD, the camera image Ica is not displayed, and only the additional information can be superimposed and displayed in the field of view. When the display device 2 is a display, the additional information input from the additional information DB 4 to the camera image Ica is superimposed and displayed at the position corrected by the camera posture input from the camera posture estimation device.

  In the camera posture estimation device 3, as will be described in detail later, when the camera image Ica obtained by photographing the observation object M is input, the camera viewpoint estimation unit 3 a receives the two-dimensional feature points detected from the camera image Ica. Correspondence between coordinates and 3D coordinates of observation object M and its likelihood (3D coordinate correspondence), and correspondence between the feature point and viewpoint and its likelihood (viewpoint correspondence) are estimated to improve accuracy. Output to 3b. Based on the viewpoint estimation result, the high-accuracy unit 3b eliminates the three-dimensional coordinates of the feature points that could not be obtained from the observation target at the estimated viewpoint from the corresponding point candidates, thereby supporting the 3D coordinates. Increase the accuracy of the relationship.

  The camera posture calculation unit 3c estimates the posture of the AR system 1 with respect to the observation target M based on the highly accurate correspondence relationship (2D-3D correspondence relationship) and the internal and external parameters of the camera.

  Conventionally, methods for estimating the camera pose (external parameters of the camera) that explain the relationship from the match between the 3D coordinate and the 2D coordinate have been studied. The relationship between the 3D coordinate and the 2D coordinate is generally the following: It is expressed by equation (1).

      [u, v, 1] ^ T = sAW [X, Y, Z, 1] ^ T… (1)

  Here, [u, v] and [X, Y, Z] represent a two-dimensional pixel coordinate value and a 3D coordinate value, respectively, and [•] ^ T represents a transposed matrix. A and W represent an internal parameter and an external parameter (camera posture) of the camera, respectively. The internal parameters of the camera are obtained in advance by camera calibration. Camera posture W = [R, t] = [r1, r2, r3, t], which is represented by a rotation matrix R and a translation vector t. The camera pose W can be estimated using the match between the 3D coordinates [X, Y, Z, 1] ^ T and the 2D coordinates [u, v, 1] ^ T and the internal parameters of the camera.

  In FIG. 1, the globe is used as an example of the observation target M, but a three-dimensional object having a primitive structure such as a rectangular parallelepiped shape, a cylindrical shape, or a spherical shape, or an object having a more complicated three-dimensional structure is to be observed. Even in this case, if the 3D model is given, a similar AR system can be constructed. According to such an AR system, it is possible to realize information posting intuitive to the user by performing superimposed display according to the observation target. In the example of the globe, the surface of the earth is superimposed so as to display the altitude visually, the past continental shape is superimposed, the information updated when there is a change in the border or country name is superimposed, and combined with gesture recognition A usage example in which the name of the country pointed to is displayed is assumed.

  FIG. 2 is a diagram schematically showing the function of the camera viewpoint estimator 3a. A feature point is detected from a camera image Ica obtained by photographing the observation object M, and a local feature amount extracted from each feature point is classified into a classifier. To adapt. In this classifier, the relationship (3D coordinate probability distribution) between the local feature quantities of many feature points obtained by observing the 3D model of the observation target M from many viewpoints and its 3D coordinates is used for coordinate matching. In addition to learning in advance, the relationship between the local feature amount and the viewpoint (viewpoint probability distribution) is also learned in advance for viewpoint classification.

  In this embodiment, as will be described in detail later, a rule for coordinate matching and a rule for viewpoint classification are made common, and by applying a local feature amount to this common rule, a feature can be obtained only once. The 3D coordinates and viewpoint corresponding to the local feature of the point can be output together with its likelihood. As regards the viewpoint, as will be described in detail later, a polyhedron U that can accommodate the observation object M in the center is virtualized, and its vertex is defined as a viewpoint candidate.

[First embodiment]
FIG. 3 is a functional block diagram of the first embodiment of the camera viewpoint estimation unit 3a, and FIG. 4 is a diagram schematically representing the operation thereof. The feature point detector 31 detects a large number of feature points from the camera image Ica output from the camera 1 using the feature point detector 31a, and outputs each feature point and local feature information in the vicinity thereof to the matching unit 32.

  Any kind of feature point detector 31a can be used as long as it can identify two-dimensional coordinates having features, such as a Harris corner detector, a Hessian key point detector, or a FAST corner detector. In the present embodiment, the FAST corner detector employed in the method of Patent Document 2 is used.

  The local feature information is information for identifying a feature point such as a SIFT descriptor or a SURF descriptor, and any kind of general feature information can be used. In the present embodiment, the patch image adopted in Patent Document 2 is used as local feature information.

  Here, the patch image is an image cut out from the original image at a specific size, and in this embodiment, an arbitrary width and height (for example, a width of 32 pixels and a height of 32 pixels) centering on the feature point. It is the image of. When a feature point also has size (scale) information, the width and height of the patch image can be changed according to the size. The patch image is acquired at a very high speed because it is only necessary to cut out the image as compared with general local feature information.

  In the matching unit 32, in the 3D coordinate DB 32e, patch images of a large number of feature points detected from each of a large number of images obtained by observing the 3D model of the observation target M from various viewpoints are displayed as 3D coordinates. It is stored in advance together with a label representing and a label representing the viewpoint. In the present embodiment, the matching unit 32 has a learning function and an identification function, and the correspondence between the patch image of each feature point detected from the camera image Ica and the 3D coordinates and viewpoint of the observation object M is largely divided into the following two stages. Done in

(1) In the learning stage, the learning unit 32c functions as a lead, and among the many feature points stored in the 3D coordinate DB 32e, feature points that are easily detected as feature points in the camera image are selected. Selection is performed in advance by the selection unit 32f, and each label of the 3D coordinates and the viewpoint is attached to the selected patch image and applied to the classifier 32a.
The classifier 32a aggregates the correspondence between the classification result of each patch image and its 3D coordinate label and viewpoint label, thereby calculating the probability distribution between each patch image (feature point) and its 3D coordinate, and each patch image. And the probability distribution between the viewpoint and the viewpoint are obtained and registered in the probability distribution DB 32b in advance as a learning model (3D coordinate probability distribution and viewpoint probability distribution).

  (2) In the identification stage, the identification unit 32d functions predominantly, and after the learning model is registered, the patch image of the feature point detected from the camera image Ica obtained by photographing the observation target M is applied to the classifier 32a. Based on the classification result and the probability distribution learned in the probability distribution DB 32b, the correspondence X1 between each feature point of the camera image Ica and the 3D coordinate of the observation object M and its likelihood X2, and the camera image Ica The correspondence relationship Y1 between each feature point and the viewpoint and its likelihood Y2 are identified. The viewpoint determination unit 33 selects at least one viewpoint having a high likelihood as a viewpoint candidate of the camera image Ica based on the correspondence Y1 between each feature point (patch image) of the camera image Ica and the viewpoint and its likelihood Y2. decide.

  As such a matching unit 32, a Ferns classifier capable of classifying patch images can be used. As shown in FIG. 4, the Ferns classifier is composed of a plurality of decision trees (Fern), and each Fern has a multistage branch point (node) and a terminal end (leaf) having a decision rule for branching a patch image. ). The decision rule is to branch left and right depending on the magnitude relationship of the luminance of two pixels randomly selected from the patch image. The latter node further branches the patch image by another decision rule, but the nodes having the same number of steps have the same decision rule. Therefore, the node type is equal to the number of nodes. The node (leaf node) where the patch image finally arrives becomes the patch image classification result.

  Each leaf node holds a probability distribution of a class corresponding to a patch image acquired by learning. Since the Ferns classifier has multiple Ferns, the patch image is input to each Fern, the probability distribution corresponding to each reached leaf node is obtained, and the probability is multiplied using the naive Bayes classifier. The probability distribution of the class corresponding to the final patch image is determined.

  Each leaf node holds a discrete probability distribution corresponding to each class that is a classification candidate, and the total number of sections is the total number of classes that are classification candidates. Learning of the probability distribution is performed by classifying patch images to which teacher labels (correct answer classes) are assigned in advance. In this embodiment, the 3D model of the observation target M is used in the same manner as in Patent Document 2.

  As described above, the classification result in the present embodiment is information indicating which leaf each patch image with the 3D coordinate label and the viewpoint label has reached, and a probability distribution is obtained for each 3D coordinate label and the viewpoint label. .

  In the present embodiment, the correspondence between the patch image of the feature point and the 3D coordinate and its likelihood [X1, X2], and the correspondence between the patch image and the viewpoint and its likelihood [Y1, Y2] The correspondence relationship is learned in advance, and the patch image cut out from each feature point of the camera image Ica is applied to the matching unit 32 only once, so that the 3D coordinates and the viewpoint corresponding to the patch image of each feature point can be obtained. There is a feature in the point.

  Next, the learning process will be described with reference to the flowchart of FIG. 5 and the schematic diagram of FIG. Here, it is assumed that a 3D model in which the shape, pattern, color, size, and 3D coordinates of each part of the observation target M are faithfully expressed is constructed and prepared by a known method in advance.

  In addition, many feature points are detected from images obtained by observing the 3D model from various viewpoints, and the patch image cut out from each feature point has a 3D coordinate label and viewpoint that can uniquely identify the 3D coordinates. It is assumed that the viewpoint label that can be uniquely identified is associated with the viewpoint label and stored in advance in the database. In the present embodiment, the observation object M is photographed from about 10,000 viewpoints, and the 3D coordinates of each feature point and its patch image are stored for each viewpoint.

  Note that the 3D coordinates and viewpoints handled as classes need to be discrete, and if there are too many classes, the classification accuracy deteriorates. Therefore, it is necessary to limit the number of 3D coordinates and viewpoints to be learned to an appropriate number. Therefore, a fixed number of 3D coordinates is selected in the order in which the 3D coordinates are easily detected as feature points when captured as a camera image Ica. In this embodiment, as shown in an example in FIG. 7, all feature points detected from an image obtained for each viewpoint are classified by their 3D coordinates, and the top N vests with the highest number of detections, that is, the viewpoints, are classified. Feature points that are easy to detect are selected as teacher data.

Since the viewpoints need to be arranged at equal distances from the center of the 3D model and so as to be evenly spaced from each other, as shown in an example in FIG. 8, a polyhedron that can accommodate the 3D model in the center ( In this embodiment, an 80-hedron is hypothesized and its vertices (42 in this embodiment) are defined as viewpoint candidates.
Then, the approximately 10,000 viewpoints are classified into the same group in association with one of the nearest vertex candidates based on the focal position, and the same viewpoint label representing the viewpoint candidate is assigned to the viewpoint of the same group. Is attached. That is, in this embodiment, about 10,000 viewpoints are classified into the same group as any of the 42 viewpoint candidates, and the same viewpoint label is attached to the viewpoints of the same group. The focal position P of the camera can be calculated using an external parameter matrix W of the camera as shown in FIG.
In this embodiment, by using the vertices of the polyhedron, the viewpoint with respect to the observation target M is discretized at substantially equal intervals with respect to the direction, but if the viewpoint is discretized at approximately equal distances with respect to the distance, FIG. As shown in FIG. 5, a plurality of polyhedrons having different sizes may be arranged in a hierarchical manner, and each viewpoint may be classified as a vertex of one of the nearest polyhedrons.
Such discretization with respect to distance is performed by arranging three 80-hedrons whose distance from the center of the 3D model is increased stepwise, for example, as shown in FIG. 14, and 126 of these vertices (42 × 3). This is realized by using as a viewpoint candidate.

  Referring to FIG. 5, in step S1, one of the unselected viewpoint candidates is selected from the 42 viewpoint candidates as the current viewpoint viewpoint candidate. In step S2, one of the unselected viewpoints belonging to the target viewpoint candidate group is selected as the current target viewpoint. In step S3, one of the unselected feature points detected from the current viewpoint image is selected as the current feature point.

  In step S4, as shown in FIG. 6, the patch image cut out from the current feature point is applied to the classifier 32a together with the viewpoint label and the 3D coordinate label, and the classification result is voted on any leaf. The In step S5, it is determined whether or not voting has been completed for patch images of all detected feature points with respect to the current viewpoint of interest. If not completed, the process returns to step S3, an unselected feature point is selected again, and the above processes are repeated.

  In step S <b> 6, it is determined whether or not voting has been completed for the patch image cut out from images of all viewpoints belonging to the same group for the current viewpoint of interest. If not completed, the process returns to step S2, the unselected viewpoint is selected again, and the above processes are repeated. In step S7, it is determined whether or not each of the above processes has been completed for all viewpoint candidates. If not completed, the process returns to step S1 to reselect an unselected viewpoint candidate, and the above-described processes are repeated.

  As described above, when voting of 3D coordinate labels and viewpoint labels based on patch images of all feature points of all viewpoints belonging to each viewpoint candidate group is completed for each viewpoint candidate, each voting result becomes “3D “Coordinate probability distribution” and “viewpoint probability distribution” are registered in the probability distribution DB 32b.

  Next, the process of the identification stage will be described with reference to the flowchart of FIG. 10 and the schematic diagram of FIG. In step S31, all feature points are detected from the input camera image Ica, and their 2D coordinates are determined. In step S32, a patch image is cut out from each feature point.

  In step S33, the patch image of each feature point is sequentially applied to each Fern (Fern1 to FernN) of the classifier 32a, and node branching proceeds for each Fern, and the destination leaf is determined. In step S34, as shown in FIG. 11, the probability distributions registered in each destination leaf are aggregated, and the logarithmic sum of 3D coordinates and viewpoint probabilities (likelihood) is calculated for each feature point.

  In step S35, the identification unit 32d identifies the 3D coordinate and viewpoint corresponding to the current feature point based on the class ID whose logarithmic sum indicates the maximum value. In step S36, it is determined whether or not identification of all feature points detected from the camera image Ica has been completed. If not completed, the process returns to step S33, and the same identification procedure is repeated for all remaining feature points.

  As described above, when the correspondence between the patch image of each feature point detected from the camera image Ica of the observation target M and the 3D coordinate of the observation target M and the correspondence between each patch image and the viewpoint are acquired. The viewpoint determination unit 33 comprehensively evaluates them and determines a final viewpoint estimation value.

  In the present embodiment, the probability distribution of each viewpoint linked to each correspondence relationship is set as a comprehensive viewpoint probability distribution by calculating multiplication or logarithmic sum using a naive Bayes classifier. Here, it is possible to obtain a comprehensive probability distribution of viewpoints by averaging the probability distribution of each viewpoint. Then, based on this probability distribution (probability of each viewpoint with respect to the object in the camera image Ica), one viewpoint with the highest likelihood or a plurality of viewpoints with the highest N highest likelihoods are identified, and this is determined as the viewpoint determination result. Output as.

  Returning to FIG. 1, the high-accuracy unit 3b leaves only the 3D coordinate correspondences corresponding to the top N viewpoints with the determined 1 to likelihoods, and deletes the other 3D coordinate correspondences. Improve the coordinate correspondence. That is, the 3D coordinate correspondence includes three-dimensional coordinates of feature points detected from images obtained by observing the observation object M from various viewpoints in advance as corresponding point candidates. In the learning stage before the article, the three-dimensional coordinates of the feature points not obtained when observing the observation object M from the determined viewpoint are excluded from the corresponding point candidates. Thus, in the prior learning stage, for example, a feature point that is not possible as a corresponding point this time, such as a feature point obtained when observing the observation target M from a viewpoint opposite to the determined viewpoint, for example, The original coordinates are excluded from the corresponding point candidates. The camera posture calculation unit 3c estimates a camera posture based on the highly accurate 3D coordinate correspondence.

  According to the present embodiment, in the learning stage, the patch model cut out from each feature point of the observation target M is applied to the common rule of the classifier 32a together with the 3D coordinate label and the viewpoint label, and the learning model is simply applied once. Can be built.

  In the recognition stage, the patch image of each feature point detected from the camera image Ica of the observation object M is applied once to the common rule of the classifier, and each patch image and the 3D coordinates of the observation object M are And the correspondence between each patch image and the viewpoint can be acquired. Therefore, the 3D coordinates of the observation target and the 2D coordinates of the camera image can be associated with each other at high speed and with high accuracy without increasing the calculation cost.

[Second Embodiment]
FIG. 12 is a functional block diagram of the second embodiment of the camera viewpoint estimation unit 3a, and FIG. 13 is a diagram schematically representing the operation of this embodiment. In any of the drawings, the same reference numerals as those described above represent the same or equivalent parts, and thus the description thereof is omitted.

  In the present embodiment, each feature point output from the matching unit 32 and its 3D coordinates for each feature point (patch image) output from the matching unit 32 and the corresponding relationship [Y1, Y2]. There is a feature in that a threshold processing unit 34 that performs threshold processing based on the correspondence relationship [X1, X2] is provided.

  The threshold processing unit 34 determines a correspondence relationship in which the likelihood X2 is less than a predetermined threshold, that is, the likelihood from the correspondence relationship [X1, X2] between the patch image of each feature point detected from the camera image Ica and its 3D coordinates. Are selected, and the viewpoint correspondence [Y1, Y2] corresponding to the feature point is deleted. That is, the viewpoint correspondence [Y1, Y2] obtained as the classification result of the patch images with low likelihood in the 3D coordinate correspondence [X1, X2] is deleted regardless of the likelihood Y2.

  According to the results of experiments by the inventors, when the correspondence between the corresponding point accuracy of the 3D coordinate correspondence [X1, X2] and the viewpoint correspondence [Y1, Y2] and each likelihood X2, Y2 is compared, 3D coordinate correspondence Regarding the relationship [X1, X2], the corresponding point accuracy and the likelihood X2 showed a high correlation, and it was confirmed that the corresponding point accuracy increases as the likelihood X2 increases. On the other hand, for the viewpoint correspondence [Y1, Y2], the correlation between the corresponding point accuracy and the likelihood X2 is low, and it was confirmed that the likelihood Y2 may not accurately represent the corresponding point accuracy. .

  Generally, the likelihood X2 of the 3D coordinate correspondence is different from the likelihood Y2 of the viewpoint correspondence, but when the likelihood X2 is low, the feature point is not detected as a class or selected from the background There is a high possibility that the feature point is not used for learning, such as a surface point. Therefore, in this embodiment, by performing threshold processing on the viewpoint correspondence based on the 3D coordinate correspondence, by omitting the correspondence that is estimated to have low corresponding point accuracy from the viewpoint correspondence [Y1, Y2], the probability Increased the percentage of feature points used to learn the distribution.

  Then, the viewpoint determination unit 33 uses the probability distribution of each viewpoint linked to the viewpoint correspondence after the threshold processing to obtain a comprehensive viewpoint probability distribution. Then, based on this probability distribution (probability of each viewpoint with respect to the object in the camera image Ica), one viewpoint with the highest likelihood or a plurality of viewpoints with the highest N highest likelihoods are identified, and this is determined as the viewpoint determination result. And The high accuracy unit 3b leaves only the 3D coordinate correspondences corresponding to the top N viewpoints with the determined 1 to 1 likelihoods, and deletes the other 3D coordinate correspondences to obtain the 3D coordinate correspondences with high accuracy. Turn into.

  According to the present embodiment, based on the 3D coordinate correspondence where the correlation between the corresponding point accuracy and the likelihood is high, the viewpoint correspondence relationship in which the correlation between the corresponding point accuracy and the likelihood may be low is thresholded. Since the viewpoint is determined based on the likelihood of the processed viewpoint correspondence, the viewpoint estimation accuracy can be improved.

[Third embodiment]
In each of the above embodiments, the classification by the matching unit 32 is performed only once at the time of learning as well as at the time of recognition. The coordinate probability distribution learning and the viewpoint learning may be performed separately. In this embodiment, the 3D coordinate probability distribution and the viewpoint probability distribution are separately learned, thereby enabling more accurate 3D coordinate estimation.

That is, in each of the above embodiments, when learning the 3D coordinate probability distribution and the viewpoint probability distribution, only the patch image related to 3D coordinates (feature points) that are easily detected as feature points is selected in advance by the selection unit 32f of the learning unit 32c, By applying this to the classifier as teacher data (first teacher data) with 3D coordinates and viewpoint teacher labels, the 3D coordinate probability distribution and viewpoint probability distribution were obtained all at once.
In contrast, in the present embodiment, only the 3D coordinate probability distribution is first obtained based on the first teacher data. Next, the viewpoint probability distribution is learned. Here, all patch images including patch images not selected by the selection unit 32f, that is, all patch images stored in the 3D coordinate DB 32e are used as teacher data (first data). The viewpoint probability distribution is learned by attaching a viewpoint label and applying it to the classifier.
In the recognition stage, some feature points may be detected from other than the 3D coordinates that are easily detected as the feature points, and the patch image may be applied to the matching unit 32. However, according to the above procedure, the teacher data used for learning of the viewpoint probability distribution can be brought close to that which can be input at the recognition stage, so that more accurate viewpoint estimation can be performed. As a result, the accuracy of the high accuracy unit 3b is improved, and more accurate 3D coordinate estimation is possible.
Furthermore, when the threshold value processing unit 34 described in the second embodiment is used at the recognition stage, more accurate viewpoint estimation can be performed by utilizing this at the learning stage.
When learning the viewpoint probability distribution, the patch image of the second teacher data is first applied to the matching unit 32, and the correspondence [X1, X2] between each patch image and 3D coordinates is obtained. Next, the threshold processing unit 34 is applied to this, and a patch image with a low likelihood is deleted from the second teacher data.
The second teacher data is corrected by the above processing, and the viewpoint probability distribution is learned using the corrected second teacher data. By applying the threshold processing unit 34 at the recognition stage, a patch image having a low likelihood of the correspondence [X1, X2] with the 3D coordinates is not actually used for viewpoint estimation.
In the present embodiment, by correcting the teacher data described above, the teacher data used for learning of the viewpoint probability distribution can be brought close to what can be input to the recognition stage, and more accurate viewpoint estimation can be performed. As a result, the accuracy of the high accuracy unit 3b is improved, and more accurate 3D coordinate estimation is possible.

  In each of the above embodiments, the camera viewpoint estimation unit 3a of the AR system 1 learns the probability distribution by itself based on the patch image of the feature point detected from the projection image of the three-dimensional model simulating the observation target M. Although it has been described that a model is constructed and registered in advance in the probability distribution DB 32b, the present invention is not limited to this.

  That is, a dedicated system capable of constructing a learning model in the same or similar method as the learning unit 32c is prepared separately, and a probability distribution is established in advance by the same method and procedure in the dedicated system. You may make it register and use for a database. In this way, if a learning model is separately constructed and registered in the database of the AR system 1, a learning operation on the user side becomes unnecessary.

  DESCRIPTION OF SYMBOLS 1 ... Imaging device (camera), 2 ... Display apparatus, 3 ... Camera attitude estimation apparatus, 3a ... Camera viewpoint estimation part, 3b ... High precision part, 3c ... Camera attitude calculation part, 4 ... Additional information database (DB), 31 ... Feature point detector, 31a ... Feature point detector, 32 ... Matching unit, 32a ... Classifier, 32b ... Probability distribution DB, 32c ... Learning unit, 32d ... Identifier, 32e ... 3D coordinate DB, 33 ... Determination of viewpoint Part, 34... Threshold processing part

Claims (10)

An image that associates the three-dimensional coordinates of each first feature point detected from each image obtained by observing the observation target from a plurality of viewpoints and each second feature point detected from the camera image obtained by photographing the observation target. In the processing device,
A classifying means for classifying the local features according to a predetermined classification rule;
A database of probability distributions that give respective correspondence relationships of viewpoints with respect to observation targets in three-dimensional coordinates and camera images with respect to the classification results of the local feature amounts;
Identification means for identifying the correspondence between the second feature point, the three-dimensional coordinates, and the viewpoint based on the classification result obtained by applying the local feature amount of the second feature point to the classification means and the probability distribution;
An image processing apparatus comprising: means for improving the correspondence between the second feature point and the three-dimensional coordinates based on the correspondence between the second feature point and the viewpoint. Means for determining at least one viewpoint for the observation target in the camera image based on the correspondence between the second feature point and the viewpoint;
The image processing apparatus according to claim 1, wherein the high-accuracy means excludes three-dimensional coordinates of a first feature point that cannot be obtained from the observation target at the determined viewpoint from corresponding point candidates. .   The means for improving the accuracy can be detected from the image of the determined viewpoint when the three-dimensional coordinates of the first feature points are detected from the images obtained by observing the observation target from a plurality of viewpoints. The image processing apparatus according to claim 2, wherein the three-dimensional coordinates of the first feature point that did not exist are excluded from the corresponding point candidates. Means for thresholding the correspondence between the second feature point and the viewpoint based on each likelihood in the correspondence between the second feature point and the three-dimensional coordinate;
The means for determining the viewpoint determines the viewpoint for the observation target in the camera image based on the correspondence between the second feature point after the threshold processing and the viewpoint. An image processing apparatus according to claim 1.   2. The learning apparatus according to claim 1, further comprising learning means for learning each probability distribution by applying a local feature amount of each first feature point with a three-dimensional coordinate label and a viewpoint label to the classifier. 5. The image processing device according to any one of 4.   Each first feature point detected from each image obtained by observing the observation object from a plurality of viewpoints is less than the number of viewpoints based on the viewpoint position, and is approximately equidistant from the observation object. The first viewpoint is classified into the nearest one among a plurality of viewpoint candidates discretized at intervals, and the same viewpoint label is attached to the first feature point classified as the same viewpoint candidate. 5. The image processing apparatus according to 5. Each image in which the first feature point is detected is obtained by observing the observation target from a plurality of directions and distances,
Each of the first feature points is classified into the closest viewpoint from among a plurality of viewpoint candidates discretized at equal intervals including the distance from the observation target based on the viewpoint position, The image processing apparatus according to claim 5, wherein the classified first feature point is assigned the same viewpoint label. Means for selecting three-dimensional coordinates that are easy to detect as feature points in a camera image from the first feature points;
The learning means learns a probability distribution of the three-dimensional coordinates based on local feature quantities of feature points corresponding to the three-dimensional coordinates that are easily detected. Image processing device. Means for selecting three-dimensional coordinates that are easy to detect as feature points in a camera image from the first feature points;
The learning means learns each probability distribution of the three-dimensional coordinate and the viewpoint based on a local feature amount of a feature point corresponding to the three-dimensional coordinate that is easily detected. An image processing apparatus according to 1.   The image processing apparatus according to claim 1, wherein the local feature information is a patch image cut out from each feature point and its vicinity.