JP2006065399A - Image processing device and method, storage medium and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To surely recognize an object irrespective of the expansion and shrinkage of an image. <P>SOLUTION: A multi-resolution image is generated from a model image by a multi-resolution generation part 21, and characteristic quantities of characteristic points of respective resolutions are extracted by a characteristic quantity extraction part 21 registered in a model dictionary registration part 24. The multi-resolution of the inputted object image is generated by a multi-resolution generation part 31, and in a characteristic quantity comparison part 35, the characteristic point and the characteristic quantity are compared with the characteristic quantity registered in the model dictionary registration part 24. This comparison is carried out by using a kd tree constructed by a kd tree construction part 34. A model attitude estimation part 35 estimates an attitude of the model image included in the object image based on the comparison result of the characteristic quantity by the characteristic quantity comparison part 35 and outputs an attitude parameter of the object. This invention can be applied to a robot. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly to an image processing apparatus that is resistant to changes in viewpoint and brightness and can reliably recognize an object regardless of enlargement or reduction of the image. And a method, a recording medium, and a program.

  For example, many of the target object recognition techniques that have been put into practical use for recognizing a target object by a robot use a residual sequential test method or a template matching method using a cross-correlation coefficient. However, the template matching method is effective in special cases where it can be assumed that the detection target object appears in the input image without deformation, but it is not effective in an object recognition environment from a general image with a fixed viewpoint or illumination state. .

  On the other hand, a shape matching method that matches the shape feature of the target object with the shape feature of each region in the input image cut out by the image segmentation method has also been proposed, but in the general object recognition environment as described above In this case, the result of region segmentation is not stable, and it is difficult to describe a good shape of an object in the input image. In particular, when the detection target object is partially hidden by another object, recognition becomes very difficult.

  For the matching method using the whole features of the whole input image or area as described above, characteristic points and edges are extracted from the image, and the spatial positional relationship of the line segment set and edge set that constitute them A method has also been proposed in which is expressed as a line figure or graph, and matching is performed based on the structural similarity between line figures or between graphs. However, these techniques work well for certain specialized objects, but due to the deformation of the image, sometimes a stable structure between feature points is not extracted, especially as described above partially. Recognition of hidden objects becomes difficult.

  Therefore, a matching method has been proposed in which a characteristic point (feature point) is extracted from an image and a feature amount obtained from image information of the feature point and its local neighborhood is used. In such a matching method using local feature amounts that are invariant to partial image deformation of feature points, even when the detection target is partially hidden compared to the above-described method, even in the case of image deformation. Stable detection is possible. As a method of extracting feature points that are invariant to scaling conversion, a scale space of the image is constructed, and the scale direction of the local maximum points and local minimum points of the Difference of Gaussian (DoG) filter output of each scale image A method of extracting a point whose position does not change due to a change in scale as a scale feature point (Non-Patent Document 1 or Non-Patent Document 2) or a scale space of an image is constructed and extracted from each scale image by a Harris corner detector A method of extracting a point giving a local maximum of a Laplacian of Gaussian (LoG) filter output of a scale space image from corner points as a feature point has been proposed (Non-Patent Document 3).

Furthermore, it is preferable to select a feature quantity that is invariant to the line-of-sight change in the feature points extracted in this way. For example, Schmid & Mohr has proposed a matching method using a corner detected using a Harris corner detector as a feature point and using a rotation-invariant feature near the feature point (Non-Patent Document 4).
D. Lowe, “Object recognition from local scale-invariant features,” in Proc. International Conference on Computer Vision, Vol. 2, pp. 1150-1157, September 20-25, 1999, Corfu, Greece. D. Lowe, “Distinctive image features from scale-invariant keypoints,” accepted for publication in the International Journal of Computer Vision, 2004. K. Mikolajczyk, C. Schmid, “Indexing based on scale invariant interest points,” International Conference on Computer Vision, 525-531, July 2001. K. Mikolajczyk, C. Schmid, “Indexing based on scale invariant interest points,” International Conference on Computer Vision, 525-531, July 2001. Schmid, C., and R. Mohr, “Local grayvalue invariants for image retrieval,” IEEE PAMI, 19, 5, 1997, pp. 530-534. Schmid, C., and R. Mohr, “Local grayvalue invariants for image retrieval,” IEEE PAMI, 19, 5, 1997, pp. 530-534.

  However, since the feature amount of the corner does not have invariance to the enlargement / reduction conversion of the image, there is a problem that accurate recognition is difficult in the case of the enlargement / reduction conversion.

  The present invention has been made in view of such a situation, and even when an image is enlarged or reduced, the influence of a change in viewpoint or a change in brightness can be reduced, and an object can be reliably recognized. To do.

  The image processing apparatus according to claim 1, in the image processing apparatus that recognizes a pre-registered model image from the input image, reduces the resolution of the input image at a predetermined rate, thereby Multi-resolution image generation means for generating a multi-resolution image composed of images of different resolutions, feature point extraction means for extracting feature points for each resolution image of the multi-resolution image, and at least two local points in the feature points A feature quantity extracting means for extracting a typical feature quantity, comparing the feature quantity of the input image with the feature quantity of the model image, and obtaining a candidate corresponding feature point set as a set of feature points having similar feature quantities Comparing means to be generated and posture estimation means for estimating the posture of the input image based on the candidate corresponding feature point set.

  The feature quantity extraction means can extract a density gradient direction histogram in the vicinity of a feature point as a first type feature quantity and extract a dimensional degenerate density gradient vector as a second type feature quantity. .

  The comparison means searches for k Nearest Neighbor (k-NN) based on the feature amount of the input image for the feature amount group of the model image, which is a kd tree for each type, so that a candidate corresponding feature point set is obtained. Can be generated.

  The comparison means may make a feature point having a feature amount extracted in common for each type as a candidate corresponding feature point set.

  The candidate corresponding feature point set is further voted on a parameter space defined by an image conversion parameter that determines the position and orientation of the model image, and further includes a narrowing means for narrowing down by comparing the maximum number of votes with a threshold value, and an attitude estimating means Can estimate the posture of the input image based on the narrowed candidate corresponding feature point set.

  The posture estimation means projects an image conversion parameter for determining the position and posture of the model image determined by the N candidate corresponding feature point sets selected at random to the parameter space, and among the clusters formed on the parameter space, A cluster having the largest number of members can be obtained, and image conversion parameters for determining the position and orientation of the model image obtained from the members by the least square method can be output as a recognition result for recognizing the model image.

  The posture estimation means can detect a centroid of a cluster having the largest number of members and output an image conversion parameter for determining a position and posture of a model image composed of centroids as a recognition result for recognizing the model image. .

  The multi-resolution image generation means can generate a multi-resolution image with coarser accuracy than in the case of learning.

  An image processing method according to claim 9 is an image processing method of an image processing apparatus for recognizing a model image registered in advance from an input image, and reduces the resolution of the input image at a predetermined rate. A multi-resolution image generation step of generating a multi-resolution image composed of a plurality of different resolution images, a feature point extraction step of extracting a feature point for each resolution image of the multi-resolution image, and A feature amount extraction step for extracting at least two local feature amounts, and comparing the feature amount of the input image with the feature amount of the model image, and corresponding candidates as a set of feature points having similar feature amounts A comparison step for generating a feature point set, and a posture estimation step for estimating the posture of the input image based on the candidate corresponding feature point set. It is characterized in.

  The recording medium program according to claim 10 is a program for an image processing apparatus that recognizes a model image registered in advance from an input image, and reduces the resolution of the input image at a predetermined rate. A multi-resolution image generation step for generating a multi-resolution image composed of a plurality of different resolution images, a feature point extraction step for extracting a feature point for each resolution image of the multi-resolution image, and the feature point A feature amount extracting step for extracting at least two local feature amounts in the image, comparing the feature amounts of the input image with the feature amounts of the model image, and candidates as sets of feature points having similar feature amounts A comparison step for generating a corresponding feature point set, and a posture estimation step for estimating the posture of the input image based on the candidate corresponding feature point set. Characterized in that it comprises and.

  A program according to an eleventh aspect is a program for an image processing apparatus that recognizes a model image registered in advance from an input image, and reduces the resolution of the input image at a predetermined rate. A multi-resolution image generating step for generating a multi-resolution image composed of a plurality of different resolution images, a feature point extracting step for extracting a feature point for each resolution image of the multi-resolution image, and at least two of the feature points A feature quantity extracting step for extracting two local feature quantities; comparing the feature quantities of the input image with the feature quantities of the model image; and candidate corresponding feature points as a set of feature points having similar feature quantities A comparison step for generating a set and a posture estimation step for estimating the posture of the input image based on the candidate corresponding feature point set. Characterized in that to execute the Yuta.

  The image processing device according to claim 12 is an image processing device that learns a model image for comparison with an image to be recognized, by reducing the resolution of the model image at a predetermined rate, Multi-resolution image generation means for generating a multi-resolution image composed of a plurality of different resolution images with finer precision than in the case of recognition, and feature point extraction means for extracting feature points for each resolution image of the multi-resolution image And feature amount extraction means for extracting at least two local feature amounts at the feature points, and registration means for registering the feature amounts of the model image.

  The feature quantity extraction means can extract a density gradient direction histogram in the vicinity of a feature point as a first type feature quantity and extract a dimensional degenerate density gradient vector as a second type feature quantity. .

  The registration unit may register the feature amount group of the model image as a kd tree for each type.

  The image processing method according to claim 15 is an image processing method of an image processing apparatus that learns a model image for comparison with an image to be recognized, and the resolution of the model image is reduced at a predetermined rate. By doing so, a multi-resolution image generation step for generating a multi-resolution image composed of a plurality of different resolution images with finer precision than in the case of recognition, and feature points are extracted for each of the multi-resolution image resolution images. The method includes a feature point extracting step, a feature amount extracting step for extracting at least two local feature amounts in the feature point, and a registration step for registering the feature amount of the model image.

  The program of the recording medium according to claim 16 is a program of an image processing apparatus that learns a model image for comparison with an image to be recognized, and the resolution of the model image is set at a predetermined ratio. By reducing, a multi-resolution image generation step for generating a multi-resolution image composed of a plurality of different resolution images with a finer accuracy than in the case of recognition, and extracting feature points for each resolution image of the multi-resolution image A feature point extracting step, a feature amount extracting step for extracting at least two local feature amounts in the feature point, and a registration step for registering the feature amount of the model image.

  The program according to claim 17 is a program for an image processing apparatus that learns a model image for comparison with an image to be recognized, and reduces the resolution of the model image at a predetermined rate. A multi-resolution image generation step for generating a multi-resolution image composed of a plurality of different resolution images with a finer accuracy than in the case of recognition, and a feature point for extracting a feature point for each resolution image of the multi-resolution image The computer is caused to execute an extraction step, a feature amount extraction step for extracting at least two local feature amounts at the feature point, and a registration step for registering the feature amount of the model image.

  In the present invention, a multi-resolution image is generated from the input image, and feature points are extracted for each resolution image of the multi-resolution image. At least two local feature values at the feature points are extracted, the feature values are compared with the feature values of the model image, and a candidate corresponding feature point set as a set of feature points having similar feature values is generated. Based on this, the posture of the input image is estimated.

  In the present invention, a multi-resolution image is generated with a finer accuracy than in the case of recognition. Feature points are extracted for each resolution image of the multi-resolution image, and at least two local feature amounts at the feature points are extracted and registered.

  According to the present invention, an object can be recognized. In particular, according to the present invention, even when the image is enlarged or reduced, regardless of the change in the image due to the viewpoint change or brightness change, and even when the object is partially hidden, An object can be reliably recognized.

  Further, according to the present invention, it is possible to register a feature quantity that can recognize an object. In particular, according to the present invention, even when the image of the recognition target object is enlarged or reduced with respect to the registered image, the image change due to the posture change can be performed regardless of the image change caused by the viewpoint change or the brightness change. Feature points and feature quantities that can be recognized robustly can be extracted and registered.

  BEST MODE FOR CARRYING OUT THE INVENTION The best mode of the present invention will be described below. The correspondence relationship between the disclosed invention and the embodiments is exemplified as follows. Although there is an embodiment which is described in the specification but is not described here as corresponding to the invention, it means that the embodiment corresponds to the invention. It doesn't mean not. Conversely, even if an embodiment is described herein as corresponding to an invention, that means that the embodiment does not correspond to an invention other than the invention. Absent.

  Further, this description does not mean all the inventions described in the specification. In other words, this description is for the invention described in the specification and not claimed in this application, i.e., for the invention that will be applied for in the future or that will appear as a result of amendment and added. It does not deny existence.

  The image processing apparatus according to claim 1 is an image processing apparatus that recognizes a model image registered in advance from an input image (for example, the image processing apparatus 1 having the recognition unit 12 in FIG. 1). Multi-resolution image generation means (for example, FIG. 13) that generates a multi-resolution image (for example, the multi-resolution image of FIG. 4) composed of a plurality of different resolution images by reducing the resolution at a predetermined rate. The multi-resolution generation unit 31 in FIG. 1 that executes the process of step S132, and feature point extraction means that extracts feature points for each resolution image of the multi-resolution image (for example, the process of step S136 in FIG. 13 is executed) 1 and the at least two local feature amounts of the feature points (for example, a direction histogram (type of density gradient in the vicinity of the feature points in FIG. 8) ), And the feature extraction unit (for example, steps S138 to S142 in FIG. 13, and steps S143 to S145 in FIG. 1 that performs the above-described processing, the feature amount of the input image is compared with the feature amount of the model image, and a candidate corresponding feature point set as a set of feature points having similar feature amounts Is used to estimate the posture of the input image based on the candidate corresponding feature point set and the comparison means (for example, the feature amount comparison unit 35 of FIG. 1 that executes the processing of steps S150 and S151 of FIG. 14). Posture estimation means (for example, the model posture estimation unit 36 in FIG. 1 that executes the processing in step S158 in FIG. 15).

  The feature amount extraction unit extracts a density gradient direction histogram in the vicinity of the feature point (for example, a density gradient direction histogram in the vicinity of the feature point in FIG. 8) as a first type feature amount. A dimensional degenerate density gradient vector (for example, the type 2 feature quantity that has been resampled by linear interpolation at the bottom in FIG. 11) is extracted as the feature quantity.

  The comparison means searches for k Neest Neighbor (k-NN) based on the feature amount of the input image for the feature amount group of the model image, which is a kd tree for each type, so that the candidate correspondence A feature point set is generated (for example, the processing of FIG. 17).

  The comparison means sets the feature points having the feature quantities extracted in common for each type as the candidate corresponding feature point set (for example, a pair of a rectangle and a circle in FIG. 17, a pair of a rectangle and a cross shape, However, squares, pentagons, triangles, circles, or crosses in the figure represent feature points).

  In the image processing apparatus, the candidate corresponding feature point set is defined by a parameter space (for example, image conversion parameters (scl, θ, dX, dY)) defined by image conversion parameters that determine the position and orientation of the model image. The narrowing down means (for example, the corresponding feature point pair narrowing unit 61 in FIG. 21 that executes the processing of steps S301 to S310 in FIG. 22) further narrows down by voting on the parameter space) and comparing the maximum number of votes with a threshold value. The posture estimation means estimates the posture of the input image based on the narrowed candidate corresponding feature point set.

  The posture estimation means includes image conversion parameters (for example, Euclidean transformation, similarity transformation, affine transformation, projective transformation) that determine the position and orientation of the model image determined by N candidate selection feature point sets selected at random. A cluster having the largest number of members is obtained from the clusters formed on the parameter space by projecting to the parameter space, and the position and orientation of the model image obtained by the least square method from the members (that is, the corresponding feature point set group) are obtained. The determined image conversion parameter is output as a recognition result for recognizing the model image (for example, processing in steps S201 to S206 in FIG. 19).

  The multi-resolution image generation unit generates the multi-resolution image with coarser accuracy than in the case of learning (for example, the process of step S132 in FIG. 13).

  An image processing method according to claim 9, a program for a recording medium according to claim 10, and a program according to claim 11 include an image processing apparatus that recognizes a model image registered in advance from an input image (for example, the recognition unit in FIG. 1). In the image processing method or program of the image processing apparatus 1) having 12, a multi-resolution image (for example, a plurality of different resolution images (for example, A multi-resolution image generation step (for example, step S132 of FIG. 13) for generating the multi-resolution image of FIG. 4 and a feature point extraction step (for example, FIG. 13 step S136) and at least two local feature quantities (for example, the direction of the density gradient in the vicinity of the feature point in FIG. 8) at the feature point. A feature amount extraction step (for example, steps S138 to S142 in FIG. 13 and steps S143 to S145 in FIG. 14) for extracting a strogram, the type 2 feature amount that has been resampled at the bottom of the linear interpolation in FIG. A comparison step (for example, steps S150 and S151 in FIG. 14) that compares the feature amount of the obtained image with the feature amount of the model image and generates a candidate corresponding feature point set as a set of feature points having similar feature amounts. And a posture estimation step (for example, step S158 in FIG. 15) for estimating the posture of the input image based on the candidate corresponding feature point set.

  The image processing apparatus according to claim 12 is an image processing apparatus that learns a model image to be compared with an image to be recognized (for example, the image processing apparatus 1 having the learning unit 11 in FIG. 1). Is generated at a predetermined rate, so that a multi-resolution image (for example, the multi-resolution image in FIG. 4) composed of a plurality of different resolutions is generated with finer accuracy than in the case of recognition. Resolution image generation means (for example, the multi-resolution generation unit 21 in FIG. 1 that executes the processing of step S12 in FIG. 2), and feature point extraction means (for example, the feature point extraction means for extracting the resolution points of the multi-resolution image) The feature point extraction unit 22 in FIG. 1 that executes the process of step S16 in FIG. 2 and at least two local feature amounts (for example, density in the vicinity of the feature point in FIG. 8) FIG. 11 is a diagram for executing feature amount extraction means (for example, step S19 in FIG. 2 and step S25 in FIG. 3) for extracting the gradient direction histogram and the type 2 feature amount that has been resampled at the bottom in FIG. 1 feature amount extraction unit 23) and registration means for registering the feature amount of the model image (for example, the model dictionary registration unit 24 in FIG. 1 that executes the process of step S29 in FIG. 3). And

  The feature amount extraction unit extracts a density gradient direction histogram in the vicinity of the feature point (for example, a density gradient direction histogram in the vicinity of the feature point in FIG. 8) as a first type feature amount. A dimensional degenerate density gradient vector (for example, the type 2 feature quantity that has been resampled by linear interpolation at the bottom in FIG. 11) is extracted as the feature quantity.

  The registration unit registers the feature amount of the model image as a kd tree for each type (for example, processing of the kd tree construction unit 34 in FIG. 1).

  An image processing method according to claim 15, a program for a recording medium according to claim 16, and a program according to claim 17 include an image processing device that learns a model image for comparison with an image to be recognized (for example, FIG. 1). In the image processing method or program of the image processing apparatus 1) having the learning unit 11, the resolution of the model image is reduced at a predetermined rate, thereby allowing a multi-resolution image (for example, an image having a plurality of different resolutions). , The multi-resolution image generation step (for example, step S12 in FIG. 2) for generating the fine-resolution image of the multi-resolution image with finer accuracy than in the case of recognition, and feature points for the respective resolution images of the multi-resolution image. A feature point extraction step (for example, step S16 in FIG. 2) to be extracted, and at least two local feature amounts (for example, the feature point) 8 is a feature amount extraction step for extracting the density gradient direction histogram in the vicinity of the feature point and the linear interpolation resampled type 2 feature amount at the bottom of FIG. 11 (for example, step S19 in FIG. 2, step in FIG. 3). S25) and a registration step (for example, step S29 in FIG. 3) for registering the feature quantity of the model image.

  Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. In this embodiment, the present invention compares an object image, which is an input image including a plurality of objects, with a model image (registered in advance) including a model to be detected, and extracts a model from the object image. This is applied to an image processing apparatus.

  A schematic configuration of an image processing apparatus according to the present embodiment is shown in FIG. The image processing apparatus 1 includes two parts, a learning unit 11 that performs model learning processing and a recognition unit 12 that recognizes an object in an input image.

  The learning unit 11 includes a multi-resolution generation unit 21, a feature point extraction unit 22, a feature amount extraction unit 23, and a model dictionary registration unit 24.

  The multi-resolution generation unit 21 generates a multi-resolution image from the input model image. The feature point extraction unit 22 extracts feature points from each multi-resolution image generated by the multi-resolution generation unit 21. The feature amount extraction unit 23 extracts the feature amount of each feature point extracted by the feature point extraction unit 22. The model dictionary registration unit 24 registers the feature amount group of the model image extracted by the feature amount extraction unit 23.

  The recognition unit 12 includes a multi-resolution generation unit 31, a feature point extraction unit 32, a feature amount extraction unit 33, a kd tree construction unit 34, a feature amount comparison unit 35, and a model posture estimation unit 36.

  The multi-resolution generation unit 31 generates multi-resolution images from the input object image. The feature point extraction unit 32 extracts feature points from the multi-resolution image generated by the multi-resolution generation unit 31. The feature quantity extraction unit 33 extracts the feature quantity of each feature point extracted by the feature point extraction unit 32. The processing performed by the multi-resolution generation unit 31, the feature point extraction unit 32, and the feature amount extraction unit 33 is performed by the multi-resolution generation unit 21, the feature point extraction unit 22, and the feature amount extraction unit 23 in the learning unit 11. This is the same processing as that described.

  The kd tree construction unit 34 constructs a kd tree from the feature amounts registered in the model dictionary registration unit 24. The feature quantity comparison unit 35 performs the feature quantity extracted by the feature quantity extraction unit 33 and the entire model image (or model-by-model processing) that is a recognition target expressed as a kd tree constructed by the kd tree construction unit 34. Is compared with the feature quantity group of each model image). The model posture estimation unit 36 estimates the presence / absence of a model included in the object image and its posture (model posture) based on the comparison result by the feature amount comparison unit 35, and a parameter (object posture parameter) representing the posture of the model Is output.

  Note that both the learning unit 11 and the recognition unit 12 do not always have to exist simultaneously. As a result of learning in advance by the learning unit 11, the model dictionary registration unit 24 in which necessary information is registered may be mounted on the recognition unit 12 or may be used wirelessly.

  Next, the learning process in the learning unit 11 will be described with reference to the flowcharts of FIGS. 2 and 3.

The multi-resolution generation unit 21 repeats the processes in steps S11 to S27 until it is determined in step S28 described later that all models have been processed. Therefore, in step S11, the multi-resolution generation unit 21 selects one unprocessed model. In step S12, the multiresolution generation unit 21 generates a multiresolution group. Specifically, the multiresolution generation unit 21 reduces the input model image to be learned according to a predetermined magnification, and generates a multiresolution image group. For example, when the reduction ratio from the original image that is the lowest resolution image is α and the number of output multi-resolution images is N (including the original image), the kth (original image is k = 0) multiplexing The resolution image I ^[k] of the resolution is generated by linearly reducing the original image I ^[0] at a reduction rate α × (N−k).

Or as another method, the reduction ratio for generating a one-step lower image resolution and gamma (fixed value), i.e. with a reduction ratio gamma ^k of I ^[0], I ^[k by linear interpolation reduced ^] Is also possible.

FIG. 4 shows a multi-resolution image group generated when parameters N = 10 and α = 0.1. Figure in the example 4, the image was reduced at a reduction ratio 0.9 of the original image ^{I [0] I [1]} , the image I ^[2] which is reduced at a reduction ratio 0.8, ..., the image I was reduced at a reduction ratio 0.1 A total of 10 multi-resolution images of ^[9] are generated. As the coefficient k that defines the reduction ratio increases, the image is reduced to a smaller size. As a result, the image frame itself of each frame also decreases as the coefficient k increases.

Next, the feature point extraction unit 22 repeats the processing of steps S13 to S26 until it determines that all resolution images have been processed in step S27 described later, and each resolution image I ^{[ k]} (k = 0,..., N−1), feature points (scale invariant feature points) that are robustly extracted even if there is enlargement / reduction conversion (scale conversion) of the image are extracted. However, as a method of extracting scale invariant feature points, the scale space of the image is constructed, and the local maximum point (maximum point of the local predetermined range) and local minimum of the Difference of Gaussian (DoG) filter output of each scale image A method of extracting points whose position does not change even when the scale direction changes from among the points (local minimum points in a predetermined range) as scale feature points (D. Lowe, “Object recognition from local scal e-invariant features, ”in Proc. International Conference on Computer Vision, Vol. 2, pp. 1150-1157, September 20-25, 1999, Corfu, Greece.) Of corner points extracted by Harris corner detector from a point that gives local maximum of Laplacian of Gaussian (LoG) filter output of scale space image as feature points (K. Mikolajczyk, C. Schmid, “Indexing based on scale invariant interest points, ”International Conference on Computer Vision, 525-531, July 2001.). Any extraction method that can extract scale-invariant feature points can be applied to the feature point extraction unit 22.

  Here, as an embodiment of the invention, as a method for extracting scale invariant feature points, a method proposed by D. Lowe (“Distinctive image features from scale-invariant keypoints,” accepted for publication in the International Journal of Computer Vision, 2004.). In this method, scale space representation of scale-invariant feature point extraction target image (T. Lindeberg, “Scale-space: A framework for handling image structures at multiple scales.”, Journal of Applied Statistics, vol. 21, no. 2, pp. 224-270, 1994 "), local maximum points and local minimum points taking into account the scale direction are extracted as feature points from the DoG filter output of the image.

Therefore, in step S13, the feature point extraction unit 22 selects an unprocessed resolution image among the resolution images. In step S14, the feature point extraction unit 22 generates a resolution image of the scale space. That is, one resolution image of the scale invariant feature point extraction target image I (each resolution image generated by the multi-resolution generation unit 21 (resolution images of k = 0, 1, 2,..., 9)) A scale space of a feature point extraction target image is generated. The s-th (s = 0,..., S−1) resolution image L _s of the scale space is obtained by using the two-dimensional Gaussian function represented by the equation (1) for the target image I and σ = k ^s σ _0. Is generated by convolution integration (Gaussian filtering).

Here, σ ₀ is a parameter for determining the blur level for the purpose of noise removal of the target image I, k is a constant factor regarding the blur level common to each resolution of the scale space, and the resolution image I ^[k] This is a different factor from k. The horizontal direction of the image is the X axis and the vertical direction is the Y axis.

FIG. 5 shows an example of the scale space generated in this way. In this example, the resolution images L _{0 to} L ₄ generated by using the following five two-dimensional Gaussian functions for the image I are shown.

  Note that the term on the right side of the convolution integral symbol on the right side of Equations (2) to (6) represents the following equation. That is, it is substantially the same as the formula (1).

  In FIG. 5, the number of resolution levels S = 5.

Next, in step S15, the feature point extraction unit 22 calculates a DoG filter output image. In other words, a DoG filter output image of each resolution image L _{s in} the scale space to be feature point extraction obtained in this way is obtained. This DoG filter is a kind of second-order differential filter that is used for image contour enhancement. As an approximate model of processing that is performed from the retina to the outer knee-like body in the human visual system, Often used with LoG filters. The output of the DoG filter can be obtained efficiently by taking the difference between the two Gaussian filter output images. That is, as shown in the center column of FIG. 5, the DoG filter output image D _s of the s-th (s = 0,..., S-2) resolution is the resolution image L _s one level higher. It is obtained by subtracting from the resolution image L _{s + 1} of the layer (calculating L _{s + 1} −L _s ).

Next, in step S16, the feature point extraction unit 22 extracts scale invariant feature points. Specifically, among the pixels on the DoG filter output image D _s (s = 1,..., S-3), a region immediately adjacent to the DoG filter output image D _s (predetermined in the case of the present embodiment) 3 × 3 pixel area), the DoG filter output image D _s-1 one level lower than it, and the same position (corresponding position) on the DoG filter output image D _{s + 1} one level higher than that ) Pixels having a local maximum (maximum value of 27 pixels) and a local minimum (minimum value of 27 pixels) are extracted as scale-invariant feature points in 27 pixels in total in the immediate vicinity region of the feature point group. It is held as K _s (s = 1,..., S-3). The feature point group K _s is shown in the right column of FIG. Thus extracted feature point with respect to factor change in resolution k ² (i.e. scale change), a scale invariant feature points with invariant position.

The feature point extraction unit 22 repeats the processing of steps S13 to S16 until it determines that all resolution images have been processed in step S27 described later, and the multi-resolution level image I ^[k] generated by the multi-resolution generation unit 21 ^. A scale invariant feature point group is extracted for each of.

Next, the feature amount extraction unit 23 repeats the processing of steps S17 to S25 until it is determined that all the feature points have been processed in step S26, and the feature amount extraction unit 23 performs the processing for each feature point extracted from each multi-resolution level image I ^[k] . Extract features. Hereinafter, the feature amount at the feature point is referred to as a feature point feature amount or simply as a feature amount depending on the context.

  As the feature point feature amount, a feature amount that is invariant to image rotation conversion and brightness change is used. A plurality of feature amounts may be assigned to one feature point. In that case, the subsequent feature quantity comparison unit 35 needs to integrate the comparison results of different feature quantities. In the case of this embodiment, two feature quantities derived from density gradient information (density gradient strength and density gradient direction at each point) in the vicinity of the feature points of the image from which the feature points are extracted are used as the feature quantities. It is done. One is a direction histogram corrected in a dominant density gradient direction (hereinafter referred to as a canonical direction) in the region near the feature point, and the other is a low-dimensional degenerate corrected in the canonical direction. It is a density gradient vector.

The first feature amount (type 1 feature amount) is obtained by zero-correcting a histogram (direction histogram) regarding the density gradient direction in the vicinity of the feature point in the dominant direction. In order to extract the first feature amount, the feature amount extraction unit 23 selects one unprocessed feature point in step S17. In step S18, the feature amount extraction unit 23 obtains the density gradient strength M _{x, y} and the direction R _{x, y} . That is, as shown in FIG. 6, the density gradient intensity M _{x, y} near the feature point (in this embodiment, a pixel group that falls within a range of 7 pixels in diameter (radius 3.5 pixels) around the feature point P). , And the direction R _{x, y} are obtained by the equations (8) and (9), respectively. In the formula, x and y are coordinates on the image of the pixel for which the density gradient is obtained, and I _{x, y} is the pixel value.

Next, in step S19, the feature amount extraction unit 23 generates a direction histogram. Specifically, a direction histogram (in this embodiment, Δθ = 10 °) corresponding to a class width Δθ and a class number 360 ° / Δθ, based on the direction R _{x, y} of each pixel in the vicinity of the feature point. The frequency of each pixel is accumulated in the class. At this time, in order to reduce the influence of the quantization error of the class as shown in FIG. 7, the frequency (vertical axis in FIG. 7) is set to the direction R _{x, y} from the center value of the class (horizontal axis in FIG. 7). A value proportional to the proximity of the distance to is accumulated. That is, if the two classes closest to the direction R _{x, y} are g and g + 1 _, and the distance between each center value and the direction R _{x, y} is d ₁ and d ₂ , the numerical value to be added to the classes g and g + 1. Are d ₂ / (d ₁ + d ₂ ) and d ₁ / (d ₁ + d ₂ ), respectively. This reduces the quantization error.

  Next, in step S20, the feature amount extraction unit 23 normalizes the frequency. That is, the frequency of the obtained direction histogram is normalized by dividing by the number of pixels near the feature point (the number of pixels falling within the range of 7 pixels in diameter). In this way, by accumulating only the gradient direction, it is possible to obtain a feature quantity that is strong against changes in brightness.

  Further, the feature quantity extraction unit 23 extracts the canonical direction in step S21, and normalizes the angle in the canonical direction in step S22. Specifically, in order to obtain a feature quantity that is invariant to rotation transformation, a canonical direction is extracted as an angle that gives a strong peak in the obtained direction histogram, and the histogram is such that the angle as the canonical direction becomes 0 degrees. The angle is normalized by shifting. In the histogram relating to the feature points extracted near the corner, a plurality of strong peaks appear in the direction perpendicular to the edge. In such a case, the angle is corrected so that the angle is 0 degrees for each strong peak (normal) A directional histogram is generated. That is, as many features as the number of canonical directions are generated. The reference for the peak in the canonical direction is, for example, a peak direction that gives an accumulated value of 80% or more of the maximum accumulated value.

For example, in the directional histogram shown in FIG. 8, there are two peaks, a frequency V ₈₀ at an angle of ₈₀ degrees and a frequency V _{200 at} an angle of 200 degrees. That is, an angle of 80 degrees and an angle of 200 degrees are the canonical directions. In this case, as shown in FIG. 9, the histogram normalized so that the angle 80 degrees as the canonical direction becomes 0 degrees, and as shown in FIG. 10, the angle 200 degrees as the canonical direction is 0 degrees. A normalized histogram is generated so that

  The type 1 feature quantity obtained by the above processing is a feature vector having the same dimension as the number of classes in the direction histogram (in this embodiment, a 36 (= 360 ° / 10 °) dimension vector, that is, 36 classes. A vector of numbers representing frequencies).

  Next, a low-dimensional degenerate density gradient vector is obtained as the second feature amount (type 2 feature amount). The type 1 feature quantity ignores the spatial arrangement of the pixels in the vicinity of the feature point and focuses only on the direction tendency (frequency) of the density gradient vector in the local region near the feature point, whereas the type 2 feature quantity For the feature amount, attention is paid to the spatial arrangement of each density gradient vector in the vicinity of the feature point. By using these two types of feature amounts for feature amount comparison by a method to be described later, recognition that is strong against viewpoint change and brightness change is realized.

  In order to extract the type 2 feature quantity, first, in step S23, the feature quantity extraction unit 23 rotationally corrects the feature point neighborhood image. That is, the feature point vicinity image is rotationally corrected so that the canonical direction in the vicinity of the feature point obtained by the above-described processing is 0 degree. Further, in step S24, the feature amount extraction unit 23 calculates a density gradient vector group. For example, when the density gradient of the pixels near the feature points shown in the upper part of FIG. 11 is distributed as shown in FIG. 8, the canonical directions are directions of 80 degrees and 200 degrees as described above. It becomes. Therefore, as shown in the left diagram in the middle of FIG. 11, the feature point neighboring image is rotated clockwise in the case of this example so that the canonical direction of 80 degrees becomes 0 degrees. Then, the density gradient vector group is calculated. This is equivalent to obtaining the density gradient vector group of the direction histogram of FIG. 9 obtained by normalizing the canonical direction at an angle of 80 degrees of FIG. 8 as 0 degrees.

  Similarly, as shown on the right side of the middle stage of FIG. 11, the feature point neighborhood image is rotationally corrected so that the canonical direction of 200 degrees is 0 degrees. Then, the density gradient vector group of the image is calculated. This is equivalent to obtaining the density gradient vector group of the direction histogram of FIG. 10 obtained by normalizing the canonical direction of the angle of 200 degrees of FIG. 8 as 0 degree.

  Next, in step S25, the feature amount extraction unit 23 performs dimension reduction on the density gradient vector group. That is, in order to be able to absorb the deviation of the feature point extraction position of about several pixels, this density gradient vector group is, for example, a circle of 7 pixels in diameter as shown on the left and right in the lower part of FIG. Dimensional degeneration is performed by linearly re-sampling a vector group of 5 × 5 pixels in a rectangle almost inscribed inward into a 3 × 3 vector group.

  Specifically, as shown in FIG. 12, the linear interpolation resample is performed by calculating the pixel value of the resampled image by the ratio of the distance from four neighboring original image pixels according to the following equation. Is called.

f (X, Y) = (1-q). {(1-p) .f (x, y) + p.f (x + 1, y)}
+ Q · {(1−p) · f (x, y + 1) + p · f (x + 1, y + 1)}
... (10)

  In the above formula, (X, Y) is a pixel of the resampled image, and (x, y), (x + 1, y), (x, y + 1), (x + 1, y + 1) are near the resampled image (X, Y). Original image pixels, f (a, b) is the pixel value of coordinates (a, b), and p, q are the x coordinates from the neighboring pixels to the resampled image (X, Y) as shown in FIG. The distance ratio between the direction and the y-coordinate direction.

  Thus, by applying the x and y components of the dimension-reduced vector to each dimension of the feature vector, a type 2 feature amount can be obtained. When resampled into a 3 × 3 vector group by linear interpolation resample, the feature amount is 18 (= 3 × 3 × 2) dimensions.

  If the target image size after re-sampling is less than half of the original image size, the original image is reduced by 0.5 times, and when a minimum 0.5 times multiplier size image larger than the target size is obtained, By performing resampling of the equation (10) from the image, it is possible to reduce the error at the time of resampling. For example, when an image having a size 0.2 times that of the original image is created by linear interpolation resampling, linear interpolation resampling of Expression (10) is performed on a 0.25 times size image of the original image obtained by applying 0.5 times resampling twice. .

  In step S26, the feature quantity extraction unit 23 determines whether all feature points have been processed. If there is a feature point that has not yet been processed, the process returns to step S17, and the subsequent processing is repeatedly executed. To do. When it is determined in step S26 that all feature points have been processed (when the processing in steps S17 to S25 has been performed for all feature points), in step S27, the feature point extracting unit 22 displays the full resolution image. Is processed. If there is a resolution image that has not yet been processed, the processing returns to step S13, and the subsequent processing is repeatedly executed. When it is determined that the processing from step S13 to step S25 has been performed for all resolution images, in step S28, the multi-resolution generation unit 21 determines whether all models have been processed. If there is a model that has not yet been processed, the process returns to step S11, and the subsequent processes are repeatedly executed. If it is determined that the processes in steps S11 to S25 have been executed for all models, the process proceeds to step S29.

  In step S29, the model dictionary registration unit 24 labels and registers the feature point feature quantities extracted as described above. In this case, it is labeled so as to refer to the feature quantity of which feature point extracted from which scale of which image of which model's multi-resolution image group, and is registered in the model dictionary.

  As described above, in the model dictionary registration unit 24, a model image as a target object to be recognized is registered in advance as a feature amount.

  When the image processing apparatus 1 has both the learning unit 11 and the recognition unit 12, the recognition unit 12 can use the model dictionary registration unit 24 as it is. When the learning unit 11 and the recognition unit 12 are configured as separate image processing devices, the model dictionary registration unit 24 in which necessary information is registered as described above is mounted on the image processing device having the recognition unit 12. Or made available by wireless communication.

  Next, processing when the recognition unit 12 recognizes an input image object will be described with reference to the flowcharts of FIGS.

  The multi-resolution generation unit 31, the feature point extraction unit 32, and the feature amount extraction unit 33 perform the multi-resolution generation unit 21 of the learning unit 11 in steps S11 to S27 on the input object image in steps S131 to S147. Processing similar to that performed by the feature point extraction unit 22 and the feature amount extraction unit 23 is performed. Since the description is repeated, it is omitted. However, the configuration of the multi-resolution image determined by the parameters N and α is different at the time of recognition from that at the time of learning.

  The multi-resolution generation unit 21 generates a multi-resolution image at the time of learning with a fine accuracy with a wide magnification range, whereas the multi-resolution generation unit 31 generates a multi-resolution image with a rough accuracy at the time of recognition. Specifically, the parameters applied in the present embodiment are N = 10 and α = 0.1 during learning in step S12, whereas N = 2 and α = 0.5 during recognition in step S132. . The reason is as follows.

1) In order to increase the recognition accuracy, it is desirable to perform feature amount comparison using more feature point feature amount information. That is, it is desirable to extract feature points from more multi-resolution images.
2) In order to obtain the robustness of the scale change, it is desirable to make the scale range of the multi-resolution image configuration as wide as possible.
3) Since the real-time property does not need to be emphasized so much during model learning, it is possible to increase the number of model resolution multi-resolution images, widen the scale range, and extract and hold feature point feature quantities.
4) In the present embodiment, k-Nearest Neighbor (k-NN) search of a kd tree constructed from all feature point feature values of all models is performed on each feature point feature value extracted from the object image (described later). Since the feature quantities are compared using the method, the calculation cost for the feature quantity comparison increases in proportion to the number of feature points extracted from the object image. When the kd tree is constructed from the target model, the calculation cost can be suppressed to the order of logn (that is, O (logn)) if all model feature points are n.
5) On the other hand, since real-time characteristics are important during recognition, it is necessary to reduce the calculation cost by reducing the number of multi-resolution images as much as possible.
6) However, if a multi-resolution image is not generated from an input object image and only the original image is used, the size of the recognition target object in the object image is larger than the size of the original image of the model image. In such a case, the object cannot be recognized.

  For the above reasons, as shown in FIG. 16, from the model image at the time of learning, more (k = 0 to 9) multi-resolution image groups are generated in a wider range (N = 10, α = 0.1) While extracting more feature points, at the time of recognition, a minimum resolution (k = 0, 1) multi-resolution image group necessary for recognition is generated from the object image (N = 2, α = 0.5). Then, feature points are extracted, and feature quantity comparison is performed by applying a k-NN search on the kd tree, so that it is possible to realize recognition with low calculation cost and high accuracy. In FIG. 16, the original image is too large and there is no model image of the corresponding scale scale. However, the original image (k = 0) is reduced by 0.5 times (k = 1). It is shown that a model image of a hierarchy of size scales can be found.

  If the processes in steps S131 to S145 have been performed for all feature points and all resolution images, the process proceeds to step S148.

  As will be described later, each feature point feature amount extracted from the object image (dimensionally reduced density gradient vector group) is compared with each feature point feature amount of the registered recognition target model, and similar model feature points are obtained. The feature amount and the candidate corresponding feature point set are combined. The simplest feature comparison method is a full search. In other words, for each feature point feature quantity of the object image, it is best to calculate the similarity between feature quantities with all feature point feature quantities of all recognition target models, and select the corresponding feature point set based on the similarity degree. It is a simple method. However, the full search method is not practical in terms of calculation cost. Therefore, in the embodiment of the present invention, a tree search method using a data structure called a kd tree (JH Friedman, JL Bentley, RA Finkel, “An algorithm for finding best” matches in logarithmic expected time, ”ACM Transactions on Mathematical Software, Vol. 3, No. 3, pp. 209-226, September 1977.). Kd-tree means k-dimensional tree structure.

  In the case where the kd tree construction unit 34 only needs to recognize a part of the models registered in the model dictionary in the learning process so far, only the model to be recognized is recognized in step S148. A kd tree is constructed from all feature point features. In the case of the present embodiment, a type 1 feature amount 36d tree (k = 36) and a type 2 feature amount 18d tree (k = 18) are respectively constructed. For each leaf (terminal node) of the tree, one feature point feature quantity is the feature quantity of which feature point is extracted from which scale of which image of which model feature resolution multi-resolution image group. It is held with a label that can be referred to.

  On the other hand, when all the models registered in the model dictionary are recognized, the tree is reconstructed every time the model is additionally learned, and the tree itself is registered in the model dictionary. In this case, the kd tree construction process in step S148 is omitted.

  In step S149, the feature amount comparison unit 35 selects an unprocessed feature point of the object image. In step S150, the feature amount comparison unit 35 pairs the feature point feature amounts of type 1 of the object image with the feature point feature amounts of k similar models. Similarly, in step S151, the feature amount comparison unit 35 pairs a feature point feature amount of type 2 of the object image with feature point feature amounts of k similar models. That is, the feature point feature amounts of the object images extracted by the feature point extraction unit 32 and the feature amount extraction unit 33 are k pieces of feature amounts similar by the k-NN search by the feature amount comparison unit 35 (see FIG. 17). In the example, four model feature point feature values are paired (the k value in the k-NN search and the k value in the kd tree use the same letter k, but are arbitrary. (Of course, it may be the same value). In the present embodiment, as the dissimilarity used for the k-NN search of the feature quantity of type 1, the Euclidean distance of the expression (11) (the larger the value, the more dissimilar) is represented. As the similarity of the feature amount, a cosine correlation value shown in Expression (12) (the larger the value, the more similar the value) is used.

However, in Expression (11), u _V and v _V are feature quantity vectors whose dissimilarities are to be calculated, u _n and v _n are values in the n dimensions of u _V and v _V , respectively, and N is u _V and v Represents the number of dimensions of the _V vector.

In Expression (12), u _V and v _V are feature quantity vectors whose similarity is to be calculated, and u _V · v _V represents an inner product of the vectors. When k pairs with similar feature quantities are extracted, threshold determination may be made for dissimilarity (for type 1 feature quantities) and similarity (for type 2 feature quantities). . The reason why the cosine correlation value is used as the similarity calculation scale for the type 2 feature quantity is to prevent the feature quantity from being affected by the change in the intensity of the local density gradient vector due to the change in brightness. Further, instead of the similarity based on the cosine correlation value, u _V and v _V may be normalized with a vector length of 1, and their Euclidean distance may be set as a dissimilarity to be a type 2 feature amount. Also in this case, the feature amount is not affected by the change in the intensity of the local density gradient vector due to the change in brightness.

  The feature amount comparison unit 35 performs the processing from step S149 to step S151 on the feature points of each object image. In step S152, the feature amount comparison unit 35 determines whether all feature points have been processed. If there is a feature point that has not yet been processed, the process returns to step S149, and the subsequent processing is performed. Run repeatedly. If it is determined in step S152 that all feature points have been processed, the process proceeds to step S153.

  Since two types of feature quantities of type 1 and type 2 are used, the feature quantity comparison unit 35 obtains a feature point pair for the feature point of the input object image for each feature quantity type by the above method, In S153, only the feature point pairs extracted in common for both Type 1 and Type 2 are selected as candidate corresponding feature point groups, and are classified for each model. Then, this candidate corresponding feature point set is supplied to the model posture estimation unit 36 in the subsequent stage. Since the model posture estimation unit 36 performs processing for each model, the extracted candidate corresponding feature point set is classified and passed for each model, so that the processing efficiency can be improved.

  FIG. 17 schematically illustrates the above processing. The kd tree construction unit 34 generates a 36d tree structure of type 1 feature values and an 18d tree structure of type 2 feature values. From the feature amount group of the object image, four similar pair groups of the type 1 feature amount are searched from the 36d tree structure of the type 1 feature amount by k-NN search (in this case, k = 4). In this example, a feature point feature amount represented by a rectangle of an object image (a quadrilateral, pentagon, triangle, circle, or cross figure in the figure represents a feature point feature amount) is a type 1 feature amount. Searched as similar to a pentagon, triangle, circle, or cross in a 36d tree structure. Also, four similar pair groups of type 2 feature quantities are searched from the 18d tree structure of type 2 feature quantities by k-NN search. In this example, the quadrilateral of the object image is searched as being similar to the parallelogram, cross, circle, or rhombus of the type 2 feature quantity 18d tree structure.

  A common similar pair group is selected from four similar pair groups of type 1 feature values and four similar pair groups of type 2 feature values. In the case of this example, there are four similar pairs of type 1 feature quantities: a square and a pentagon, a square and a triangle, a square and a circle, and a square and a cross. On the other hand, there are four similar pairs of type 2 feature quantities: a square and a parallelogram, a square and a cross, a square and a circle, and a square and a rhombus. Therefore, since a similar pair group of a quadrangle and a circle, and a quadrangle and a cross is a feature point pair common to the two types, it is selected as a candidate corresponding feature point pair.

  As described above, for each feature quantity type, one kd tree is constructed from all feature point feature quantities of all recognition target models, and k-NN of each feature point feature quantity of the input image is searched. Instead, a kd tree may be constructed for each feature type and for each model, and the k-NN of the feature point feature amount for each input image may be searched for each model. In any case, the output is a candidate corresponding feature point group grouped for each model, and the later stage processing described later is common.

  Through the above processing, a pair group (a pair group of model feature points and object feature points) having similar local density gradient information in the vicinity of the feature points can be extracted. In addition to the “true feature point pairs (inliers)” in which the spatial positional relationship between the corresponding feature points is consistent with the posture (model posture) on the model object image (input image), the paired pairs are inconsistent Such a “false feature point pair (outlier)” is included.

  FIG. 18 schematically shows inliers and outliers. As shown in the figure, if the triangle model image shown on the left side of the figure corresponds to the object to be detected (object) of the triangle of the object image shown on the right side of the figure, the model image near the vertex of the triangle of the model image The feature points P1 to P4 correspond to the feature points P11 to P14 of the detection target object, respectively. That is, the feature point P1 corresponds to the feature point P11, the feature point P2 corresponds to the feature point P12, the feature point P3 corresponds to the feature point P13, and the feature point P4 corresponds to the feature point P14. Therefore, these candidate corresponding feature point sets constitute an inlier. In FIG. 18, the inlier is indicated by a solid line.

  On the other hand, the feature point P5 of the model image is located approximately at the center inside the triangle, and the feature point P6 is located outside the vicinity of the periphery of the triangle. In contrast, the feature point P15 of the object image paired with the feature point P5 and the feature point P16 of the object image paired with the feature point P6 are points far from the detection target object. . That is, the candidate corresponding feature point set of the feature point P5 and the feature point P15 and the candidate corresponding feature point set of the feature point P6 and the feature point P16 are outliers. In FIG. 18, the outlier is indicated by a broken line.

  As a method for deriving an image conversion parameter for determining the position and orientation of the model image in the input image from the candidate corresponding feature point set group, a method for obtaining the estimated image conversion parameter by least square estimation can be considered. A more accurate model pose is obtained by repeating the process of deriving the estimated image transformation parameters by least square estimation again with the remaining pairs, eliminating the corresponding pairs of the estimated model pose that are inconsistent with the spatial positional relationship. be able to.

  However, when the number of outliers in the candidate corresponding feature point group group is large or when there are outliers that deviate significantly from the true image conversion parameters, the estimation result by the least square estimation is generally satisfactory. (Hartley R., Zisserman A., “Multiple View Geometry in Computer Vision.”, Chapter 3, pp.69-116, Cambridge University Press, 2000). Therefore, the model posture estimation unit 36 in the present embodiment extracts and extracts “true feature point pairs (inliers)” from the spatial positional relationship of the candidate corresponding feature point group group under a certain image conversion constraint. The image conversion parameters that determine the position and orientation of the model are estimated using the inlier.

  The model posture estimation processing by the model posture estimation unit 36 is performed for each recognition target model, and the presence / absence of each model is estimated. The candidate-corresponding feature point group that appears in the following description means a pair group in which only the pairs related to the model among the candidate-corresponding feature point pairs that are the output of the feature amount comparison unit 35 are collected.

  Examples of the image conversion include Euclidean transformation, similarity transformation, affine transformation, projective transformation, and the like. In the present embodiment, detailed description will be given of the case where posture estimation is performed under the constraint of affine transformation. The affine transformation is a transformation that allows shear deformation to the similarity transformation that adds the scaling transformation to the translation and rotation transformation (Euclidean transformation), and the points that are aligned on the straight line in the original figure are aligned on the straight line after the conversion. A parallel line is a transformation that maintains its geometric properties, such as a parallel line after transformation. In order to determine an affine transformation parameter, three or more candidate feature point pairs are required.

  As described above, the affine transformation parameters cannot be determined unless there are three or more candidate feature point sets. Therefore, after selecting one unprocessed model in step S154, the model posture estimation unit 36 determines whether there are three or more candidate corresponding feature point pairs (groups) in step S155. When the number of candidate corresponding feature point sets is two or less, the model posture estimation unit 36 determines that the model does not exist in the object image (input image) or the model posture detection has failed in step S156, and “recognition is impossible”. Output. On the other hand, when there are three or more candidate corresponding feature point sets, the model posture estimation unit 36 can detect the model posture, and thus estimates the affine transformation parameters. For this reason, the model posture estimation unit 36 performs coordinate conversion in step S157. That is, the model feature point position coordinates of the candidate corresponding feature point set are converted to position coordinates on the model original image, and the object image feature point position coordinates are converted to position coordinates of the object original image. In step S158, the model posture estimation unit 36 performs posture estimation processing.

Here, the affine transformation parameters will be described. The affine transformation of the model feature point [xy] ^{T to} the object feature point [u v] ^T is given by the following equation (13).

In this equation (13), a _i (i = 1,..., 4) represents parameters for determining rotation, enlargement / reduction, and shear deformation, and [b ₁ b ₂ ] ^T represents a translation parameter. Since there are six affine transformation parameters a ₁ ,..., A ₄ and b ₁ , b ₂ to be determined, the affine transformation parameters can be determined if there are three candidate corresponding feature point sets.

A pair group P composed of three sets of candidate corresponding feature points is expressed as ([x ₁ y ₁ ] ^T , [u ₁ v ₁ ] ^T ), ([x ₂ y ₂ ] ^T , [u ₂ v ₂ ]. ^T ), ([x ₃ y ₃ ] ^T , [u ₃ v ₃ ] ^T ), the relationship between the pair group P and the affine transformation parameters can be expressed by the linear system shown in the following equation (14). it can.

When this equation (14) is rewritten as Ax _V = b _V (subscript V indicates that the attached character (for example, x of x _V ) is a vector. The same applies hereinafter). The least squares solution of the affine transformation parameter x _V is given by the following equation (15).

x _V = A ⁻¹ b _V (15)

  When the pair group P is repeatedly selected at random so that one or more outliers are mixed from the candidate corresponding feature point set group, the affine transformation parameters are scattered and projected on the parameter space. On the other hand, when a pair group P composed only of inliers is selected repeatedly at random, the affine transformation parameters are all very similar to the true affine transformation parameters of the model pose, that is, close in the parameter space. Become. Therefore, when the pair P is randomly selected from the candidate corresponding feature point set group and the process of projecting the affine transformation parameters onto the parameter space is repeated, the inliers have a high density in the parameter space (the number of members). Many clusters) and outliers appear scattered. That is, if clustering is performed in the parameter space, the cluster element having the largest number of members becomes an inlier.

Details of the posture estimation processing in the model posture estimation unit 36 will be described with reference to the flowchart of FIG. As a clustering method in the model posture estimation unit 36, an NN (Nearest Neighbor) method is used. At this time, since the parameters b ₁ and b ₂ described above can take various values depending on the recognition target image, the selection of the clustering threshold in clustering depends on the recognition target even in the x space. Therefore, the model pose estimation unit 36 says that “there are almost no pair groups P that give affine transformation parameters such that the true parameters and a ₁ ,..., A ₄ are similar, but b ₁ and b ₂ are different”. Under the assumption, clustering is performed only on the parameter space defined by the parameters a ₁ ,..., A ₄ (hereinafter referred to as a _V ). Even if a situation in which the above assumption does not hold, clustering is performed in the parameter space defined by the parameters b ₁ and b ₂ independently of the a _V space, and the result is taken into consideration, so that the problem can be easily solved. Can be avoided.

First, in step S201, the model posture estimation unit 36 performs initialization. Specifically, the count value cnt serving as a variable representing the number of repetitions is set to cnt = 1, randomly selecting three sets of pairs as a pair group P ₁ from the candidates for the corresponding feature point groups, the affine transformation parameter a _V1 Desired. The model posture estimation unit 36 sets a variable N representing the number of clusters to N = 1, and creates a cluster Z ₁ centered on a _V1 on the affine transformation parameter space a _V. The model posture estimation unit 36 updates the centroid c _V1 of the cluster Z ₁ to c _V1 = a _V1 , sets the variable nz ₁ representing the number of members of the cluster to nz ₁ = 1, and updates the counter value cnt to cnt = 2. .

Next, in step S202, the model posture estimation unit 36 randomly selects three pairs as a pair group P _cnt from the candidate corresponding feature point set group, and calculates an affine transformation parameter a _Vcnt . Then, the model posture estimation unit 36 projects the calculated affine transformation parameter a _Vcnt on the parameter space.

Next, in step S203, the model posture estimation unit 36 clusters the affine transformation parameter space by the NN method. Specifically, the model posture estimation unit 36 firstly calculates the distance d () between the affine transformation parameter a _Vcnt and the centroid c _Vi (i = 1,..., N) of each cluster Z _i according to the following equation (16). a minimum distance d _min is obtained from a _Vcnt , c _Vi ).

d _min = min _{1 ≦ i ≦ N} {d (a _Vcnt , c _Vi )} (16)

Then, the model posture estimation unit 36 makes a _Vcnt belong to the cluster Z _i that gives d _min if d _min <τ with respect to a predetermined threshold τ (for example, τ = 0.1), and all members including a _Vcnt To update the centroid c _i of the cluster Z _i . Further, the number of members nz _i cluster Z _i is set to nz _i = nz _i +1. On the other hand, if d _min ≧ τ, the model posture estimation unit 36 creates a new cluster Z _{N + 1} with a _Vcnt as a centroid c _{VN + 1} on the affine transformation parameter space a _V , and the number of members of the cluster nz _{N + 1 is set} to nz _{N + 1} = 1, and the number N of clusters is set to N = N + 1.

  Subsequently, in step S204, the model posture estimation unit 36 determines whether or not the repeated end condition is satisfied. The repetition end condition is, for example, when the maximum number of members exceeds a predetermined threshold (for example, 15) and the difference between the maximum number of members and the second largest number of members exceeds a predetermined threshold (for example, 3), or the repetition number counter This count value cnt can be set so as to exceed a predetermined threshold (for example, 5000 times). If it is determined in step S204 that the repetition end condition is not satisfied (when it is determined No), the model posture estimation unit 36 sets the count value cnt of the number of repetitions to cnt = cnt + 1 in step S205. The process returns to step S202, and the subsequent processes are repeated.

  On the other hand, when it is determined in step S204 that the repeated end condition is satisfied (when determined as Yes), in step S206, the model posture estimation unit 36 sets the inliers obtained by the above processing to three pairs. If not, the affine transformation parameters cannot be determined, so the recognition result is output as “recognition target model not detected”, and if three or more pairs of inliers are extracted, the model is calculated by the least square method based on the inliers. The affine transformation parameters that determine the posture are estimated and output as recognition results.

If the inlier is ([x _IN1 y _IN1 ] ^T , [u _IN1 v _IN1 ] ^T ), ([x _IN2 y _IN2 ] ^T , [u _IN2 v _IN2 ] ^T ), etc., the relationship between the inlier and the affine transformation parameters Can be expressed by a linear system shown in the following equation (17).

When this equation (17) is rewritten as A _IN x _VIN = b _VIN , the least squares solution of the affine transformation parameter x _VIN is given by the following equation (18).

x _VIN = (A _IN ^T A _IN ) ¹ A _IN ^T b _VIN (18)

In step S206, the model posture estimation unit 36 outputs the model posture determined by the affine transformation parameter _xVIN as a model recognition result.

  Returning to FIG. 15, after step S158 or step S156, in step S159, the model posture estimation unit 36 determines whether all models have been processed. If there is a model that has not been processed yet, the processing returns to step S154, and the subsequent processing is repeatedly executed. If it is determined in step S159 that all models have been processed, the process ends.

  The processes in steps S154 to S159 in FIG. 15 are performed for each recognition target model. This process is schematically shown in FIG. In this example, three candidate corresponding feature point group groups p1, p3, and p4 are first selected at random from the candidate corresponding feature point group groups p1 to p6, and the affine parameters calculated based on the candidate candidate feature point group groups p1, p3, and p4 are projected onto the parameter space. The Next, three candidate corresponding feature point group groups p3, p4, and p6 are selected at random, and based on them, the calculated affine parameters are projected onto the parameter space. Similar processing is further repeated. In this example, three candidate corresponding feature point group groups p5, p4, and p1 are selected, and affine parameters are calculated based on the selected candidate feature point group groups p5, p4, and p1. Then, adjacent affine parameters are clustered in the parameter space, and the model orientation is determined by applying the least square method to the clustered affine transformation parameters.

  By using the above method, even when a large number of outliers are included in the candidate corresponding feature point set group, outliers can be excluded and posture estimation (conversion parameter derivation) can be performed with high accuracy.

  In the above embodiments, details of posture estimation under affine transformation constraints have been described. Under the affine transformation constraint, if the plane area is dominant, for example, a three-dimensional object such as a box or a book, it is possible to perform posture estimation that is robust to a change in viewpoint about the dominant plane. However, in order to perform robust posture estimation of a three-dimensional object in which curved surfaces and unevenness are dominant, it is necessary to extend the affine transformation constraint to a projection transformation constraint. However, even in this case, the above-described method can be easily expanded only by increasing the dimension of the conversion parameter to be estimated.

  The model posture thus determined is indicated by a broken line in FIGS. 16 and 18, for example. As shown in these figures, in the present embodiment, not only the presence / absence of the detection target object corresponding to the model image is detected, but also the posture when the detection target object exists. Is also estimated and output.

  The model posture estimated by the model posture estimation unit 36 means a relative posture of the object image with respect to the detection target object. Therefore, when the model posture is considered as a reference posture, the model posture estimation unit 36 This means that the posture of the detection target object with respect to the model image is estimated.

  In the above description, the threshold τ is assumed to be a constant value. However, when performing the iterative processing from step S203 to step S206, first, a rough inlier extraction is performed using a relatively large threshold τ, and the repetition is repeated. A technique such as a so-called “annealing method” that uses a gradually smaller threshold τ each time the number of times increases may be applied. Thereby, an inlier can be extracted with high accuracy.

  In the above description, the cluster having the largest number of members in the parameter space is repeated by selecting a pair group P randomly from the candidate corresponding feature point set group and projecting the affine transformation parameters onto the parameter space. The affine transformation parameters that determine the model pose by the least squares method were estimated using the elements of, but this is not a limitation. For example, the centroid of the cluster with the largest number of members is used to determine the model pose. It does not matter as a conversion parameter. Further, the set may be composed of three or more feature points.

  As described above, each time the object image is updated, the multi-resolution generation unit 31, the feature point extraction unit 32, the feature amount extraction unit 33, the feature amount comparison unit 35, and the model posture estimation unit 36 at the time of object recognition of the object image. Since it is sequentially processed, object recognition in real time becomes possible. Further, the feature point pairs extracted by the feature amount comparison unit 35 for each model are classified for each model, and posture estimation is performed for each model in the model posture estimation unit 36, so that a plurality of model objects are included in the input image. It is possible to recognize a model object even in such an image.

  By the way, as the ratio of outliers in the candidate corresponding feature point pair group generated by the feature quantity comparison unit 35 increases, the selection probability of the inlier in the model posture estimation unit 36 decreases, and the number of iterations increases when estimating the model posture. Therefore, the calculation time increases. Therefore, it is desirable to remove outliers as much as possible from the candidate corresponding feature point pair group input to the model posture estimation unit 36. Therefore, in the image processing apparatus 1 according to the present embodiment, as shown in FIG. 21, the corresponding feature point pair is narrowed down between the feature amount comparison unit 35 and the model posture estimation unit 36. The part 61 can be inserted. In the corresponding feature point pair narrowing-down unit 61, the image transformation constraint having a smaller parameter dimension than the image transformation constraint assumed in the model posture estimation process of the model posture estimation unit 36 (for example, the model posture estimation unit 36 uses the affine transformation constraint). If the corresponding feature point pair narrowing unit 61 uses a similarity transformation constraint), a generalized Hough transform is performed on each model to estimate a rough image transformation parameter and to extract a corresponding feature point pair group that supports it. Do.

  Next, with reference to the flowchart of FIG. 22, the candidate corresponding feature point pair narrowing process by the corresponding feature point pair narrowing unit 61 when the similar transformation constraint is used as the image transformation constraint will be described. The similarity transformation is a transformation obtained by adding an enlargement / reduction transformation (scl) to the translation (the displacement dX in the X-axis direction and the displacement dY in the Y-axis direction because it is on a two-dimensional image) and the rotation transformation (θ).

  In step S301, the corresponding feature point pair narrowing unit 61 defines a range for each axis of the four-dimensional parameter space defined by the similarity transformation parameters dX, dY, θ, scl, and the step width defined within the range. Create a voting space for generalized Hough transform by cutting the bin with For example, the range is ± 200 for the dX and dY axes, the step width is 20 (in pixels), the range is 0 to 360 for the θ axis, the step width is 15 (in degrees), and the range is 0.4 to 1.6 for the scl axis. If the step width is 0.3 (the unit is the enlargement / reduction ratio), 20 x 24 x 5 = 2400 voting bins are created.

  After step S302, the process is performed for each model. In step S302, the corresponding feature point pair narrowing unit 61 initializes the voting space and selects an unprocessed model. That is, for example, the value of each voting bin is set to 0, and one model is selected.

  Since step S303 to step S309 are processing for each candidate corresponding feature point pair (set) of the model, in step S303, the corresponding feature point pair narrowing unit 61 selects an unprocessed candidate corresponding feature point pair. That is, one unprocessed candidate corresponding feature point pair is selected as a processing target.

Subsequently, in step S304, the corresponding feature point pair narrowing unit 61 obtains the rotation conversion angle θ. For example, as shown in FIG. 23A, the canonical direction of the feature point P of the model image is D _P, the canonical direction of the feature point Q of the object image as a feature point P and the pair (set) as shown in Figure 23B Thus, in the case of D _Q , as shown in FIG. 24, the difference between the canonical directions D _Q and D _P obtained when the feature point P and the feature point Q are arranged at the corresponding positions is the rotation conversion angle. θ.

  Steps S305 to S309 relate to the representative scl values of the bins defined on the scl axis (in the above example, the representative scl values of 5 bins cut on the scl axis are 0.4, 0.7, 1.0, 1.3, and 1.6, respectively) The processing to be performed. Therefore, the corresponding feature point pair narrowing unit 61 first selects an unprocessed scl value in step S305. That is, one unprocessed scl value is selected.

  Next, in step S306, the corresponding feature point pair narrowing unit 61 obtains translational conversion amounts dX and dY when the model image is enlarged or reduced by the scl value as shown in FIG. The translation conversion amounts dX and dY are the rotation conversion angle θ obtained in step S304, the coordinates Xi and Yi of the feature point Q on the object image, the distance γ from the origin in the polar coordinates of the feature point P on the model image, and The following equation (19) and equation (20) can be derived from the angle α from the axis.

dX = Xi-scl · γ · cos (α + θ) (19)
dY = Yi-scl · γ · sin (α + θ) (20)

  Since the four parameters (scl, θ, dX, dY) for determining the similarity transformation have been obtained, in step S307, the corresponding feature point pair narrowing unit 61 votes the parameters on the voting space. Specifically, the parameter is voted for the corresponding bin in the voting space.

  The corresponding feature point pair narrowing unit 61 performs the above-described processing from step S303 for all the corresponding feature point pairs and all the scl values for the model in accordance with the branch processing of steps S308 and S309. That is, in step S308, the corresponding feature point pair narrowing unit 61 determines whether all scl values have been processed. If there is an scl value that has not yet been processed, the process returns to step S305, and the subsequent steps Repeat the process. If it is determined in step S308 that all scl values have been processed, in step S309, the corresponding feature point pair narrowing unit 61 determines whether all corresponding feature point pairs have been processed, and corresponding features that have not yet been processed. If a point pair exists, the process returns to step S303, and the subsequent processes are repeated.

  When it is determined in step S309 that all the corresponding feature point pairs have been processed, in step S310, the corresponding feature point pair narrowing unit 61 obtains the most-voting-bin bin and the number of votes. That is, as a result of voting by the process of step S307, the bin with the largest number of votes and the number of votes obtained are obtained. In step S311, the corresponding feature point pair narrowing unit 61 determines whether the maximum number of votes obtained in step S310 is greater than or equal to a threshold value. This threshold is set in advance. If the maximum number of votes is greater than or equal to the threshold value, in step S312, the corresponding feature point pair narrowing unit 61 determines that there is a model, and outputs the candidate corresponding feature point pairs voted in the largest number of vote bins. That is, the similarity transformation parameter indicated by the most vote bin is the estimated model posture under the similarity transformation constraint on the object image of the model, and the candidate corresponding feature point pair voted to the most vote bin is the similarity transformation constraint. Inliers (which may include a very small number of outliers) that support the estimated model pose under. If it is determined in step S311 that the maximum number of votes is not greater than or equal to the threshold, the corresponding feature point pair narrowing unit 61 determines that there is no model in step S313. In other words, for models that do not exist in the object image, the votes will vary, so the maximum number of votes for the result of the generalized Hough transform for each model is below the threshold. In this case, it is determined that the model has not been detected.

  As described above, the processing from step S302 to step S313 is performed for all the recognition target models in accordance with the branch processing of step S314. That is, after the process of step S312 or step S313, in step S314, the corresponding feature point pair narrowing unit 61 determines whether all models have been processed. If there is a model that has not yet been processed, the process returns to step S302, and the subsequent processes are repeatedly executed. If it is determined in step S314 that all models have been processed, the candidate-corresponding feature point pair narrowing-down process ends.

  For the model determined to have a model in step S312, an inlier for the model is supplied to the model posture estimation unit 36, and a more accurate model posture estimation is performed by the above-described method.

  In the processing by the corresponding feature amount pair narrowing unit 61 described in detail above, coarse image conversion parameter estimation is performed under an image conversion constraint having a smaller parameter dimension than the image conversion constraint assumed in the model posture estimation process of the model posture estimation unit 36. This makes it possible to narrow down candidate-corresponding feature point pairs at high speed. By using the refined feature point pairs, the model posture estimation unit 36 estimates the image conversion parameters, so that it is faster than when the corresponding feature point pair narrowing unit 61 is not inserted in the previous stage of the model posture estimation unit 36. Image conversion parameters can be estimated with high accuracy. In the processing by the corresponding feature amount pair narrowing unit 61, an error between the actual conversion and the estimated conversion occurs due to the reduction of the dimension of the image conversion parameter. However, the size of each bin in the voting space can be tolerated. The problem is solved by taking it large.

  As described above, according to the image processing apparatus 1 in the present embodiment, the following becomes possible.

1. Since the resolution image group is configured, noise is removed as a result, and robust recognition can be performed against noise.
2. Since the feature amount obtained by normalizing the canonical direction of the local density gradient of the feature point to 0 degrees is used, it is possible to recognize the rotation change robustly.
3. Since the direction information is used for the feature quantity and its matching instead of the intensity information of the local area concentration gradient that is easily affected by the brightness change, it is possible to perform robust recognition against the brightness change.
4). Since matching between feature amounts is performed between all resolutions, robust recognition can be made with respect to enlargement / reduction changes.
5. Since the affine constraint and the generalized Hough transform are used after the feature point pair extraction, robust recognition with respect to the affine transformation becomes possible.
6). Since local feature matching is used, recognition is possible even when the recognition target object is partially hidden.
7). Since feature point pairs are extracted for each model and posture estimation is performed for each model, it is possible to recognize a model object even in an image in which a plurality of model objects are included in the input image.
8). During learning, feature points and feature quantities are extracted with a finer scale sampling with a wider scale range, while during recognition, feature points and feature quantities are extracted with a coarse scale range and coarse scale sampling, and feature quantity comparison is performed using a kd tree. By using the k-NN search method, it is possible to reduce the recognition calculation cost without degrading the recognition accuracy. That is, the object can be recognized accurately in real time.

  The series of processes described above can be executed by hardware or can be executed by software. In this case, for example, the image processing apparatus 1 is configured by a personal computer as shown in FIG.

  26, a CPU (Central Processing Unit) 121 executes various processes according to a program stored in a ROM (Read Only Memory) 122 or a program loaded from a storage unit 128 to a RAM (Random Access Memory) 123. To do. The RAM 123 also appropriately stores data necessary for the CPU 121 to execute various processes.

  The CPU 121, ROM 122, and RAM 123 are connected to each other via a bus 124. An input / output interface 125 is also connected to the bus 124.

  The input / output interface 125 includes an input unit 126 including a keyboard and a mouse, a display including a CRT (Cathode Ray Tube) and an LCD (Liquid Crystal display), an output unit 127 including a speaker, and a hard disk. A communication unit 129 including a storage unit 128 and a modem is connected. The communication unit 129 performs communication processing via a network including the Internet.

  A drive 130 is connected to the input / output interface 125 as necessary, and a removable medium 131 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is appropriately mounted, and a computer program read from them is It is installed in the storage unit 128 as necessary.

  When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, a general-purpose personal computer is installed from a network or a recording medium.

  As shown in FIG. 26, the recording medium is distributed to provide a program to the user separately from the apparatus main body, and includes a magnetic disk (including a floppy disk) on which the program is recorded, an optical disk (CD- In addition to ROM (Compact Disk-Read Only Memory), DVD (including Digital Versatile Disk)), magneto-optical disk (including MD (Mini-Disk)), or removable media 131 composed of semiconductor memory, etc. The program is provided with a ROM 122 in which a program is recorded and a hard disk included in the storage unit 128, which is provided to the user in a state of being incorporated in the apparatus main body in advance.

  In the present specification, the step of describing the program recorded on the recording medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. It also includes processes that are executed individually.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  The present invention can be applied to a robot apparatus.

It is a block diagram which shows the structural example of the image processing apparatus to which this invention is applied. It is a flowchart explaining the learning process of the learning part of FIG. It is a flowchart explaining the learning process of the learning part of FIG. It is a figure explaining a resolution image. It is a figure explaining the scale space of a DoG filter. It is a figure explaining the density | concentration gradient direction of the feature point vicinity. It is a figure explaining the calculation method of the frequency of a histogram. It is a figure which shows the example of a direction histogram. It is a figure which shows the example of a direction histogram. It is a figure which shows the example of a direction histogram. It is a figure explaining the process of feature-value extraction. It is a figure which shows the example of resampling. It is a flowchart explaining the recognition process of the recognition part of FIG. It is a flowchart explaining the recognition process of the recognition part of FIG. It is a flowchart explaining the recognition process of the recognition part of FIG. It is a figure explaining the multi-resolution at the time of learning and recognition. It is a figure explaining the comparison process of the feature-value. It is a figure explaining an inlier and an outlier. It is a flowchart explaining the detail of an attitude | position estimation process. It is a figure explaining the attitude | position estimation process of FIG. It is a block diagram which shows the other structural example of the image processing apparatus to which this invention is applied. It is a flowchart explaining the process of the corresponding feature point pair narrowing-down part of FIG. It is a figure explaining the corresponding feature point of a model image and an object image. It is a figure explaining the rotation conversion angle of the feature point of a model image and an object image. It is a figure explaining the translation displacement amount. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image processing apparatus, 11 Learning part, 12 Recognition part, 21 Multi-resolution production | generation part, 22 Feature point extraction part, 23 Feature-value extraction part, 24 Model dictionary registration part, 31 Multi-resolution production part, 32 Feature point extraction part, 33 Feature extraction unit, 34 kd tree construction unit, 35 feature comparison unit, 36 model pose estimation unit

Claims (17)

In an image processing apparatus that recognizes a pre-registered model image from an input image,
Multi-resolution image generation means for generating a multi-resolution image composed of a plurality of different resolution images by reducing the resolution of the input image at a predetermined rate;
Feature point extraction means for extracting feature points for each resolution image of the multi-resolution image;
Feature quantity extraction means for extracting at least two local feature quantities at the feature points;
Comparing means for comparing the feature quantity of the input image with the feature quantity of the model image, and generating a candidate corresponding feature point set as a set of feature points having similar feature quantities;
An image processing apparatus comprising: posture estimation means for estimating the posture of an input image based on the candidate corresponding feature point set. The feature quantity extraction unit extracts a density gradient direction histogram near the feature point as a first type feature quantity, and extracts a dimensional degenerate density gradient vector as a second type feature quantity. The image processing apparatus according to claim 1. The comparison means searches for k Neest Neighbor (k-NN) based on the feature amount of the input image for the feature amount group of the model image, which is a kd tree for each type, so that the candidate correspondence The image processing apparatus according to claim 2, wherein a feature point set is generated. The image processing apparatus according to claim 3, wherein the comparison unit sets the feature points having the feature amount extracted in common for each type as the candidate corresponding feature point set. The candidate corresponding feature point set is further voted on a parameter space defined by an image conversion parameter for determining the position and orientation of the model image, and further includes a narrowing means for narrowing down by comparing the maximum number of votes with a threshold value,
The image processing apparatus according to claim 4, wherein the posture estimation unit estimates a posture of an input image based on the narrowed candidate corresponding feature point set. The posture estimation means projects image conversion parameters that determine the position and posture of the model image determined by the randomly selected N sets of candidate corresponding feature point sets to a parameter space, and clusters formed on the parameter space The image conversion parameter for determining the position and orientation of the model image obtained from the member by the least square method is output as a recognition result for recognizing the model image. The image processing apparatus according to 1. The posture estimation means detects a centroid of the cluster having the largest number of members, and outputs an image conversion parameter for determining a position and posture of the model image composed of the centroid as a recognition result for recognizing the model image. The image processing apparatus according to claim 6. The image processing apparatus according to claim 1, wherein the multi-resolution image generation unit generates the multi-resolution image with coarser accuracy than in the case of learning. In an image processing method of an image processing apparatus that recognizes a model image registered in advance from an input image,
A multi-resolution image generation step of generating a multi-resolution image consisting of a plurality of different resolution images by reducing the resolution of the input image at a predetermined rate;
A feature point extracting step of extracting feature points for each resolution image of the multi-resolution image;
A feature quantity extraction step of extracting at least two local feature quantities at the feature points;
A comparison step of comparing the feature amount of the input image with the feature amount of the model image and generating a candidate corresponding feature point set as a set of feature points having similar feature amounts;
A posture estimation step of estimating a posture of the input image based on the candidate corresponding feature point set. A program of an image processing apparatus that recognizes a model image registered in advance from an input image,
A multi-resolution image generation step of generating a multi-resolution image consisting of a plurality of different resolution images by reducing the resolution of the input image at a predetermined rate;
A feature point extracting step of extracting feature points for each resolution image of the multi-resolution image;
A feature quantity extraction step of extracting at least two local feature quantities at the feature points;
A comparison step of comparing the feature amount of the input image with the feature amount of the model image and generating a candidate corresponding feature point set as a set of feature points having similar feature amounts;
And a posture estimation step of estimating a posture of an input image based on the candidate corresponding feature point set. A recording medium on which a computer-readable program is recorded. A program of an image processing apparatus that recognizes a model image registered in advance from an input image,
A multi-resolution image generation step of generating a multi-resolution image consisting of a plurality of different resolution images by reducing the resolution of the input image at a predetermined rate;
A feature point extracting step of extracting feature points for each resolution image of the multi-resolution image;
A feature quantity extraction step of extracting at least two local feature quantities at the feature points;
A comparison step of comparing the feature amount of the input image with the feature amount of the model image and generating a candidate corresponding feature point set as a set of feature points having similar feature amounts;
A program causing a computer to execute a posture estimation step of estimating a posture of an input image based on the candidate corresponding feature point set. In an image processing apparatus for learning a model image for comparison with an image to be recognized,
A multi-resolution image generating means for generating a multi-resolution image consisting of a plurality of different resolution images with a finer precision than in the case of recognition by reducing the resolution of the model image at a predetermined rate;
Feature point extraction means for extracting feature points for each resolution image of the multi-resolution image;
Feature quantity extraction means for extracting at least two local feature quantities at the feature points;
An image processing apparatus comprising: a registration unit configured to register the feature amount of the model image. The feature quantity extraction unit extracts a density gradient direction histogram near the feature point as a first type feature quantity, and extracts a dimensional degenerate density gradient vector as a second type feature quantity. The image processing apparatus according to claim 12. The image processing apparatus according to claim 13, wherein the registration unit registers the feature amount group of the model image as a kd tree for each type. In an image processing method of an image processing apparatus for learning a model image for comparison with an image to be recognized,
A multi-resolution image generation step for generating a multi-resolution image composed of images of a plurality of different resolutions with a finer precision than in the case of recognition by reducing the resolution of the model image at a predetermined rate;
A feature point extracting step of extracting feature points for each resolution image of the multi-resolution image;
A feature quantity extraction step of extracting at least two local feature quantities at the feature points;
A registration step of registering the feature quantity of the model image. A program of an image processing device for learning a model image for comparison with an image to be recognized,
Reducing the resolution of the model image at a predetermined rate, thereby generating a multi-resolution image composed of a plurality of different resolution images with a finer precision than in the case of recognition; and
A feature point extracting step of extracting feature points for each resolution image of the multi-resolution image;
A feature quantity extraction step of extracting at least two local feature quantities at the feature points;
And a registration step for registering the feature quantity of the model image. A recording medium on which a computer-readable program is recorded. A program of an image processing device for learning a model image for comparison with an image to be recognized,
Reducing the resolution of the model image at a predetermined rate, thereby generating a multi-resolution image composed of a plurality of different resolution images with a finer precision than in the case of recognition; and
A feature point extracting step of extracting feature points for each resolution image of the multi-resolution image;
A feature quantity extraction step of extracting at least two local feature quantities at the feature points;
A program causing a computer to execute a registration step of registering the feature amount of the model image.

Images