JP2013058174A - Image processing program, image processing method, and image processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce a calculation amount in associating coordinates of respective images with each other.SOLUTION: An image processing device 100 extracts feature points from learning target image data 130a and a viewpoint variation image data group 130b with different camera viewpoints, and classifies the respective feature points to generate a registration table 130c. The image processing device 100 compares a centroid coordinate of each feature point group of the registration table 130c with the coordinate of a feature point extracted from recognition target image data 130d to associate the feature point of the learning target image data 130a with the feature point of the recognition target image data 130d.

Description

  The present invention relates to an image processing program and the like.

  There is a technique of learning an image in advance and recognizing the posture of the captured image based on the previously learned image when the same image as that image is captured from a certain camera viewpoint. In order to recognize the orientation of the photographed image, a process of associating the coordinates of the image learned in advance with the coordinates of the photographed image is performed. Techniques for associating the coordinates of each image include ASIFT and Randomized Trees.

  For example, in the technique based on Randomized Trees, the coordinates of a learned image and a captured image are randomly selected, and the pixels of the selected coordinates are sequentially compared to associate the coordinates of each image.

JP 2010-204826 A Japanese Patent Laid-Open No. 2005-215988

J. M. Morel and G. Yu, “ASIFT: A new framework for fully affine invariant image comparison”, SIAM Journal on Imaging Sciences, 2, 2, pp. 438-469, 2009 V. Lepetit and P. Fua, “Keypoint recognition using randomized trees”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 28, 9, pp. 1465-1479, 2006

  However, the above-described conventional technique has a problem that the amount of calculation is large when the coordinates of the learned image are associated with the coordinates of the image that is the posture recognition target.

  For example, in the method based on Randomized Trees, since there are many combinations of pixels to be compared, if each combination is compared one by one, a long time is required for calculation.

  The disclosed technique has been made in view of the above, and an object thereof is to provide an image processing program, an image processing method, and an image processing apparatus that can reduce the amount of calculation when the coordinates of each image are associated. To do.

  The disclosed image processing program causes a computer to execute the following processing. The image processing program causes a computer to generate a plurality of viewpoint variation images having different camera viewpoints with respect to the learning target image by projective conversion of the learning target image obtained by capturing the image from the front camera viewpoint. A computer extracts feature points from the learning target image and the plurality of viewpoint variation images. By associating each feature point extracted from a plurality of viewpoint variation images with each feature point extracted from the learning target image on a computer, each feature point extracted from the plurality of viewpoint variation images is converted into a plurality of feature point groups. Let them be classified. The computer registers the region including the feature point group in the registration table. The computer acquires a recognition target image and causes feature points to be extracted from the recognition target image. The computer associates the feature point of the recognition target image with the feature point of the learning target image based on the relationship between the position of the feature point extracted from the recognition target image and the region of the registration table.

  According to the disclosed image processing program, it is possible to reduce the amount of calculation when the coordinates of each image are associated.

FIG. 1 is a functional block diagram of the configuration of the image processing apparatus according to the first embodiment. FIG. 2 is a diagram for explaining the relationship between learning target image data and viewpoint variation image data. FIG. 3 is a diagram illustrating an example of feature points extracted from learning target image data. FIG. 4 is a diagram illustrating an example of a feature point extraction result from learning target image data. FIG. 5 is a diagram illustrating an example of feature points extracted from the viewpoint variation image data group. FIG. 6 is a diagram illustrating an example of a feature point extraction result from the viewpoint variation image data. FIG. 7 is a diagram illustrating an example of the conversion result. FIG. 8 is a diagram for explaining the barycentric coordinates on the feature space. FIG. 9 is a diagram illustrating an example of the data structure of the registration table according to the first embodiment. FIG. 10 is a flowchart illustrating the processing procedure of the learning phase according to the first embodiment. FIG. 11 is a flowchart illustrating the processing procedure of the recognition phase according to the first embodiment. FIG. 12 is a functional block diagram of the configuration of the image processing apparatus according to the second embodiment. FIG. 13 is a diagram for explaining the processing of the classification unit according to the second embodiment. FIG. 14 is a diagram illustrating an example of a data structure of a registration table according to the second embodiment. FIG. 15 is a functional block diagram of the configuration of the image processing apparatus according to the third embodiment. FIG. 16 is a diagram (1) for explaining the process of the classification unit according to the third embodiment. FIG. 17 is a diagram (2) for explaining the process of the classification unit according to the third embodiment. FIG. 18 is a diagram illustrating an example of the data structure of the registration table according to the third embodiment. FIG. 19 is a diagram illustrating an example of a computer that executes an image processing program.

  Embodiments of an image processing program, an image processing method, and an image processing apparatus disclosed in the present application will be described below in detail with reference to the drawings. Note that the present invention is not limited to the embodiments.

  An image processing apparatus according to the first embodiment will be described. FIG. 1 is a functional block diagram of the configuration of the image processing apparatus according to the first embodiment. As illustrated in FIG. 1, the image processing apparatus 100 includes a camera 110a, an input unit 110b, an output unit 120, a storage unit 130, and a control unit 140.

  The camera 110a captures an image of the object. In the learning phase, the camera 110a captures an image of an object to be learned, and outputs captured image data to the control unit 140. In the recognition phase, the camera 110a captures an image of a target object whose posture is to be recognized, and outputs data of the captured image to the control unit 140. In the following description, data of an image to be learned is referred to as learning target image data. The image data to be recognized is referred to as recognition target image data.

  The learning object image data is data of an image obtained by photographing the object from the front. The recognition target image data is data of an image obtained by photographing the target object from an arbitrary direction. The object to be imaged in the learning object image data and the object to be imaged in the recognition object image data are the same object. The object may have some unevenness, but is preferably planar.

  The input unit 110 b is an input device that inputs various data to the image processing apparatus 100. For example, the input unit 110b corresponds to an input key, a touch panel, or the like. The output unit 120 is an output device that outputs the processing result of the control unit 140.

  The storage unit 130 stores learning target image data 130a, a viewpoint variation image data group 130b, a registration table 130c, and recognition target image data 130d. The storage unit 130 corresponds to, for example, a semiconductor memory device such as a random access memory (RAM), a read only memory (ROM), and a flash memory, or a storage device such as a hard disk or an optical disk.

  As described above, the learning target image data 130a is data of an image to be learned taken by the camera 110a.

  The viewpoint changing image data group 130b is an image data group corresponding to images obtained by photographing the object from various camera viewpoints.

  The registration table 130c is a table that holds various types of information related to feature points extracted from the learning target image data 130a and the viewpoint variation image data 130b. The data structure of the registration table 130c will be described later.

  As described above, the recognition target image data 130d is data of an image to be recognized captured by the camera 110a.

  The control unit 140 includes a data management unit 140a, a viewpoint variation image generation unit 140b, a feature point extraction unit 140c, a classification unit 140d, and an association unit 140e. The control unit 140 corresponds to an integrated device such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array). The control unit 140 corresponds to an electronic circuit such as a CPU or MPU (Micro Processing Unit).

  The data management unit 140 a is a processing unit that manages the storage unit 130. For example, in the learning phase, the data management unit 140a acquires learning target image data from the camera 110a, and stores the acquired learning target image data in the storage unit 130. In the recognition phase, the data management unit 140a acquires recognition target image data from the camera 110a, and stores the acquired recognition target image data in the storage unit 130.

  The viewpoint variation image generation unit 140b is a processing unit that generates a viewpoint variation image data group 130b based on the learning target image data 130a. The viewpoint variation image generation unit 140b generates viewpoint variation image data with different camera viewpoints for each camera viewpoint with respect to the learning target image data 130a.

  FIG. 2 is a diagram for explaining the relationship between learning target image data and viewpoint variation image data. The learning target image data 130 a is data of an image obtained by capturing the target object 10 from the camera viewpoint 20 in front of the target object 10. The line-of-sight variation image data group 130b corresponds to each data of an image obtained by capturing the object 10 from different camera viewpoints 30a to 30d for θ, ψ, and φ. For example, θ, ψ, and φ are set as a roll angle, a pitch angle, and a yaw angle, respectively.

  The viewpoint variation image generation unit 140b generates viewpoint variation image data by applying projective transformation having θ, ψ, and φ as parameters to the learning target image data 130a. The viewpoint variation image generation unit 140b sets projection transformation parameters with values of ψ and φ at intervals of 10 ° in a range of −60 ° to 60 °. In addition, the viewpoint variation image generation unit 140b sets projection transformation parameters at intervals of 45 ° in the range of θ from 0 ° to 360 °.

  The viewpoint variation image generation unit 140b performs projective transformation of each parameter, and generates viewpoint variation image data for each parameter. The viewpoint variation image generation unit 140b stores the generated plurality of viewpoint variation image data in the storage unit 130 as a viewpoint variation image data group 130b.

  The projection transformation performed by the viewpoint variation image generation unit 140b may use any conventional projection transformation. For example, the viewpoint variation image generation unit 140b performs projective transformation based on literature (Akira Kanazawa, Kenichi Kanaya, 'Automatic search for feature point correspondence between two images', Image Lab, pp.20-23, 2004). .

  The feature point extraction unit 140c extracts feature points from the learning target image data 130a. In addition, the feature point extraction unit 140c extracts feature points from each viewpoint variation image data of the viewpoint variation image data group 130b. The feature point extraction unit 140c outputs data on the feature points extracted from the learning target image data 130a and data on the feature points extracted from the viewpoint variation image data group 130b to the classification unit 140d.

  The feature point extraction unit 140c extracts feature points using, for example, SIFT (Scale Invariant Feature Transform). SIFT is a technique for scanning image data and detecting points suitable for feature extraction. For example, the feature point extraction unit 140c scans the image data, and in each portion, a plurality of elements such as the sharpness of the edge of the image, the gradient of the gray value, and the direction of the gradient change are digitized. Then, a portion where the numerical value of each element takes an extreme value is extracted as a feature point.

  The feature point extraction unit 140c extracts feature points from the learning target image data 130a using SIFT. FIG. 3 is a diagram illustrating an example of feature points extracted from learning target image data. In the example illustrated in FIG. 3, the feature point extraction unit 140c extracts feature points 40A, 40B, and 40C from the learning target image data 130a. The feature point extraction unit 140c outputs the extraction result from the learning target image data 130a to the classification unit 140d.

  FIG. 4 is a diagram illustrating an example of a feature point extraction result from learning target image data. As shown in FIG. 4, the extraction result 1A associates feature point identification information with coordinates on learning target image data.

  The feature point extraction unit 140c uses SIFT to extract feature points from each viewpoint variation image data included in the viewpoint variation image data group 130b. FIG. 5 is a diagram illustrating an example of feature points extracted from the viewpoint variation image data group. In the example shown in FIG. 5, the viewpoint variation image data group 130b includes viewpoint variation image data 13a, 13b, 13c, and 13d.

  The feature point extraction unit 140c extracts the feature points 1a and 1b from the viewpoint variation image data 13a. The feature point extraction unit 140c extracts feature points 1c and 1d from the viewpoint variation image data 13b. The feature point extraction unit 140c extracts feature points 1e, 1f, and 1g from the viewpoint variation image data 13c. The feature point extraction unit 140c extracts feature points 1h and 1i from the viewpoint variation image data 13d. The feature point extraction unit 140c outputs the extraction result from the viewpoint variation image data group 130b to the classification unit 140d.

  FIG. 6 is a diagram illustrating an example of a feature point extraction result from the viewpoint variation image data. As shown in FIG. 6, this extraction result 1B associates the feature point identification information with the coordinates on the viewpoint variation image data. The feature point extraction unit 140c associates the numerical value of each element with the feature point in the extraction result 1B. Here, the numerical value of each element is obtained by numerically expressing, for example, the sharpness of the edge of the image, the gradient of the gray value, the direction of the gradient change, and the like.

  The classification unit 140d is a processing unit that classifies a plurality of feature points extracted from the viewpoint variation image data group 130b. For example, the classification unit 140d classifies the feature points based on the extraction result 1A and the extraction result 1B.

  The classification unit 140d converts the coordinates on the viewpoint variation image data of the extraction result 1B into the coordinates on the learning target image data. The classification unit 140d converts the inverse matrix of the projective transformation to the coordinates on the learning target image data by applying the inverse matrix to the coordinates on the viewpoint variation image data. FIG. 7 is a diagram illustrating an example of the conversion result.

  As shown in FIG. 7, the conversion result 1C associates identification information, coordinates, conversion coordinates, and each element. Of these, the transformation coordinates are the coordinates of the feature points transformed by the inverse matrix of the projective transformation.

  The classification unit 140d classifies each feature point extracted from the viewpoint variation image data according to the distance between the coordinates of the extraction result 1A in FIG. 4 and the conversion coordinates of the conversion result 1C in FIG. For example, when the conversion coordinates of a certain feature point extracted from the viewpoint variation image data are closest to the coordinates of the feature point 40A among the feature points 40A to 40C, the feature point is classified as the feature point 40A. The classification unit 140d performs the above process on each feature point extracted from the viewpoint variation image data, and classifies the feature point as one of the feature point 40A, the feature point 40B, and the feature point 40C.

  The classifying unit 140d excludes the feature points from the classification target when the converted coordinates of the feature points are separated from the coordinates of the feature points 40A to 40C by a predetermined threshold or more. For example, the predetermined threshold is 2 pixels.

  The classification unit 140d classifies each feature point extracted from the viewpoint variation image data, and then obtains the barycentric coordinate in the feature space for each classified feature point group. FIG. 8 is a diagram for explaining the barycentric coordinates on the feature space. For example, the x-axis in FIG. 8 corresponds to the feature point element 1, the y-axis corresponds to the feature point element 2, and the z-axis corresponds to the feature point element 3. Here, as an example, the feature space is shown in three dimensions, but when the number of feature point elements is n, the feature space becomes n dimensions. In FIG. 8, circles correspond to the feature points classified as the feature points 40A. The triangle mark corresponds to the feature point classified as the feature point 40B. The square marks correspond to the feature points classified as the feature points 40C.

  The classification unit 140d obtains the center of gravity 2A on the feature space based on each feature point classified as the feature point 40A. The classification unit 140d obtains the center of gravity 2B on the feature space based on each feature point classified as the feature point 40B. The classification unit 140d obtains the center of gravity 2C on the feature space based on each feature point classified as the feature point 40C.

  The classification unit 140d obtains covariance based on each feature point classified as the feature point 40A. The classification unit 140d obtains covariance based on each feature point classified as the feature point 40B. The classification unit 140d obtains covariance based on each feature point classified as the feature point 40C. The classification unit 140d may obtain a standard deviation instead of the covariance.

  The classification unit 140d registers the result of the above processing in the registration table 130c. FIG. 9 is a diagram illustrating an example of the data structure of the registration table according to the first embodiment. As shown in FIG. 9, the registration table 130c has a cluster number, coordinates, barycentric coordinates, and covariance.

  The cluster number in FIG. 9 is a number that uniquely identifies the classified feature point group. For example, the cluster number of the feature point group classified as the feature point 40A is 1. The feature point group classified into the feature points 40B is set as cluster number 2. The feature point classified as the feature point 40C is set as cluster number 3.

  The coordinates in FIG. 9 are the coordinates of the feature points extracted from the learning target image data 130a. For example, the coordinates (x1, y1) in FIG. 9 are the coordinates of the feature point 40A. The barycentric coordinates in FIG. 9 correspond to the barycentric coordinates shown in FIG. For example, barycentric coordinates (xg1, yg1, zg1) are coordinates on the feature space of each feature point corresponding to cluster number 1. The covariance is a covariance for each classified feature point group. For example, covariance “v1” is the covariance of each feature point corresponding to cluster number 1.

  Note that the classification unit 140d may exclude the feature point group from the classification target when the number of classified feature point groups is smaller than a predetermined number. For example, when the number of feature points classified into the feature points 40C is smaller than a predetermined number, the classification unit excludes each feature point classified as the feature point 40A from the registration table 130c.

  The association unit 140e is a processing unit that associates the feature points of the recognition target image data 130d with the feature points of the learning target image data 130a based on the registration table 130c.

  The associating unit 140e extracts feature points from the recognition target image data 130d using SIFT. The associating unit 140e compares the coordinates on the feature space of the feature points extracted from the recognition target image data 130d with the barycentric coordinates of the registration table 130c, and determines the cluster number corresponding to the barycentric coordinates with the shortest distance between the coordinates. judge. The associating unit 140e associates the coordinates of the determined cluster number with the feature points of the recognition target image data 130d.

  For example, the associating unit 140e determines the cluster number 1 if the distance between the feature point coordinates and the cluster number 1 centroid coordinates of the cluster numbers 1 to 3 is the shortest. In this case, the associating unit 140e associates the feature points of the recognition target image data with the coordinates (x1, y1) of the cluster number 1. The coordinates (x1, y1) of cluster number 1 are the coordinates of the feature point 40A in FIG.

  The associating unit 140e performs the above process on each feature point extracted from the recognition target image data 130d, and outputs the execution result to the output unit 120. When the association unit 140e executes such processing, the feature point of the recognition target image data 130d and the feature point of the learning target image data 130a are associated with each other.

  The association unit 140e may associate the feature points of the recognition target image data 130d and the feature points of the learning target image data 130a using the Mahalanobis distance. For example, the associating unit 140e sets a weight corresponding to covariance. The association unit 140e sets a smaller value as the weight as the covariance value is larger. For example, the weights of the covariances v1, v2, and v3 are g1, g2, and g3. When the magnitude relationship of each covariance is v1> v2> v3, the weight is g1 <g2 <g3.

  The associating unit 140e has a value G1 obtained by multiplying the distance between the coordinates of the cluster number 1 and the coordinates of the feature point by g1, a value G2 obtained by multiplying the distance between the coordinates of the cluster number 2 and the coordinates of the feature point by g2, A value G3 obtained by multiplying the distance between the coordinates of the cluster number 3 and the coordinates of the feature points by g3 is calculated. Then, the associating unit 140e determines the cluster number corresponding to the smallest value among the values G1 to G3. The associating unit 140e associates the coordinates of the determined cluster number with the feature points of the recognition target image data 130d. When the Mahalanobis distance is used, a cluster number having a larger covariance value is more easily associated.

  Next, the learning phase processing procedure and the recognition phase processing procedure of the image processing apparatus 100 according to the first embodiment will be described in order. FIG. 10 is a flowchart illustrating the processing procedure of the learning phase according to the first embodiment. The process illustrated in FIG. 10 is executed when the learning target image data 130a is stored in the storage unit 130, for example.

  As illustrated in FIG. 10, the image processing apparatus 100 acquires learning target image data 130a (step S101), and generates a viewpoint variation image data group 130b (step S102). The image processing apparatus 100 extracts feature points from the learning target image data 130a (step S103).

  The image processing apparatus 100 extracts feature points from each viewpoint variation image data (step S104). The image processing apparatus 100 converts the coordinates of the feature points extracted from each viewpoint variation image data into the coordinate system of the learning target image data (step S105).

  The image processing apparatus 100 classifies each feature point (step S106) and registers various information in the registration table 130c (step S107).

  FIG. 11 is a flowchart illustrating the processing procedure of the recognition phase according to the first embodiment. The process illustrated in FIG. 11 is executed, for example, when the recognition target image data 130d is stored in the storage unit 130.

  The image processing apparatus 100 acquires the recognition target image data 130d (step S201), and extracts feature points from the recognition target image data 130d (step S202). The image processing apparatus 100 associates the feature points based on the feature points of the recognition target image data 130d and the registration table 130c (step S203).

  Next, effects of the image processing apparatus 100 according to the first embodiment will be described. The image processing apparatus 100 extracts feature points from the learning target image data 130a and the viewpoint variation image data group 130d having different camera viewpoints, classifies each feature point, and generates a registration table 130c. The image processing apparatus 100 compares the barycentric coordinates of each feature point group in the registration table 130c with the coordinates of the feature points extracted from the recognition target image data 130b, and the feature points of the learning target image data 130a and the recognition target image data. 130d feature points are associated. For this reason, according to the image processing apparatus 100, each feature point can be associated by comparing the barycentric coordinates of each feature point group with the coordinates of the feature points extracted from the recognition target image data 130d. It is possible to reduce the amount of calculation when associating.

  Further, since the image processing apparatus 100 associates each feature point using the Mahalanobis distance, it is possible to associate the coordinates of each image accurately in consideration of the degree of dispersion of each feature point.

  An image processing apparatus according to the second embodiment will be described. FIG. 12 is a functional block diagram of the configuration of the image processing apparatus according to the second embodiment. As shown in FIG. 12, the image processing apparatus 200 includes a camera 210a, an input unit 210b, an output unit 220, a storage unit 230, and a control unit 240.

  The description regarding the camera 210a, the input unit 210b, and the output unit 220 is the same as the description regarding the camera 110a, the input unit 110b, and the output unit 120 illustrated in FIG.

  The storage unit 230 stores learning target image data 230a, a viewpoint variation image data group 230b, a registration table 230c, and recognition target image data 230d. The storage unit 230 corresponds to, for example, a semiconductor memory device such as a RAM, a ROM, or a flash memory, or a storage device such as a hard disk or an optical disk.

  The learning target image data 230a corresponds to the learning target image data 130a of the first embodiment. The viewpoint variation image data group 230b corresponds to the viewpoint variation image data group 130b of the first embodiment. The recognition target image data 230d corresponds to the recognition target image data 130d of the first embodiment.

  The registration table 230c is a table that holds various types of information regarding feature points extracted from the learning target image data 230a and the viewpoint variation image data group 230b. The data structure of the registration table 230c will be described later.

  The control unit 240 includes a data management unit 240a, a viewpoint variation image generation unit 240b, a feature point extraction unit 240c, a classification unit 240d, and an association unit 240e. The control unit 240 corresponds to, for example, an integrated device such as an ASIC or FPGA. Moreover, the control part 240 respond | corresponds to electronic circuits, such as CPU and MPU, for example.

  The data management unit 240 a is a processing unit that manages the storage unit 230. For example, in the learning phase, the data management unit 240a acquires learning target image data from the camera 210a, and stores the acquired learning target image data in the storage unit 230. In the recognition phase, the data management unit 240a acquires recognition target image data from the camera 210a, and stores the acquired recognition target image data in the storage unit 230.

  The viewpoint variation image generation unit 240b is a processing unit that generates a viewpoint variation image data group 230b based on the learning target image data 230a. Specific processing of the viewpoint variation image generation unit 240b is the same as the processing of the viewpoint variation image generation unit 140b of the first embodiment.

  The feature point extraction unit 240c extracts feature points from the learning target image data 230a. In addition, the feature point extraction unit 240c extracts feature points from each viewpoint variation image data of the viewpoint variation image data group 230b. The feature point extraction unit 240c outputs data relating to feature points extracted from the learning target image data 230a and data relating to feature points extracted from the viewpoint variation image data group 230b to the classification unit 240d.

  The specific processing of the feature point extraction unit 240c is the same as the processing of the feature point extraction unit 140c of the first embodiment. The feature point extraction unit 240c outputs the extraction result 1A shown in FIG. 4 and the extraction result 1B shown in FIG. 6 to the classification unit 240d.

  The classification unit 240d is a processing unit that classifies a plurality of feature points extracted from the viewpoint variation image data group 230b. The classification unit 240d classifies the feature points based on the extraction result 1A and the extraction result 1B.

  In particular, the classification unit 240d according to the second embodiment further classifies the feature points classified by the classification unit 140d according to the first embodiment. The processing until the classification unit 240d classifies each feature point extracted from the viewpoint variation image data into one of the feature point 40A, the feature point 40B, and the feature point 40C is the same as that of the classification unit 140d of the first embodiment. .

  FIG. 13 is a diagram for explaining the processing of the classification unit according to the second embodiment. Here, the case where the classification unit 240d further classifies each feature point classified into the feature points 40A will be described as an example. The classification unit 240d classifies the feature points into groups 3A to 3C by applying k-means to each feature point. The classification unit 240d obtains centroids k1-1 to k3-1 of each feature point for each group. Each center of gravity is expressed as a key point.

  Here, an example of processing by k-means used by the classification unit 240d will be described. The classification unit 240d assigns clusters to each feature point at random, and calculates the center of gravity for each cluster. The classification unit 240d changes the cluster of each feature point to the cluster with the nearest center of gravity. The separation unit 240d classifies the feature points into a plurality of groups by repeating the above processing until the cluster of each feature point does not change.

  The classification unit 240d further classifies the feature points classified into the feature points 40B and the feature points classified into the feature points 40C in the same manner as the feature points classified into the feature points 40A. The classification unit 240d registers the classification result in the registration table 230c.

  The classification unit 240d calculates a covariance value for each group based on the feature points included in the group further classified by k-means, and registers the covariance value in the registration table 230c.

  FIG. 14 is a diagram illustrating an example of a data structure of a registration table according to the second embodiment. As shown in FIG. 14, this registration table 230c has cluster numbers, coordinates, key numbers, barycentric coordinates, and covariance.

  The cluster number in FIG. 14 is a number that uniquely identifies the classified feature point group. For example, the cluster number of the feature point group classified as the feature point 40A is 1. The feature point group classified into the feature points 40B is set as cluster number 2. The feature point classified as the feature point 40C is set as cluster number 3.

  The coordinates in FIG. 14 are the coordinates of feature points extracted from the learning target image data 230a. For example, the coordinates (x1, y1) in FIG. 14 are the coordinates of the feature point 40A. The key number is a number that uniquely identifies a key point. The barycentric coordinates correspond to the barycentric coordinates of the key points. The covariance is a covariance value of feature points included in a group further classified by k-means.

  The association unit 240e is a processing unit that associates the feature points of the recognition target image data 230d with the feature points of the learning target image data 230a based on the registration table 230c.

  The associating unit 240e extracts feature points from the recognition target image data 230d using SIFT. The associating unit 240e compares the coordinates on the feature space of the feature points extracted from the recognition target image data 230d with the centroid coordinates of the registration table 230c, and determines the cluster number corresponding to the centroid coordinates with the shortest distance between the coordinates. judge. The association unit 240e associates the coordinates of the determined cluster number with the feature points of the recognition target image data 230d.

  The associating unit 240e determines that the feature point corresponds to the cluster number 1 when the distance between any one of the barycentric coordinates of the key numbers k1-1 to k1-3 and the feature point is the shortest. The associating unit 240e determines that the feature point corresponds to the cluster number 2 when the distance between any one of the barycentric coordinates of the key numbers k2-1 to k2-3 and the feature point is the shortest. The associating unit 240e determines that the feature point corresponds to cluster number 3 when the distance between any one of the barycentric coordinates of the key numbers k3-1 to k3-3 and the feature point is the shortest.

  The associating unit 240e performs the above processing on each feature point extracted from the recognition target image data 230d, and outputs the execution result to the output unit 220. As the association unit 240e executes such processing, the feature points of the recognition target image data 230d are associated with the feature points of the learning target image data 230a.

  Note that the association unit 240e may associate the feature points of the recognition target image data 230d and the feature points of the learning target image data 230a using the Mahalanobis distance in the same manner as in the first embodiment.

  Next, effects of the image processing apparatus 200 according to the second embodiment will be described. The image processing apparatus 200 further classifies the feature point group classified in the first embodiment using k-means. For this reason, even if the feature points are distorted and distributed in the feature space or separated, the feature points of the recognition target image data 230d and the learning target image data 230a are accurately detected. Feature points can be associated with each other.

  An image processing apparatus according to the third embodiment will be described. FIG. 15 is a functional block diagram of the configuration of the image processing apparatus according to the third embodiment. As illustrated in FIG. 15, the image processing apparatus 300 includes a camera 310 a, an input unit 310 b, an output unit 320, a storage unit 330, and a control unit 340.

  The description regarding the camera 310a, the input unit 310b, and the output unit 320 is the same as the description regarding the camera 110a, the input unit 110b, and the output unit 120 illustrated in FIG.

  The storage unit 330 stores learning target image data 330a, a viewpoint variation image data group 330b, a registration table 330c, and recognition target image data 330d. The storage unit 330 corresponds to, for example, a semiconductor memory device such as a RAM, a ROM, or a flash memory, or a storage device such as a hard disk or an optical disk.

  The learning target image data 330a corresponds to the learning target image data 130a of the first embodiment. The viewpoint variation image data group 330b corresponds to the viewpoint variation image data group 130b of the first embodiment. The recognition target image data 330d corresponds to the recognition target image data 130d of the first embodiment.

  The registration table 330c is a table that holds various types of information regarding feature points extracted from the learning target image data 330a and the viewpoint variation image data group 330b. The data structure of the registration table 330c will be described later.

  The control unit 340 includes a data management unit 340a, a viewpoint variation image generation unit 340b, a feature point extraction unit 340c, a classification unit 340d, and an association unit 340e. The control unit 340 corresponds to an integrated device such as an ASIC or FPGA, for example. The control unit 340 corresponds to, for example, an electronic circuit such as a CPU or MPU.

  The data management unit 340 a is a processing unit that manages the storage unit 330. For example, in the learning phase, the data management unit 340a acquires learning target image data from the camera 310a, and stores the acquired learning target image data in the storage unit 330. In the recognition phase, the data management unit 340a acquires recognition target image data from the camera 310a, and stores the acquired recognition target image data in the storage unit 330.

  The viewpoint variation image generation unit 340b is a processing unit that generates a viewpoint variation image data group 330b based on the learning target image data 330a. Specific processing of the viewpoint variation image generation unit 340b is the same as the processing of the viewpoint variation image generation unit 140b of the first embodiment.

  The feature point extraction unit 340c extracts feature points from the learning target image data 330a. In addition, the feature point extraction unit 340c extracts feature points from each viewpoint variation image data of the viewpoint variation image data group 330b. The feature point extraction unit 340c outputs the data on the feature points extracted from the learning target image data 330a and the data on the feature points extracted from the viewpoint variation image data group 330b to the classification unit 340d.

  The specific process of the feature point extraction unit 340c is the same as the process of the feature point extraction unit 140c of the first embodiment. The feature point extraction unit 340c outputs the extraction result 1A shown in FIG. 4 and the extraction result 1B shown in FIG. 6 to the classification unit 340d.

  The classification unit 340d is a processing unit that classifies a plurality of feature points extracted from the viewpoint variation image data group 330b. The classification unit 340d classifies the feature points based on the extraction result 1A and the extraction result 1B.

  In particular, the classification unit 340d according to the third embodiment compares the centroid coordinates of the feature point groups classified by the classification unit 140d according to the first embodiment, summarizes the feature point groups having the centroid coordinates close to each other, and collects the feature point groups. , K-means is applied to reclassify the feature points with similar barycentric coordinates. The processing until the classification unit 340d classifies each feature point extracted from the viewpoint variation image data into one of the feature point 40A, the feature point 40B, and the feature point 40C is the same as that of the classification unit 140d of the first embodiment. .

  FIGS. 16 and 17 are diagrams for explaining the processing of the classification unit according to the third embodiment. In FIG. 16, the circles correspond to the feature points classified as the feature points 40A. The triangle mark corresponds to the feature point classified as the feature point 40B. The square marks correspond to the feature points classified as the feature points 40C.

  The classification unit 340d compares the distances of the centroids 2A to 2C of the feature point groups, and determines a set of centroids in which the distance between the centroids is less than a threshold. In the third embodiment, as an example, a case where the distance between the center of gravity 2A and the center of gravity 2B is less than the threshold will be described.

  The classification unit 340d performs k-means on the feature point group corresponding to the determined centroid 2A and the feature point group corresponding to the centroid 2B. Here, the feature point group corresponding to the center of gravity 2A is each of the feature points indicated by circles classified as the feature point 40A. The feature point group corresponding to the center of gravity 2B is each feature point of the triangle mark classified into the feature points 40B.

  When the classification unit 340d executes k-means, the feature points are classified into groups 4A to 4E as shown in FIG. The classification unit 340d obtains centroids k1 to k5 of each feature point for each group. Each center of gravity is expressed as a key point. Further, the classification unit 340d calculates the covariance of each feature point for each group.

  Referring to FIG. 17, the groups 4A and 4B include only the circle feature points. For this reason, the classification unit 340d associates the groups 4A and 4B with the feature point 40A. The cluster number of group 4A is 1, and the cluster number of group 4B is 2.

  Referring to FIG. 17, group 4C includes feature points with circles and feature points with triangles. In this case, the classification unit 340d associates the group 4C with both the feature point 40A and the feature point 40B. The cluster number of group 4C is set to 3.

  Referring to FIG. 17, the groups 4D and 4E include only triangular feature points. For this reason, the classification unit 340d associates the groups 4D and 4E with the feature point 40B. The cluster number of group 4D is 4, and the cluster number of group 4E is 5.

  The classification unit 340d sets the cluster number of each feature point of the square mark to 6 and the center of gravity to k6. k6 corresponds to 2C in FIG.

  The classification unit 340d registers the classification result in the registration table 330c. FIG. 18 is a diagram illustrating an example of the data structure of the registration table according to the third embodiment. The registration table 330c has a cluster number, a key number, the number of configuration data, coordinates, barycentric coordinates, and covariance.

  The cluster numbers in FIG. 18 are numbers that uniquely identify the classified feature points. Cluster numbers 1 and 2 are cluster numbers of the feature point group classified into the feature points 40A. Cluster number 3 is a cluster of feature points grouped as feature point 40A or feature point 40B as described above. Cluster numbers 4 and 5 are cluster numbers of the feature point group classified into the feature points 40B. The cluster number 6 is the cluster number of the feature point group classified into the feature points 40C.

  The key number in FIG. 18 is a number that uniquely identifies the key point of each feature point group. The number of configuration data corresponds to the number of feature points included in each group. When different types of feature points are included in the same group, the number of feature points for each type is registered. For example, since the feature point group corresponding to the cluster number 3 includes a feature point with a circle and a feature point with a triangle, a number is registered for each type of feature point.

  The coordinates in FIG. 18 are the coordinates of the feature points extracted from the learning target image data 330a. For example, the coordinates (x1, y1) are the coordinates of the feature point 40A. The coordinates (x2, y2) are the coordinates of the feature point 40B. The coordinates (x3, y3) are the coordinates of the feature point 40C. The barycentric coordinates correspond to the barycentric coordinates of the key points. The covariance is a covariance value of feature points included in a group further classified by k-means.

  The association unit 340e is a processing unit that associates the feature points of the recognition target image data 330d with the feature points of the learning target image data 330a based on the registration table 330c.

  The associating unit 340e extracts feature points from the recognition target image data 330d using SIFT. The associating unit 340e compares the coordinates on the feature space of the feature points extracted from the recognition target image data 330d with the centroid coordinates of the registration table 330c, and determines the cluster number corresponding to the centroid coordinates with the shortest distance between the coordinates. judge. The associating unit 340e associates the coordinates of the determined cluster number with the feature points of the recognition target image data 330d.

  The associating unit 340e determines that the feature point corresponds to the feature point 40A on the learning target image data 330a when either of the barycentric coordinates of the key numbers k1 and k2 is closest to the feature point. . The associating unit 340e determines that the feature point corresponds to the feature point 40B on the learning target image data 330a when either of the barycentric coordinates of the key numbers k4 and k5 is closest to the feature point. . The associating unit 340e determines that the feature point corresponds to the feature point 40C on the learning target image data 330a when any one of the barycentric coordinates of the key number k6 is closest to the feature point.

  Note that when the distance between the barycentric coordinate of the key number k3 and the feature point is the closest, the associating unit 340e is either the feature point 40A or the feature point 40B on the learning target image data 330a. It is determined that it corresponds to. When the feature points of the recognition target image data 330d are associated with a plurality of feature points on the learning target image data 330a, the associating unit 340e finally determines the feature points with the least error as the entire feature points. Associate with.

  For example, the associating unit 340e compares the error in the case of associating with the feature point 40A with the error in the case of associating with the feature point 40B based on the least square method or the like, and the feature with the smaller error The feature points of the recognition target image data 330d are associated with the points.

  Note that the association unit 340e may associate the feature points of the recognition target image data 330d and the feature points of the learning target image data 330a using the Mahalanobis distance in the same manner as in the first embodiment.

  Next, effects of the image processing apparatus 300 according to the third embodiment will be described. When there is a possibility of corresponding to a plurality of feature points, the image processing apparatus 300 associates the feature points of the recognition target image data 330d with the plurality of feature points of the learning target image data 330a. Then, the image processing apparatus 300 narrows down feature points associated with a plurality of feature points to a single feature point based on the relationship with the entire feature points. Therefore, the feature points of the recognition target image data 330d are not forced to be associated with a single feature point, and the feature points of the recognition target image data 330d are accurately used as a plurality of feature points of the learning target image data 330a. Can be associated.

  Next, an example of a computer that executes an information processing program that implements the same functions as those of the information processing apparatuses 100, 200, and 300 described in the embodiments will be described. FIG. 19 is a diagram illustrating an example of a computer that executes an image processing program.

  As shown in FIG. 19, the computer 400 includes a CPU 401 that executes various arithmetic processes, an input device 402 that receives data input from a user, and a display 403. The computer 400 includes a reading device 404 that reads a program and the like from a storage medium, and an interface device 405 that exchanges data with another computer via a network. The computer 400 also includes a RAM 406 that temporarily stores various types of information and a hard disk device 407. The devices 401 to 407 are connected to the bus 408.

  The hard disk device 407 includes, for example, a viewpoint variation image generation program 407a, a feature point extraction program 407b, a classification program 407c, and an association program 407d. The CPU 401 reads each program 407 a to 407 d and develops it in the RAM 406.

  The viewpoint variation image generation program 407a functions as a viewpoint variation image generation process 406a. The feature point extraction program 407b functions as a feature point extraction process 406b. The classification program 407c functions as a classification process 406c. The association program 407d functions as the association process 406d.

  For example, the viewpoint variation image generation process 406a corresponds to the viewpoint variation image generation units 140b, 240b, and 340b. The feature point extraction process 406b corresponds to the feature point extraction units 140c, 240c, and 340c. The classification process 406c corresponds to the classification units 140d, 240d, and 340d. The association process 406d corresponds to the association units 140e, 240e, and 340e.

  Note that the programs 407a to 407d are not necessarily stored in the hard disk device 407 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, and an IC card inserted into the computer 400. Then, the computer 400 may read and execute each of the programs 407a to 407d from these.

  The following supplementary notes are further disclosed with respect to the embodiments including the above examples.

(Supplementary note 1)
By projectively converting a learning target image obtained by photographing the image from the front camera viewpoint, a plurality of viewpoint variation images having different camera viewpoints with respect to the learning target image are generated,
Extracting feature points from the learning object image and the plurality of viewpoint variation images;
By associating each feature point extracted from the plurality of viewpoint variation images with each feature point extracted from the learning target image, each feature point extracted from the plurality of viewpoint variation images is converted into a plurality of feature point groups. Classify and
Register the region containing the classified feature point group in the registration table,
Obtaining a recognition target image, extracting feature points from the recognition target image,
Each process of associating the feature point of the recognition target image with the feature point of the learning target image based on the relationship between the position of the feature point extracted from the recognition target image and the region of the feature point group of the registration table An image processing program for executing

(Supplementary Note 2) The process of associating the feature point of the recognition target image with the feature point of the learning target image is based on the distance between the centroid of the feature point group included in the region and the feature point of the recognition target image. The image processing program according to appendix 1, wherein a feature point of the recognition target image and a feature point of the learning target image are associated with each other.

(Supplementary note 3) The process of registering in the registration table further includes registering the distribution spread of each feature point included in the feature point group, and associating the feature point of the recognition target image with the feature point of the learning target image , Based on the distance between the centroid of the feature point group included in the region and the feature point of the recognition target image weighted with the distribution spread, the feature point of the recognition target image and the feature point of the learning target image, The image processing program according to appendix 1 or 2, characterized in that:

(Supplementary note 4) The image processing program according to supplementary note 3, wherein the distribution spread is a standard deviation or a variance.

(Additional remark 5) The process classified into the said feature point group is classified into any one of additional marks 1-4 characterized by classifying into one feature point of the above-mentioned learning object image into a plurality of feature point groups. The image processing program described.

(Additional remark 6) The process classified into the said feature point group classify | categorizes into one feature point group with respect to the some feature point of the said learning object image to any one of the additional notes 1-4 characterized by the above-mentioned. The image processing program described.

(Appendix 7) An image processing method executed by a computer,
By projectively converting a learning target image obtained by photographing the image from the front camera viewpoint, a plurality of viewpoint variation images having different camera viewpoints with respect to the learning target image are generated,
Extracting feature points from the learning object image and the plurality of viewpoint variation images;
By associating each feature point extracted from the plurality of viewpoint variation images with each feature point extracted from the learning target image, each feature point extracted from the plurality of viewpoint variation images is converted into a plurality of feature point groups. Classify and
Register the region containing the classified feature point group in the registration table,
Obtaining a recognition target image, extracting feature points from the recognition target image,
Each process of associating the feature point of the recognition target image with the feature point of the learning target image based on the relationship between the position of the feature point extracted from the recognition target image and the region of the feature point group of the registration table The image processing method characterized by performing.

(Supplementary Note 8) The process of associating the feature point of the recognition target image with the feature point of the learning target image is based on the distance between the centroid of the feature point group included in the region and the feature point of the recognition target image. The image processing method according to appendix 7, wherein a feature point of the recognition target image is associated with a feature point of the learning target image.

(Supplementary Note 9) The process of registering in the registration table further registers the distribution spread of each feature point included in the feature point group, and associates the feature point of the recognition target image with the feature point of the learning target image. , Based on the distance between the centroid of the feature point group included in the region and the feature point of the recognition target image weighted with the distribution spread, the feature point of the recognition target image and the feature point of the learning target image, The image processing method according to appendix 7 or 8, characterized in that

(Supplementary note 10) The image processing method according to supplementary note 9, wherein the distribution spread is a standard deviation or a variance.

(Additional remark 11) The process classified into the said feature point group is classified into any one of Additional remarks 7-10 characterized by classifying into one feature point of the said learning object image to several feature point groups. The image processing method as described.

(Additional remark 12) The process classified into the said feature point group classify | categorizes into one feature point group with respect to the some feature point of the said learning object image to any one of additional marks 7-10 characterized by the above-mentioned. The image processing method as described.

(Supplementary Note 13) A viewpoint variation image generation unit that generates a plurality of viewpoint variation images having different camera viewpoints with respect to the learning target image by projective transformation of the learning target image obtained by photographing the image from the front camera viewpoint;
A feature point extraction unit that extracts feature points from the learning target image and the plurality of viewpoint variation images;
By associating each feature point extracted from the plurality of viewpoint variation images with each feature point extracted from the learning target image, each feature point extracted from the plurality of viewpoint variation images is converted into a plurality of feature point groups. A classification unit that classifies and registers a region including the classified feature point group in a registration table;
A recognition target image is acquired, feature points are extracted from the recognition target image, and the recognition is performed based on a relationship between the position of the feature point extracted from the recognition target image and the region of the feature point group in the registration table. An image processing apparatus comprising: an association unit that associates a feature point of a target image with a feature point of the learning target image.

(Additional remark 14) The said matching part is based on the distance of the gravity center of the feature point group contained in the said area | region, and the feature point of the said recognition target image, The feature point of the said recognition target image, and the feature of the said learning target image 14. The image processing apparatus according to appendix 13, wherein points are associated with each other.

(Additional remark 15) The said classification | category part further registers the distribution spread of each feature point contained in the feature point group of the said registration table, The said matching part is the gravity center of the feature point group contained in the said area | region, and the said distribution 15. The image according to appendix 13 or 14, wherein the feature point of the recognition target image is associated with the feature point of the learning target image based on a distance from the feature point of the recognition target image that is weighted for spread. Processing equipment.

(Supplementary note 16) The image processing apparatus according to supplementary note 15, wherein the distribution spread is a standard deviation or a variance.

(Supplementary note 17) The image processing device according to any one of supplementary notes 13 to 16, wherein the classification unit classifies one feature point of the learning target image into a plurality of feature point groups. .

(Supplementary note 18) The image processing device according to any one of supplementary notes 13 to 16, wherein the classification unit classifies a plurality of feature points of the learning target image into one feature point group. .

DESCRIPTION OF SYMBOLS 100 Image processing apparatus 110a Camera 110b Input part 120 Output part 130 Storage part 140 Control part

Claims (8)

On the computer,
By projectively converting a learning target image obtained by photographing the image from the front camera viewpoint, a plurality of viewpoint variation images having different camera viewpoints with respect to the learning target image are generated,
Extracting feature points from the learning object image and the plurality of viewpoint variation images;
By associating each feature point extracted from the plurality of viewpoint variation images with each feature point extracted from the learning target image, each feature point extracted from the plurality of viewpoint variation images is converted into a plurality of feature point groups. Classify and
Register the region containing the classified feature point group in the registration table,
Obtaining a recognition target image, extracting feature points from the recognition target image,
Each process of associating the feature point of the recognition target image with the feature point of the learning target image based on the relationship between the position of the feature point extracted from the recognition target image and the region of the feature point group of the registration table An image processing program for executing   The process of associating the feature point of the recognition target image with the feature point of the learning target image is based on the distance between the centroid of the feature point group included in the region and the feature point of the recognition target image. The image processing program according to claim 1, wherein a feature point of the image is associated with a feature point of the learning target image.   The process of registering in the registration table further registers the distribution spread of each feature point included in the feature point group, and the process of associating the feature point of the recognition target image with the feature point of the learning target image is performed on the region. Associating the feature point of the recognition target image with the feature point of the learning target image based on the distance between the center of gravity of the included feature point group and the feature point of the recognition target image weighted with the distribution spread. The image processing program according to claim 1 or 2, characterized in that   The image processing program according to claim 3, wherein the distribution spread is a standard deviation or a variance.   5. The image according to claim 1, wherein the process of classifying into feature point groups classifies one feature point of the learning target image into a plurality of feature point groups. Processing program.   The image according to any one of claims 1 to 4, wherein the process of classifying into the feature point group classifies a plurality of feature points of the learning target image into one feature point group. Processing program. An image processing method executed by a computer,
By projectively converting a learning target image obtained by photographing the image from the front camera viewpoint, a plurality of viewpoint variation images having different camera viewpoints with respect to the learning target image are generated,
Extracting feature points from the learning object image and the plurality of viewpoint variation images;
By associating each feature point extracted from the plurality of viewpoint variation images with each feature point extracted from the learning target image, each feature point extracted from the plurality of viewpoint variation images is converted into a plurality of feature point groups. Classify and
Register the region containing the classified feature point group in the registration table,
Obtaining a recognition target image, extracting feature points from the recognition target image,
Each process of associating the feature point of the recognition target image with the feature point of the learning target image based on the relationship between the position of the feature point extracted from the recognition target image and the region of the feature point group of the registration table The image processing method characterized by performing. A viewpoint variation image generating unit that generates a plurality of viewpoint variation images having different camera viewpoints with respect to the learning target image by projective transformation of the learning target image obtained by capturing the image from the front camera viewpoint;
A feature point extraction unit that extracts feature points from the learning target image and the plurality of viewpoint variation images;
By associating each feature point extracted from the plurality of viewpoint variation images with each feature point extracted from the learning target image, each feature point extracted from the plurality of viewpoint variation images is converted into a plurality of feature point groups. A classification unit that classifies and registers a region including the classified feature point group in a registration table;
A recognition target image is acquired, feature points are extracted from the recognition target image, and the recognition is performed based on a relationship between the position of the feature point extracted from the recognition target image and the region of the feature point group in the registration table. An image processing apparatus comprising: an association unit that associates a feature point of a target image with a feature point of the learning target image.

Images