JP4873258B2 - Information processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an information processing apparatus, method, and program, and more particularly, to an information processing apparatus, method, and program that can recognize an object in an image more reliably.

  Conventionally, there is a texture-based general object recognition method using local features (see Patent Document 1). Such a method is resistant to changes in illumination and can recognize an object robustly, but has a feature that the discrimination ability is lowered when applied to an object with less texture.

In addition, the specification attached to the application already filed as Japanese Patent Application No. 2006-100705 by the applicant at the time of the filing of the present application uses the edge information and the support points, so that a local object with respect to an object having no texture is used. A technique that enables the application of a feature matching technique is disclosed. In other words, this method is to extract feature points from the model image and query image, describe the local feature values around them, match the feature values, and use outliers (mismatch) using Hough transform, RANSAC, etc. This is a technique of identifying an object in the model image and an object in the query image by the number of matching pairs after the removal.
JP 2004-326693 A

  However, these conventional methods have the following three problems. As a result, it is expected to realize a method that can recognize an object in an image more reliably than these conventional methods.

  That is, the first problem is that when the appearance reproducibility of the feature point position of the model image and the feature point position of the query image is poor, the discrimination ability is remarkably lowered. The first problem is that when edges are used, the reproducibility of the model image edges and the query image edges greatly affects the discrimination ability.

  The second problem is that the identification of the model is finally determined by the number of match pairs of the inlier (after mismatched pair removal), so regardless of the similarity between the object in the model image and the object in the query image, The problem is that there are a large number of match pairs between objects that have many feature points due to complex textures and contours, and objects that have simple textures and shapes tend to have fewer match pairs.

  The third problem is that when support points are provided around the base points and used for improving the accuracy of matching, the support point selection criteria do not take into account differences between multiple model images. is there.

  The present invention has been made in view of such a situation, and makes it possible to more reliably recognize an object in an image.

  An information processing apparatus according to one aspect of the present invention is an information processing apparatus that compares a query image and a model image and identifies a subject of the model image and a subject of the query image, and includes N ( N is an integer value of 1 or more), and when the feature values of the extracted N feature points are respectively described, the N feature points and information indicating the feature values are included. A model dictionary to be registered exists inside or outside, and a correlation image between the corresponding feature quantity and the query image for each of the N feature points of the model image registered in the model dictionary For each of the N correlation images generated by the correlation image generation unit and the correlation image generation unit for generating the corresponding feature points in the model image, By shifting the pixel position of the pixel, a shift correlation image generation unit that generates N shift correlation images, and a pixel value of each pixel of the N shift correlation images generated by the shift correlation image generation unit Based on the correlation sum image generation unit that generates a correlation sum image by addition and the correlation sum image generated by the correlation sum image generation unit, the subject of the model image matches the subject of the query image Determination means for determining whether or not to do.

  For each of the one or more feature points of the model image registered in the model dictionary, the described feature quantity of the model image, the model image from which the model image is extracted, and one or more different model images When correlation images are respectively generated and, based on the correlation images, a discrimination capability value indicating a degree of contribution for identifying the subject of the model image is calculated, the discrimination capability values also correspond to the corresponding feature The shift correlation image generation means is further registered in the model dictionary together with information indicating points, and the shift correlation image generation means further converts the pixel value of each pixel of the N shift correlation images to the identification capability value registered in the model dictionary. N weighted shift correlation images are generated by weighting in response, and the correlation sum image generation means generates the shift correlation image generation means. By adding the pixel value of each pixel of the number of weighted shift correlation image, to generate the correlation sum image.

  At least one of the one or more feature points of the model image registered in the model dictionary is a base point, and one or more support points from among the feature points existing within a certain range of the base point Is selected, information indicating their base points and support points is also registered in the model dictionary, and the correlation image generation means further includes the N images of the model images registered in the model dictionary. For each feature point, the mb support point correlation images of the feature amount of the corresponding mb support points (mb is an integer value of 0 or more) and the query image are generated, and the mb support points are generated. For each of the point correlation images, by shifting the pixel position of each pixel according to the arrangement position of the corresponding support point and the base point in the model image, the mb support images are supported. Point correlation image, and for each of the N correlation images, the sum of the pixel values of each pixel of itself and the mb support point shift correlation images is generated to generate N correlation image sums. Then, the shift correlation image generation unit generates the N shift correlation images from each of the N correlation image sums generated by the correlation image generation unit.

  For each of the one or more feature points of the model image registered in the model dictionary, the described feature quantity of the model image, the model image from which the model image is extracted, and one or more different model images When correlation images are respectively generated and, based on the correlation images, a discrimination capability value indicating a degree of contribution for identifying the subject of the model image is calculated, the discrimination capability values also correspond to the corresponding feature It is further registered in the model dictionary together with information indicating a point, and at least one of the one or more feature points of the model image registered in the model dictionary is set as a base point, and within a certain range of the base point If the feature point having a higher discrimination ability value than the base point is selected as a support point from the existing feature points, those base points and support points are selected. Information indicating points is also registered in the model dictionary, and the correlation image generation means further supports corresponding mb pieces for each of the N feature points of the model image registered in the model dictionary. Mb support point correlation images of the feature amount of the points and the query image are generated, respectively, and the corresponding support points and base points in the model image for each of the mb support point correlation images. Mb support point shift correlation images are generated by shifting the pixel position of each pixel in accordance with the arrangement position of the N and the mb support point shift correlations for each of the N correlation images. By adding the pixel values of each pixel of the image, N correlation image sums are generated, and the shift correlation image generation unit generates the previous correlation image generation unit generated by the correlation image generation unit. N weighted shift correlation images are generated by weighting the pixel value of each pixel of the N correlation image sums according to the discrimination ability value registered in the model dictionary, and the correlation sum image The generation unit generates the correlation sum image by adding pixel values of each pixel of the N weighted shift correlation images generated by the shift correlation image generation unit.

  An information processing method and program according to one aspect of the present invention are a method and program corresponding to the information processing apparatus according to one aspect of the present invention described above.

  In one aspect of the present invention, the following processing is executed as information processing for comparing the query image and the model image and identifying the subject of the model image and the subject of the query image. However, when N feature points (N is an integer value of 1 or more) are extracted from the model image and the feature values of the extracted N feature points are respectively described, the N feature points and There is a model dictionary in which information indicating these feature quantities is registered. In this case, for each of the N feature points of the model image registered in the model dictionary, a correlation image between the corresponding feature amount and the query image is generated, and the generated N images are generated. For each of the correlation images, N shift correlation images are generated by shifting the pixel position of each pixel in accordance with the arrangement position of the corresponding feature point in the model image, and the generated N A correlation sum image is generated by adding the pixel values of each pixel of the number of shift correlation images, and the subject of the model image matches the subject of the query image based on the generated correlation sum image. It is determined whether or not.

  As described above, according to one aspect of the present invention, an object in an image can be recognized. In particular, according to one aspect of the present invention, an object in an image can be recognized more reliably.

  Embodiments of the present invention will be described below. Correspondences between the constituent elements of the present invention and the embodiments described in the specification or the drawings are exemplified as follows. This description is intended to confirm that the embodiments supporting the present invention are described in the specification or the drawings. Therefore, even if there is an embodiment that is described in the specification or the drawings but is not described here as an embodiment that corresponds to the constituent elements of the present invention, that is not the case. It does not mean that the form does not correspond to the constituent requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

An information processing apparatus according to one aspect of the present invention includes:
Information for comparing a query image (for example, query image 22 in FIG. 12A) and a model image (for example, model image 21 in FIG. 12B) to identify the subject of the model image and the subject of the query image In the processing device (for example, the object recognition device in FIG. 1 having the query image recognition unit 13 in FIG. 11),
When N feature points are extracted from the model image (N is an integer value equal to or greater than 1), and the feature amounts of the extracted N feature points are respectively described, the N feature points and their feature points are described. A model dictionary (for example, the model feature dictionary 12 in FIG. 11) in which information indicating the feature is registered exists inside or outside itself,
Correlation image generation means for generating a correlation image between the corresponding feature quantity and the query image for each of the N feature points of the model image registered in the model dictionary (for example, the correlation in FIG. 11). An image generator 52);
For each of the N correlation images generated by the correlation image generation unit, the pixel position of each pixel is shifted in accordance with the arrangement position of the corresponding feature point in the model image, so that N Shift correlation image generation means (for example, shift correlation image generation unit 53 in FIG. 11) for generating a shift correlation image;
Correlation sum image generation means for generating a correlation sum image by adding pixel values of each pixel of the N shift correlation images generated by the shift correlation image generation means (for example, the correlation image sum generation unit in FIG. 11) 54)
Determination means for determining whether or not the subject of the model image and the subject of the query image match based on the correlation sum image generated by the correlation sum image generation means (for example, the determination unit 55 in FIG. 11). And.

For each of the one or more feature points of the model image registered in the model dictionary, the described feature quantity of the model image, the model image from which the model image is extracted, and one or more different model images When correlation images are respectively generated and, based on the correlation images, a discrimination capability value indicating a degree of contribution for identifying the subject of the model image is calculated, the discrimination capability values also correspond to the corresponding feature It is further registered in the model dictionary along with information indicating points,
The shift correlation image generation means further weights the pixel values of each pixel of the N shift correlation images according to the identification capability value registered in the model dictionary, thereby giving N weighted values. A shift correlation image is generated (for example, step S142 in FIG. 14 is executed),
The correlation sum image generation means generates the correlation sum image by adding pixel values of each pixel of the N weighted shift correlation images generated by the shift correlation image generation means.

At least one of the one or more feature points of the model image registered in the model dictionary is a base point, and one or more support points from among the feature points existing within a certain range of the base point Is selected, information indicating their base points and support points is also registered in the model dictionary,
The correlation image generation means further includes:
For each of the N feature points of the model image registered in the model dictionary, the feature amount of the corresponding mb support points and the mb support point correlation images of the query image are generated. Then, for each of the mb support point correlation images, mb support is provided by shifting the pixel position of each pixel according to the arrangement position of the corresponding support point and the base point in the model image. A point shift correlation image is generated (for example, S151-2 in FIG. 15 is executed)
For each of the N correlation images, N correlation image sums are generated by adding the pixel values of each pixel of itself and the mb support point shift correlation images (for example, S151-3 in FIG. 15). Run)
The shift correlation image generation means includes
The N shift correlation images are generated from each of the N correlation image sums generated by the correlation image generation means (for example, S152 in FIG. 15 is executed).

For each of the one or more feature points of the model image registered in the model dictionary, the described feature quantity of the model image, the model image from which the model image is extracted, and one or more different model images When correlation images are respectively generated and, based on the correlation images, a discrimination capability value indicating a degree of contribution for identifying the subject of the model image is calculated, the discrimination capability values also correspond to the corresponding feature It is further registered in the model dictionary along with information indicating points,
At least one of the one or more feature points of the model image registered in the model dictionary is set as a base point, and the discrimination ability value is selected from the feature points existing within a certain range of the base point. When the feature point higher than the base point is selected as a support point, information indicating the base point and the support point is also registered in the model dictionary,
The correlation image generation means further includes:
For each of the N feature points of the model image registered in the model dictionary, the feature amount of the corresponding mb support points and the mb support point correlation images of the query image are generated. Then, for each of the mb support point correlation images, mb support is provided by shifting the pixel position of each pixel according to the arrangement position of the corresponding support point and the base point in the model image. A point shift correlation image is generated (for example, S161-2 in FIG. 16 is executed)
For each of the N correlation images, N correlation image sums are generated by adding the pixel values of each pixel of itself and the mb support point shift correlation images (for example, S161-3 in FIG. 16). Run)
The shift correlation image generation means includes
N weighted shifts are performed by weighting the pixel value of each pixel of the N correlated image sums generated by the correlated image generating means in accordance with the discrimination ability value registered in the model dictionary. Generate a correlation image (for example, execute S162 of FIG. 16),
The correlation sum image generation means generates the correlation sum image by adding pixel values of each pixel of the N weighted shift correlation images generated by the shift correlation image generation means (for example, FIG. 16). Step S163 is executed).

  An information processing method and program according to one aspect of the present invention are a method and program corresponding to the information processing apparatus according to one aspect of the present invention described above. Although details will be described later, this program is recorded on, for example, a removable medium 211 in FIG. 17 or a recording medium such as a hard disk included in the storage unit 208, and is executed by the computer having the configuration in FIG.

  In addition, as one aspect of the present invention, a recording medium on which the program according to one aspect of the present invention described above is recorded is also included.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram showing a functional configuration of an object recognition apparatus according to an embodiment of the present invention.

  In FIG. 1, the object recognition apparatus includes a model feature quantity extraction unit 11, a model feature quantity dictionary 12, and a query image recognition unit 13.

  In the recognition of an object, the model feature amount extraction unit 11 extracts model feature amounts from model images 21-1 to 21-N (N is an integer value of 1 or more) each including an object to be recognized, It is registered in the model feature dictionary 12.

  The model images 21-1 to 21-N are still images themselves or frame images of moving images.

  The query image recognition unit 13 is compared with each object included in the model images 21-1 to 21 -N, extracts a query feature amount from the query image 22 including the recognized object, and registers it in the model feature amount dictionary 12. Matching is performed with the model feature quantity, and identification of each object in the model images 21-1 to 21-N and an object in the query image 22 is attempted based on the result of the matching.

  The query image 22 is a still image itself or a frame image of a moving image, like the model images 21-1 to 21-N.

  Hereinafter, the details of the model feature quantity extraction unit 11 and the query image recognition unit 13 will be described individually in that order.

  Hereinafter, when it is not necessary to distinguish the model images 21-1 to 21-N individually, in other words, when referring to one of the model images 21-1 to 21-N, the model images 21 and 21-N are simply referred to as the model image 21. Called.

  FIG. 2 is a block diagram showing a detailed configuration of the function of the model feature quantity extraction unit 11.

  The model feature amount extraction unit 11 is configured to include a feature point extraction unit 31, a feature amount description unit 32, a feature point identification capability value calculation unit 33, a support point selection unit 34, and a model feature amount information generation unit 35. The

  The feature point extraction unit 31 extracts feature points from the model image 21 and provides the extraction result to the feature amount description unit 32 and the model feature amount information generation unit 35.

  Note that the feature point extraction method itself employed by the feature point extraction unit 31 is not particularly limited.

Specifically, for example, FIG. 3 shows a feature point extraction result when a feature point extraction method using a Harris corner detector or the like is employed. The circles (white circles) in FIG. 3 indicate feature points. In this method, as shown in FIG. 3, corner points are extracted as feature points.

Further, for example, FIG. 4 shows a feature point extraction result when a feature point extraction method using a Canny edge detector or the like is employed. The circles (white circles) in FIG. 4 indicate feature points. In this method, as shown in FIG. 4, edge points are extracted as feature points.

  The feature quantity description unit 32 performs a process of describing a local feature quantity around each feature point extracted by the feature point extraction unit 31, and each process result is used as a feature point identification capability value calculation unit 33 and a model feature quantity. Provided to the information generation unit 35.

  Note that the local feature description method itself employed by the feature description unit 32 is not particularly limited.

  For example, it is possible to employ a method of performing a vector description of local feature amounts using a luminance gradient of pixel values.

  Specifically, for example, as shown in FIG. 5, when describing the luminance gradient in the range of 5 × 5 pixels around the feature point as a vector, the x component and the y component of the luminance gradient of each pixel are respectively dimensioned. (Vx (0,0), Vy (0,0), Vx (0,1), Vy (0,1), ... Vx (4,4), Vy (4,4)) constitutes a 50-dimensional vector Can be employed.

  As another method, for example, a description method using a histogram for each direction of the luminance gradient vector can be employed. For example, when the direction of the starting gradient vector around the feature point is taken every 10 degrees, a 36-dimensional vector is obtained.

  Further, for example, a method of directly using luminance information as a feature amount can be employed. For example, when the luminance information is described as a vector in the range of 5 × 5 pixels around the feature amount, it becomes a 25-dimensional vector.

  Furthermore, the various description methods described above may be combined.

  The feature point identification capability value calculation unit 33 calculates a discrimination capability value for each feature point (each feature point described by the feature amount description unit 32) extracted by the feature point extraction unit 31, Each calculation result is provided to the support point selection unit 34 and the model feature amount information generation unit 35.

  Here, the discriminating ability value is a characteristic point when the subject included in the model image 21, that is, the recognition target object is distinguished from other objects (such as objects included in other model images). A value indicating the capability (model identification capability) of the feature point for identifying the subject such as how much it contributes to the identification, that is, how much the feature point affects the identification. Say.

  FIG. 6 is a flowchart for explaining a series of processes until the discrimination ability value is calculated.

  In the following, in conjunction with the description of FIG. 6, processing in the case where the discrimination ability value for each feature point extracted from the model image 21-1 is calculated will be described. However, in practice, not only the model image 21-1, but also each feature point extracted from the other model images 21-2 to 21-N is subjected to the same processing as described below. Discrimination ability values are respectively calculated.

  In step S100 of FIG. 6, the model feature amount extraction unit 11 acquires all model images 21-1 to 21-N.

  In step S102, the feature point extraction unit 31 extracts one or more feature points from the model image 21-1, as described above. In step S103, as described above, the feature quantity description unit 32 performs feature quantity description for each feature point extracted from the model image 21-1.

In parallel with the processing of steps S102 and S103, in step S104, the feature point identification capability value calculation unit 33 extracts the feature amount images 41-1 to 41- from the model images 21-1 to 21-N, respectively. N is generated respectively.

  Here, the feature amount image 41-K (K is an integer value of 1 to N) is a local amount adopted by the feature amount description unit 32 for all the pixels of the model image 21-K. When feature amount descriptions are performed according to the same method as the feature amount description method, the description result, that is, an image configured with each feature amount as each pixel value.

In step S105, the feature point identification capability value calculation unit 33 out of each feature point of the model image 21-1 (each feature point extracted in the process of step S102 and described in the feature amount in the process of step S103). A correlation image is generated for each of P feature points (P is an integer value equal to or smaller than the number extracted in the process of step S102) for which the discrimination ability is to be calculated.

  Here, the correlation image 42-KL (K is the same value as K of the above-described feature amount image 41-K. L is any value from 1 to P) refers to the following image. . That is, the number of 1 to P is given to P feature points for which the discrimination ability is to be calculated. And the feature point of the number L which should be noted as the object of the process is called the feature point L of interest. In this case, when the feature amount of the target feature point L is matched with each pixel value (that is, each feature amount) constituting the feature amount image 41-K, and a correlation (distance) value is obtained, respectively. In addition, an image configured with each of the correlation values as a pixel value is a correlation image 42-KL. At this time, as the correlation value, for example, a normalized correlation between vectors can be used, and a scale such as a Eugrid distance can be used as the distance 0.

  That is, N correlation images 42-1L indicating the correlation with each pixel of the N feature amount images 41-1, 41-2,. 42-2L,..., 42-NL are generated.

  In other words, for one feature value image 41-K, one correlation image for each of P feature points numbered 1 to P, that is, P correlation images 42-K1. , 42-K2,..., 42-KP are generated.

  In step S106, the feature point discriminating ability value calculation unit 33 calculates the discriminating ability value from the average or maximum value of all correlation images for each of the P feature points numbered 1 to P. In other words, the feature point identification capability value calculation unit 33 gives the identification capability value as the model identification is higher in order from the lowest average or maximum value. The all correlation images are all correlation images generated for the target feature point L, that is, N correlation images 42-1L, 42-2L,..., 42-NL.

  For example, FIGS. 7 and 8 show an image of the discrimination ability value. Here, the feature points with higher discrimination ability values are brighter (whiter). That is, FIG. 7 shows an example of the discrimination ability value when an image including an object having a frog shape (hereinafter abbreviated as a frog) is the model image 21-1. As shown in FIG. 7, it can be seen that the vicinity of the frog eye has a high discrimination ability value, that is, an important part for identifying that it is a frog. On the other hand, FIG. 8 shows an example of the discrimination ability value when an image including an object having a dog shape (hereinafter abbreviated as a dog) is a model image 21-1. As shown in FIG. 8, it can be seen that the vicinity of the tail of the dog is an important part for identifying the dog having a high discrimination capability value, that is, a dog.

  Although not shown, the feature point identification capability value calculation unit 33 executes, for example, a process of rearranging the numbers of the P feature points in descending order of the identification capability value after the process of step S106 in FIG. May be. That is, the numbers of the P feature points after the processing indicate the order important for model identification.

  Returning to FIG. 2, the support point selection unit 34 selects a support point using the discrimination capability value calculated by the feature point discrimination capability value calculation unit 33.

  Here, the support points refer to the following points. That is, the point selected as a reference point from the feature points extracted by the feature point extraction unit 31 is hereinafter referred to as a base point. In this case, a feature point other than the base point that is determined depending on the base point is referred to as a support point.

  The support point determination method itself is not particularly limited. For example, in the present embodiment, in the model image 21, among the feature points within a certain range from the base point arrangement position, the discrimination ability value is higher than the base point. Assume that a method of selecting a feature point having a value as a support point is employed. When such a method is adopted, a plurality of support points may be selected for one base point. FIG. 9 is a flowchart for explaining an example of processing of the support point selection unit 34 (hereinafter referred to as support point selection processing) according to such a method.

  In step S <b> 121 of FIG. 9, the support point selection unit 34 acquires the identification capability value of each of the P feature points in the model image 21.

  In step S122, the support point selection unit 34 selects one or more base points from the P feature points. The base point selection method itself is not particularly limited.

  In step S123, the support point selection unit 34 sets a predetermined one of the one or more base points as a processing target, and extracts other feature points within a certain range from the position of the base point to be processed.

  In step S124, the support point selection unit 34 determines whether or not the extracted feature point identification capability value is higher than the base point identification capability value.

  Here, in the process of step S123, one feature point may not be extracted. In such a case, it is forcibly determined as NO in the process of step S124, and the process proceeds to step S126. The processing after step S126 will be described later.

  Conversely, a plurality of feature points may be extracted in the process of step S123. In such a case, it is determined whether or not a predetermined one of the plurality of feature points is a processing target in step S124, and the discrimination capability value of the processing target feature point is higher than the discrimination capability value of the base point. .

  If the support point selection unit 34 determines in step S124 that the discriminating ability value of the extracted feature point is higher than the discriminating ability value of the base point, in step S125, the support point selecting unit 34 extracts the extracted feature points (a plurality of feature points are extracted. If so, the feature point to be processed is selected as a support point. Thereby, the process proceeds to step S126.

  On the other hand, if it is determined in step S124 that the discriminating capability value of the extracted feature point is lower than the discriminating capability value of the base point, the process of step S125 is not executed, that is, the extracted feature point (If a plurality of feature points are extracted, the feature point to be processed) is not selected as a support point, and the process proceeds to step S126.

  In step S126, the support point selection unit 34 determines whether there are other extracted feature points.

  That is, as described above, when a plurality of feature points are extracted in the process of step S123, it is determined YES in the process of step S126, the process returns to step S124, and the subsequent processes are repeated. It is. That is, each of the plurality of feature points is sequentially processed, and the loop processing of steps S124, S125, and S126 is repeatedly executed. As a result, among the plurality of feature points, only a feature point having a higher discrimination ability value than the base point is selected as a support point. When the above loop processing is executed for all of the plurality of feature points, it is determined NO in step S126 of the last loop processing, and the processing proceeds to step S127.

  Further, when only one feature point is extracted in the process of step S123 or when one feature point is not extracted, it is immediately determined as NO in the process of step S126, and the process proceeds to step S127.

  In step S127, the support point selection unit 34 determines whether there is another base point.

  If there is still a base point that is not subject to processing, it is determined YES in step S127, the process returns to step S123, and the subsequent processing is repeated.

  In this manner, when 0 or more support points are selected for each of one or more base points, it is determined NO in the process of step S127, and the support point selection process ends.

  Specifically, for example, in each of FIGS. 10A to 10C, the selection result of the base point and the support point is shown. That is, three base points are selected from the same model image 21, and each base point is shown as a circle (white circle) in FIGS. 10A to 10C. A plurality of support points selected for these three base points are respectively shown in FIGS. 10A to 10C as ● (black circles) smaller than ○ (white circles) indicating the base points. ing.

  Returning to FIG. 2, the model feature amount information generation unit 35 generates model feature amount information (base point + support point) indicating various processing results of the feature point extraction unit 31 to the support point selection unit 34 described above. Register in the feature dictionary 12. That is, the model feature amount information is information related to each feature point extracted for each of the model images 21-1 to 21-N. Specifically, for example, these feature points are classified into base points and support points, and information including local feature amounts, discriminating ability values, support point information, and the like is model feature amounts. Information.

  The details of the model feature quantity extraction unit 11 in the object recognition apparatus of FIG. 1 have been described above. Details of the query image recognition unit 13 will be described below.

  FIG. 11 is a block diagram illustrating a detailed configuration of functions of the query image recognition unit 13.

  The query image recognition unit 13 is configured to include a feature image generation unit 51, a correlation image generation unit 52, a shift correlation image generation unit 53, a correlation image sum generation unit 54, and a determination unit 55.

  When a query image 22 including an object to be recognized is input, the feature image generation unit 51 generates a feature amount image from the query image 22. That is, the processing similar to step S104 in FIG. 6 described above is performed on the query image 22.

  The correlation image generation unit 52 stores each pixel value of the feature amount image of the query image 22 (that is, the feature amount of each pixel) and each of the model images 21-1 to 21-N recorded in the model feature amount dictionary 12. Matching with feature quantities of feature points (hereinafter, referred to as model feature points) is performed, and an image in which each correlation (distance) value is configured as each pixel value, ie, a correlation image is generated.

  The shift correlation image generation unit 53 generates an image (hereinafter referred to as a shift correlation image) obtained by shifting each pixel position of the corresponding correlation image according to the position of each model feature point. A method for generating a shift correlation image will be described later with reference to FIGS.

  The correlation image sum generation unit 54 calculates, for each model image 21-1 to 21-N, each shift correlation image of each model feature point, or the sum of each image after various image processing is performed on them. The image (hereinafter referred to as a correlation sum image) is generated. That is, the correlation sum image is an image configured by using the sum of the pixel values of two or more images as the respective pixel values.

  A specific example of the correlation sum image generation method (including examples of various image processes performed on the shift correlation image) will be described later with reference to FIGS.

  The determination unit 55 includes the objects included in the model images 21-1 to 21-N in the query image 22 based on the correlation sum images generated for the model images 21-1 to 21-N. It is determined whether or not the object is the same as the detected object, and the determination result is output.

  That is, among the correlation sum images for the predetermined model image 21-K, the pixel value at the shift position (position near the center in the example described later) at the time of generation of the shift correlation image becomes the local peak of the correlation sum image. . Then, the local peak represents a presence estimation degree indicating how much an object included in the model image 21 -K is present in the query image 22. Therefore, the determination unit 55 determines that the object included in the model image 21-K matches the image included in the query image 22 when the local peak of the correlation sum image of the model image 21-K is equal to or greater than the threshold. The object can be recognized.

  Hereinafter, the operations of the correlation image generation unit 52 to the correlation image sum generation unit 54 among the operations of the query image recognition unit 13 will be described with reference to FIGS.

  That is, FIGS. 13 to 16 show various processing results until the correlation sum image with the model image 21 of FIG. 12B is generated when the image of FIG. 12A is input as the query image 22. A specific example is shown.

  In the example of FIG. 13, as the feature amount information of the model image 21, only the feature amounts of the four base points b1 to b4 are used to generate a correlation sum image. In other words, in the example of FIG. 13, the support point information and the identification capability value are not used at all as in other examples described later. Note that the base points b1 to b4 are merely examples, and it goes without saying that the positions and the number of base points are not limited to the example of FIG. 13 and are arbitrary.

  In step S131 of FIG. 13, the correlation image generation unit 52 determines each pixel value of the feature amount image of the query image 22 (that is, a feature amount of each pixel) and each feature amount of the base points b1 to b4 of the model image 21. By performing this matching, four correlation images as shown in the frame of S131 of FIG. 13 are generated.

  In step S132, the shift correlation image generation unit 53 shifts each pixel position of the corresponding correlation image in accordance with the positions of the base points b1 to b4, as shown in the frame of S132 in FIG. Four shift correlation images are generated.

  The shift correlation image in the example of FIG. 13 is the presence position (correlation image) of the base point bn (n is an integer value, and n is any one of 1 to 4 in the example of FIG. 13). Is obtained as a result of shifting each pixel position of the correlation image so that the corresponding pixel position is shifted to the center position of the image.

  In step S133, the correlation image sum generation unit 54 simply adds these four shift correlation images to generate a correlation sum image as shown in the frame of S133 in FIG. Note that “adding” means adding the pixel values for each pixel as described above. The same applies to the following description.

  In contrast to the example of FIG. 13, in the example of FIG. 14, as the feature amount information of the model image 21, in addition to the feature amounts of the four base points b1 to b4, the weight value α1 based on their discrimination ability values A correlation sum image is generated by using .alpha.4.

  That is, in step S <b> 141, the correlation image generation unit 52 calculates each pixel value (that is, a feature amount of each pixel) of the feature amount image of the query image 22 and each feature amount of the base points b <b> 1 to b <b> 4 of the model image 21. By performing matching, four correlation images as shown in the frame of S141 in FIG. 14 are generated.

  Note that the four correlation images shown in the frame of S141 in FIG. 14 are the same as the four correlation images shown in the frame of S131 in FIG. That is, the process of step S141 and the process of step S131 are the same process.

  In the process of step S142, a shift correlation image generation process is executed. However, the process of step S142 is different from the process of step S132 of FIG.

  That is, in step S142-1, the shift correlation image generation unit 53 shifts each pixel position of the corresponding correlation image in accordance with the position of each base point b1 to b4, so that the process of S142-1 in FIG. Four shift correlation images as shown in the dotted frame are generated.

  Note that the four shift correlation images shown in the dotted frame in S142-1 in FIG. 14 are the same as the four shift correlation images shown in the frame in S132 in FIG. That is, the process of step S142-1 is the same process as the process of step S132 of FIG.

  In other words, the process of step S142 is a process in which the following process of step S142-2 is further added to the process of step S132 (= step S142-1) of FIG. In order to distinguish the shift correlation image finally obtained by the process of step S142-2 from the shift correlation image obtained as a result of the process of step S142-1, hereinafter, the former will be referred to as a weighted shift correlation image. The latter is referred to as a simple shift correlation image.

  That is, in the process of step S142-1, four simple shift correlation images shown in the dotted line frame of S142-1 in FIG. 14 are generated. Therefore, in step S142-2, the shift correlation image generation unit 53 determines the discrimination capability value of each base point b1 to b4 for each pixel value of each simple shift correlation image corresponding to each of the base points b1 to b4. Are multiplied by the weight values α1 to α4 based on the image, respectively, and an image composed of each pixel value weighted according to the discriminating ability value, that is, 4 as shown in the dotted line frame of S142-2 in FIG. One weighted shift correlation image is generated.

  In step S143, the correlation image sum generation unit 54 generates a correlation sum image as shown in the frame of S143 in FIG. 14 by simply adding these four weighted shift correlation images.

  In contrast to the examples of FIGS. 13 and 14, in the example of FIG. 15, in addition to the feature amounts of the four base points b1 to b4, the base points b1 to b4 are added as the feature amount information of the model image 21. Correlation sum images are generated using the information of the support points. However, in the example of FIG. 15, the weight values α1 to α4 based on the discrimination ability value are not used as in the example of FIG.

  In the process of step S151, a correlation image generation process is executed. However, the process of step S151 is different from the process of step S131 of FIG. 13 and step S141 of FIG.

  That is, in step S152-1, the correlation image generation unit 52 determines each pixel value of the feature amount image of the query image 22 (ie, the feature amount of each pixel) and each feature amount of the base points b1 to b4 of the model image 21. Are generated, four correlation images as shown in the frame of S151-1 in FIG. 15 are generated.

  Note that the four correlation images shown in the frame of S151-1 in FIG. 15 are the same as the four correlation images shown in the frame of S131 in FIG. 13, that is, 4 shown in the frame of S141 in FIG. Identical to two correlation images. That is, the process of step S151-1 is the same process as the process of step S131 of FIG. 13 or step S141 of FIG.

  In other words, in addition to the processing of step S131 in FIG. 13 (= step S141 in FIG. 14 = step S151-1 in FIG. 15), the processing in step S151 further includes the following steps S151-2 and S151-3. This process is added. Hereinafter, the correlation image obtained as a result of the process of step S151-1 is referred to as a base point correlation image in order to individually distinguish the correlation images obtained as a result of the processes of steps S151-1 to S151-3. The correlation image obtained as a result of the process of S151-2 is referred to as a support point shift correlation image, and the correlation image obtained as a result of the process of step S151-3 is referred to as a support point shift correlation image sum centered on the base point bn.

  That is, in the process of step S151-1, four base point correlation images shown in the dotted line frame of S151-1 of FIG. 15 are generated.

In step S151-2, the correlation image generation unit 52 determines each pixel value of the feature amount image of the query image 22 (that is, the feature amount of each pixel) and the support point at the base point bn for the base point bn of the model image 21. By performing matching with each feature quantity of snm (m is an integer value of 1 or more), m correlation images are generated. Furthermore, the correlation image generation unit 52 shifts the presence position of the support point snm (corresponding pixel position of the correlation image) to the presence position of the base point bn (corresponding pixel position of the correlation image), thereby causing the base points b1 to b4 to be shifted. For each of these, m support point shift correlation images as shown in the frame of S151-2 of FIG. 15 are generated.

  That is, there are two support points s11 and s12 at the base point b1. Therefore, a support point shift correlation image for the support point s11 and a support point shift correlation image for the support point s12 are generated.

  Similarly, there are three support points s21, s22, and s23 at the base point b2. Therefore, a support point shift correlation image for the support point s21, a support point shift correlation image for the support point s22, and a support point shift correlation image for the support point s23 are generated.

  There are two support points s31 and s32 at the base point b3. Therefore, a support point shift correlation image for the support point s31 and a support point shift correlation image for the support point s32 are generated.

  There is one support point s41 at the base point b4. Therefore, a support point shift correlation image for the support point s41 is generated.

  In step S151-3, the correlation image generation unit 52 performs the corresponding base point correlation image (the image obtained as a result of the processing in step S151-1) and the corresponding m support points for the base point bn of the model image 21. By simply adding the shift correlation image (the image obtained as a result of the processing in step S151-2), the support point shift correlation centered on the base point bn as shown in the frame of S151-3 in FIG. Generate an image sum.

  That is, for the base point b1, the base point correlation image for the base point b1, and the support point shift correlation image for the support point s11 and the support point shift correlation image for the support point s12 are added together. A support point shift correlation image sum centered on b1 is generated.

  Similarly, for the base point b2, the base point correlation image for the base point b2, the support point shift correlation image for the support point s21, the support point shift correlation image for the support point s22, and the support point s23. These support point shift correlation images are added together, and as a result, a support point shift correlation image sum centered on the base point b2 is generated.

  For the base point b3, the base point correlation image for the base point b3, and the support point shift correlation image for the support point s31 and the support point shift correlation image for the support point s32 are added together. A centered support point shift correlation image sum is generated.

  For the base point b4, the base point correlation image for the base point b4 and the support point shift correlation image for the support point s41 are added, and as a result, a support point shift correlation image sum centered on the base point b4 is generated. Is done.

  Subsequent processes in steps S152 and S153 are basically the same as the processes in steps S132 and S133 in FIG. However, the processing target in step S132 in FIG. 13 is the base point correlation image that is the processing result in step S151-1 in FIG. On the other hand, as described above, the processing target of step S152 in FIG. 15 is the support point that is the processing result of step S151-2 with respect to the base point correlation image that is the processing result of step S151-1 of FIG. This is an image obtained as a result of adding the shift correlation images, that is, a support point shift correlation image sum centered on the base point.

  The example of FIG. 16 is an example in which the example of FIG. 14 and the example of FIG. 15 are combined. That is, in the example of FIG. 16, in addition to the feature amounts of the four base points b1 to b4, as the feature amount information of the model image 21, weight values α1 to α4 based on their discrimination ability values and the base points b1 to b4 A correlation sum image is generated using both of the support point information of b4.

  In other words, the process in step S161 in FIG. 16 is the same as the process in step S151 in FIG. That is, each of steps S161-1 to S161-3 in FIG. 16 is the same processing as each of steps S151-1 to S151-3 in FIG.

  On the other hand, the process in step S162 in FIG. 16 is the same as the process in step S142 in FIG. That is, each of steps S162-1, S162-2 in FIG. 16 is the same processing as each of steps S141-1, S141-2 in FIG.

  Expressed as an expression, the processing result of step S161 in FIG. 16 is expressed as the following expression (1).

・・・（１）

... (1)

SumSpCor _bn (x, y) on the left side of Expression (1) indicates the pixel value at the coordinates (x, y) of the support point shift correlation image sum centered on the base point bn. Note that n is any value from 1 to 4 in the example of FIG. 16, but it goes without saying that it can be generalized to an arbitrary integer value.

In addition, on the right side of Equation (1), Cor _snm (x, y) indicates the pixel value in (x, y) of the correlation image at the support point snm. m _bn indicates the number of support points at the base point bn. That is, in the example of FIG. 16, m _b1 = 2, m _b2 = 3, m _b3 = 2 and m _b4 = 1. (bx _n , by _n ) indicates the coordinates of the base point bn. (snx _m , sny _m ) indicates the coordinates of the support point snm.

  Then, the final processing result of step S163 in FIG. 16 is expressed as the following equation (2). That is, the expression in Σ on the right side of Expression (2) indicates the processing result of Step S162 in FIG.

・・・（２）

... (2)

  SumCor (x, y) on the left side of Expression (2) indicates the pixel value at the coordinates (x, y) of the correlation sum image obtained as a result of the processing in step S163.

  Further, on the right side of Expression (2), (cx, cy) indicates the center coordinates of the model image 21.

  As described above, by applying the present invention, it is not necessary to consider the repeatability of model image and query image feature point extraction, and more robust recognition is possible.

  In addition, the predetermined pixel value of the correlation image sum (for example, the pixel value near the center), that is, the sum of the correlation values represents the existence estimation degree of the object. You can see if it exists with the probability of.

  Also, considering the correlation with other parts of your model image and other model images, you can calculate the feature value discrimination ability value, and you can also select the support point based on the discrimination ability value, Matching accuracy is improved.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 17 is a block diagram showing an example of the configuration of a personal computer that executes the above-described series of processing by a program. A CPU (Central Processing Unit) 201 executes various processes according to a program stored in a ROM (Read Only Memory) 202 or a storage unit 208. A RAM (Random Access Memory) 203 appropriately stores programs executed by the CPU 201 and data. The CPU 201, ROM 202, and RAM 203 are connected to each other via a bus 204.

  An input / output interface 205 is also connected to the CPU 201 via the bus 204. Connected to the input / output interface 205 are an input unit 206 made up of a keyboard, mouse, microphone, and the like, and an output unit 207 made up of a display, speakers, and the like. The CPU 201 executes various processes in response to commands input from the input unit 206. Then, the CPU 201 outputs the processing result to the output unit 207.

  The storage unit 208 connected to the input / output interface 205 includes, for example, a hard disk, and stores programs executed by the CPU 201 and various data. The communication unit 209 communicates with an external device via a network such as the Internet or a local area network.

  Further, a program may be acquired via the communication unit 209 and stored in the storage unit 208.

  The drive 210 connected to the input / output interface 205 drives a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the program or data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 208 as necessary.

  As shown in FIG. 17, a program recording medium that stores a program that is installed in a computer and can be executed by the computer is a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read). Only memory), DVD (Digital Versatile Disc), removable media 211, which is a package medium composed of a magneto-optical disk, semiconductor memory, or the like, or ROM 202 where a program is temporarily or permanently stored, or a storage unit It is constituted by a hard disk or the like constituting 208. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 209 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the order described, but is not necessarily performed in time series. Or the process performed separately is also included.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention. For example, the embodiment in which the present invention is applied to an object recognition apparatus has been described above. However, the present invention can be applied to an information processing apparatus that compares and recognizes objects in an image, for example.

It is a block diagram which shows the structure of the function of the object recognition apparatus which is one embodiment of this invention. It is a block diagram which shows the detailed structure of the function of the model feature-value extraction part of FIG. It is a figure which shows the specific example of the process result of the feature point extraction part of FIG. It is a figure which shows the specific example of the process result of the feature point extraction part of FIG. It is a figure explaining the example of the processing method of the feature-value description part of FIG. It is a flowchart explaining the process example of the feature point identification capability value calculating part of FIG. It is a figure which shows the specific example of the process result of FIG. It is a figure which shows the specific example of the process result of FIG. It is a flowchart explaining the example of a support point selection process by the support point selection part of FIG. It is a figure which shows the specific example of the process result of FIG. It is a block diagram which shows the detailed structure of the function of the query image recognition part of FIG. It is a figure which shows the specific example of the model image and query image for demonstrating the process of the query image recognition part of FIG. It is a figure which shows the specific example of the processing result of the query image recognition part of FIG. It is a figure which shows the specific example of the processing result of the query image recognition part of FIG. It is a figure which shows the specific example of the processing result of the query image recognition part of FIG. It is a figure which shows the specific example of the processing result of the query image recognition part of FIG. And FIG. 11 is a block diagram illustrating an example of a configuration of a personal computer.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 11 Model feature-value extraction part, 12 Model feature-value dictionary, 13 Query image recognition part, 21 Model image 21 Query image, 31 Feature point extraction part, 32 Feature-value description part, 33 Feature point identification capability value calculation part, 34 Support point Selection unit, 51 feature image generation unit, 52 correlation image generation unit, 53 shift correlation image generation unit, 54 correlation image sum generation unit, 55 determination unit, 201 CPU, 202 ROM, 203 RAM, 208 storage unit, 211 removable media

Claims (6)

In an information processing apparatus that compares a query image and a model image and identifies a subject of the model image and a subject of the query image,
When N feature points are extracted from the model image (N is an integer value equal to or greater than 1), and the feature amounts of the extracted N feature points are respectively described, the N feature points and their feature points are described. A model dictionary in which information indicating the feature amount is registered exists inside or outside itself,
Correlation image generation means for generating a correlation image between the corresponding feature amount and the query image for each of the N feature points of the model image registered in the model dictionary;
For each of the N correlation images generated by the correlation image generation unit, the pixel position of each pixel is shifted in accordance with the arrangement position of the corresponding feature point in the model image, so that N Shift correlation image generation means for generating a shift correlation image;
Correlation sum image generation means for generating a correlation sum image by adding pixel values of each pixel of the N shift correlation images generated by the shift correlation image generation means;
An information processing apparatus comprising: a determination unit that determines whether the subject of the model image matches the subject of the query image based on the correlation sum image generated by the correlation sum image generation unit. For each of the one or more feature points of the model image registered in the model dictionary, the described feature quantity of the model image, the model image from which the model image is extracted, and one or more different model images When correlation images are respectively generated and, based on the correlation images, a discrimination capability value indicating a degree of contribution for identifying the subject of the model image is calculated, the discrimination capability values also correspond to the corresponding feature It is further registered in the model dictionary along with information indicating points,
The shift correlation image generation means further weights the pixel values of each pixel of the N shift correlation images according to the identification capability value registered in the model dictionary, thereby giving N weighted values. Generate a shift correlation image,
The correlation sum image generation unit generates the correlation sum image by adding pixel values of pixels of the N weighted shift correlation images generated by the shift correlation image generation unit. The information processing apparatus described. At least one of the one or more feature points of the model image registered in the model dictionary is a base point, and one or more support points from among the feature points existing within a certain range of the base point Is selected, information indicating their base points and support points is also registered in the model dictionary,
The correlation image generation means further includes:
For each of the N feature points of the model image registered in the model dictionary, the feature amount of the corresponding mb support points (mb is an integer value of 0 or more) and the mb of the query image Each of the support point correlation images is generated, and for each of the mb support point correlation images, the pixel position of each pixel is determined according to the arrangement position of the corresponding support point and the base point in the model image. By shifting, mb support point shift correlation images are generated,
For each of the N correlation images, by adding the pixel values of each pixel of itself and the mb support point shift correlation images, N correlation image sums are generated,
The shift correlation image generation means includes
The information processing apparatus according to claim 1, wherein the N shift correlation images are generated from each of the N correlation image sums generated by the correlation image generation unit. For each of the one or more feature points of the model image registered in the model dictionary, the described feature quantity of the model image, the model image from which the model image is extracted, and one or more different model images When correlation images are respectively generated and, based on the correlation images, a discrimination capability value indicating a degree of contribution for identifying the subject of the model image is calculated, the discrimination capability values also correspond to the corresponding feature It is further registered in the model dictionary along with information indicating points,
At least one of the one or more feature points of the model image registered in the model dictionary is set as a base point, and the discrimination ability value is selected from the feature points existing within a certain range of the base point. When the feature point higher than the base point is selected as a support point, information indicating the base point and the support point is also registered in the model dictionary,
The correlation image generation means further includes:
For each of the N feature points of the model image registered in the model dictionary, the feature amount of the corresponding mb support points (mb is an integer value of 0 or more) and the mb of the query image Each of the support point correlation images is generated, and for each of the mb support point correlation images, the pixel position of each pixel is determined according to the arrangement position of the corresponding support point and the base point in the model image. By shifting, mb support point shift correlation images are generated,
For each of the N correlation images, by adding the pixel values of each pixel of itself and the mb support point shift correlation images, N correlation image sums are generated,
The shift correlation image generation means includes
N weighted shifts are performed by weighting the pixel value of each pixel of the N correlated image sums generated by the correlated image generating means in accordance with the discrimination ability value registered in the model dictionary. Generate correlation images,
The correlation sum image generation unit generates the correlation sum image by adding pixel values of pixels of the N weighted shift correlation images generated by the shift correlation image generation unit. The information processing apparatus described. In the information processing method of the information processing apparatus for comparing the query image and the model image and identifying the subject of the model image and the subject of the query image,
When N feature points are extracted from the model image (N is an integer value equal to or greater than 1), and the feature amounts of the extracted N feature points are respectively described, the N feature points and their feature points are described. A model dictionary in which information indicating the feature amount is registered exists inside or outside itself,
As the step executed by the information processing apparatus,
For each of the N feature points of the model image registered in the model dictionary, a correlation image between the corresponding feature amount and the query image is generated, respectively.
For each of the generated N correlation images, N shift correlation images are generated by shifting the pixel position of each pixel according to the arrangement position of the corresponding feature point in the model image. ,
A correlation sum image is generated by adding pixel values of each pixel of the generated N shift correlation images,
An information processing method including a step of determining whether or not a subject of the model image matches a subject of the query image based on the generated correlation sum image. A computer that controls an information processing apparatus that compares a query image and a model image and identifies a subject of the model image and a subject of the query image,
When N feature points are extracted from the model image (N is an integer value equal to or greater than 1), and the feature amounts of the extracted N feature points are respectively described, the N feature points and their feature points are described. In a computer in which a model dictionary in which information indicating the feature amount is registered exists inside or outside itself,
For each of the N feature points of the model image registered in the model dictionary, a correlation image between the corresponding feature amount and the query image is generated, respectively.
For each of the generated N correlation images, N shift correlation images are generated by shifting the pixel position of each pixel according to the arrangement position of the corresponding feature point in the model image. ,
A correlation sum image is generated by adding pixel values of each pixel of the generated N shift correlation images,
A program for executing a step of determining whether or not a subject of the model image matches a subject of the query image based on the generated correlation sum image.

Images