JP2012083855A - Object recognition device and object recognition method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To highly precisely recognize a three-dimensional object at a high speed compared with a conventional method.SOLUTION: An object recognition device 1 includes: a database 2 storing storage images, feature points previously extracted form the storage images, and positional relation information of the feature points; storage image specification means 11 that searches corresponding feature points between the feature points of the storage images of the database 2 and feature points extracted from an input image, votes, to the input image, a feature point calculated based on the positional relation information of the feature points of the storage image, determines that a similarity between the input image and the storage image is high as the voting points are concentrated therein and the number of votes increase, and specifies the storage image similar to the input image; and determination means 12 that generates a recall image using the specified storage image, compares the recall image with the input image and, when determining that the both are resembled, determines on the input image that the recognition object is detected.

Description

  The present invention relates to an object recognition apparatus and an object recognition method.

  Object recognition is an indispensable function for realizing the visual ability of a robot, for example, and is one of the important research themes in computer vision (see Non-Patent Document 1). In addition, there is a great deal of research on the classification of general object images under certain conditions, such as airplanes seen from the side and road signs seen from the front, because there is a common data set. It is thriving.

JP 2007-148872 A JP 2007-280040 A

Keiji Yanai, "Present state and future of general object recognition," Information processing theory: Computer vision and image media, vol.48, no.SIG16 (CVIM19), pp.1-24, 2007. M. Sun, H. Su, S. Savarese, and L. Fei-Fei, "A multiview probabilistic model for 3d object classes," IEEE Int. Conf. Comput. Vision and Pattern Recognit., Pp.1247-1254, 2009 .

  However, the method targeting general objects disclosed in Non-Patent Document 1 is still limited to use in an actual environment. One of the reasons is that, although the recognition target is a three-dimensional object, many of the existing studies have limited their recognizable viewpoints. One of the reasons is that it has not been able to cope with changes.

  In recent years, research on tackling the task of three-dimensional object recognition for general objects has been performed on this problem (see, for example, Non-Patent Document 2). However, in the recognition of a three-dimensional object, it is considered that the object has been correctly recognized if the orientation can be estimated. However, even in the conventional research exemplified in Non-Patent Document 2, the evaluation regarding the orientation estimation is regarded as very important. However, there are few studies that have been evaluated with an emphasis on quantitative evaluation.

  In addition, the methods exemplified in Non-Patent Document 2 are all model-based methods that learn the features in consideration of the target geometric structure. In 3D object recognition with large intra-class changes, a model-based method that uses 3D structural information of the target is considered to be a reasonable approach, but the shooting angle information of the target is required during learning, etc. In general, there are disadvantages that the learning cost is high and the description method of the object tends to be complicated. Hereinafter, the problems of the model-based method will be described in more detail.

  First, in order to handle a three-dimensional geometric structure, it is generally necessary to calculate the connection between various feature regions between arbitrary viewpoints and their relevance (projective transformation and the like). For this reason, there is a problem that the method of expressing the target model is likely to be complicated (implementation is likely to be complicated).

  In addition, there is a problem that the learning data creation cost is often increased because it is necessary to add such information to the learning data, for example, the shooting angle information of the learning image is required at the time of learning.

  In 3D object recognition, even if the recognition target is of the same class, the shape and appearance of each individual change greatly due to the diversity of viewpoints. For this reason, there is a problem that recognition often takes time.

  Further, a target is recognized using a feature amount extracted from an image, but since the feature amount extracts only characteristic information of the image, the information amount is smaller than that of the image. For this reason, when recognition is performed under a complicated background (scene), if there are many feature quantities similar to the learned feature quantities in the background, the recognition may be erroneously performed. That is, the ability to identify feature quantities greatly affects the recognition accuracy. On the other hand, when using feature quantities with high discrimination ability, if you want to recognize objects that are not included in the learning image but are included in the same class as the same class, you may not be able to detect them. is there. In addition, although recognition is performed by evaluating the structural connection of feature quantities, similarity as a global appearance is not considered. Therefore, for these reasons, there is a problem that the recognition accuracy is low (the false detection rate is high).

  As another technique related to the present invention, Patent Documents 1 and 2 disclose that personal authentication is performed in response to partial changes in face images that are independent of the passage of time, such as changes in glasses and beards. A face recognition device that can be used is disclosed. In the face authentication device, the authentication target image extracted from the input image is compared with the registered image learned in advance, and if it is determined that they are not the same person as a result of the comparison, a recall image close to the registered image is registered. The image is generated from the image and collated with the registered image again using the recall image instead of the authentication target image. In this way, although the detection rate itself is improved by interpolating partial hiding by glasses or sunglasses using the recall image generated from the registered image, it is difficult to improve the accuracy of the interpolated image, and there is no problem with reducing the false authentication rate. It is enough.

  Accordingly, an object of the present invention is to provide an object recognition apparatus and an object recognition method that can solve the above-described problems and can realize a three-dimensional object recognition that is faster and more accurate than the conventional method.

  The object recognition apparatus according to the first aspect of the present invention includes a plurality of stored images each including a recognition target, a feature point extracted in advance from each of the plurality of stored images, and the extraction with respect to a reference position in the stored image. A storage means for storing the positional relationship information of the feature points, a feature point extracted from the input image, a feature point of the stored image stored in the storage means, and a feature point extracted from the input image; Voting points that are calculated based on the positional relationship information of the feature points of the stored image with respect to the searched corresponding points, and the voting points are concentrated and It is determined that the similarity between the input image and the stored image is higher as the number of votes is larger, and as a result of the determination, a stored image specifying unit that specifies the stored image similar to the input image, and the stored image specifying unit Using the identified stored image to generate a recall image indicating how the input image looks, determine whether the generated recall image and the input image are similar to each other, Determination means for determining that the recognition target is detected in the input image when it is determined that the input object is detected.

  Thereby, it is possible to realize three-dimensional object recognition that is faster and more accurate than the conventional method.

  Further, the determination unit may generate the recall image by combining the plurality of stored images specified by the stored image specifying unit according to the degree of similarity with each of the input images. Good.

  Furthermore, when the determination unit compares the recall image and the input image to determine whether they are similar to each other, the total number of voting points of the input image used to generate the recall image It may be determined that the recall image and the input image are more similar as there are more.

  Further, the storage means further stores the direction information of the recognition target in association with the stored image, and the determination means compares the recall image and the input image to determine whether they are similar to each other. If it is determined that the recognition target is detected in the input image, it is detected based on the orientation information of the stored image used for generating the recall image. The direction of the recognition target may be estimated.

  An object recognition apparatus according to a second aspect of the present invention includes a stored image including a recognition target, a feature point extracted in advance from the stored image, and a positional relationship of the extracted feature point with respect to a reference position in the stored image. A database for storing a plurality of template models for each class model including information, an average scale size of the feature points, and a template size that is a value obtained by normalizing the size of the stored image with the average scale size; and an input image The input image corresponding to the feature point of the stored image is extracted by extracting the feature point from the database and performing matching between the feature point of the stored image stored in the database and the feature point extracted from the input image. A corresponding point search unit that searches for the corresponding feature point as a corresponding point, and the corresponding point of the input image searched by the corresponding point search unit. In the input image voted by the voting processing unit that calculates voting points based on the positional relationship information of the feature points of the stored image stored in the base and votes the voting points in the input image When the voting points are clustered and the number of voting points included in the same cluster is equal to or greater than a predetermined threshold, the voting point clustering unit determines that the recognition target exists around the center of the cluster, and the voting points For the cluster center that the clustering unit determines that the recognition target exists, the recognition target exists in the input image for generating a rectangular area based on the average scale size and template size of the template model corresponding to the cluster center. A candidate frame generation unit that generates a candidate frame indicating a range, and the candidate frame generation unit The generated candidate frame is generated by a recall image generation unit that generates a recall image indicating how the input image is viewed using the stored image of the template model corresponding to the candidate frame, and the recall image generation unit. The degree of difference is calculated by comparing the recalled image with the image of the candidate frame in the input image, and the candidate frame is erroneously detected when the calculated difference is greater than a predetermined threshold. And a reliability value calculation unit that determines that the recognition target is detected in the range of the input image indicated by the candidate frame.

  Thereby, it is possible to realize three-dimensional object recognition that is faster and more accurate than the conventional method.

  Moreover, the said recall image production | generation part is the said input image which acquired the said memory | storage image of the said several template model corresponding to the said candidate frame about each candidate frame produced | generated by the said candidate frame production | generation part from each said template model The recall image may be generated by combining them according to the number of votes.

  Furthermore, the confidence value calculation unit compares the recall image generated by the recall image generation unit with the image of the candidate frame in the input image to calculate a difference, and corresponds to the candidate frame Calculating the total number of votes in the input images obtained from all the template models, and if the confidence value calculated based on the calculated reciprocal of the difference and the total number of votes is smaller than a predetermined threshold, The candidate frame may be removed as erroneous detection, and the candidate frame that has not been removed may be determined to have been detected in the range of the input image indicated by the candidate frame.

  In addition, the template model further includes orientation information indicating the orientation of the recognition target, and when it is determined that the recognition target is detected by the reliability value calculation unit, the detected candidate frame of the recognition target A direction estimating unit that estimates the detected orientation of the recognition target based on the orientation information of the plurality of template models corresponding to the candidate frame may be further provided.

  The object recognition method according to the third aspect of the present invention includes a plurality of stored images each including a recognition target, a feature point extracted in advance from each of the plurality of stored images, and the extraction with respect to a reference position in the stored image. A storage step for preliminarily storing the positional relationship information of the feature points obtained, a feature point extracted from the input image, and a feature point of the storage image stored in the storage step and a feature point extracted from the input image Voting points calculated on the basis of positional relationship information of the feature points of the stored image for the corresponding points searched for, and the voting points are concentrated. And determining that the degree of similarity between the input image and the stored image is higher as the number of votes increases, and, as a result of the determination, a stored image specifying step for specifying the stored image similar to the input image; Using the stored image specified in the memory image specifying step, a recall image indicating how the input image is seen is generated, and the generated recall image and the input image are compared to determine whether they are similar to each other. And a determination step of determining that the recognition target is detected in the input image when it is determined that the images are similar to each other.

  Thereby, it is possible to realize three-dimensional object recognition that is faster and more accurate than the conventional method.

  The object recognition method according to the fourth aspect of the present invention includes a stored image including a recognition target, a feature point extracted in advance from the stored image, and a positional relationship of the extracted feature point with respect to a reference position in the stored image. Storing a plurality of template models in advance for each class model including information, an average scale size of the feature points, and a template size that is a value obtained by normalizing the size of the stored image with the average scale size; A feature point is extracted from the input image, and matching is performed between the feature point of the stored image stored in the storage step and the feature point extracted from the input image, thereby corresponding to the feature point of the stored image A corresponding point search step for searching for a feature point of the input image as a corresponding point, and a corresponding point of the input image searched in the corresponding point search step And calculating voting points based on positional relation information of feature points of the stored image stored in the storing step, and voting processing steps for voting the voting points in the input image, and voting in the voting processing step. A voting point clustering step of clustering voting points in the input image and determining that the recognition target exists around the center of the cluster when the number of voting points included in the same cluster is equal to or greater than a predetermined threshold; In the input image, a rectangular area generated based on the average scale size and the template size of the template model corresponding to the cluster center is determined in the input image with respect to the cluster center determined in the vote point clustering step that the recognition target exists. As a candidate frame indicating the range where the target exists A candidate frame generation step, and a recall image that generates a recall image that indicates how the input image is viewed using the stored image of the template model corresponding to the candidate frame for the candidate frame generated in the candidate frame generation step The difference between the image generation step, the recall image generated in the recall image generation step, and the image of the candidate frame in the input image is calculated, and the calculated difference is greater than a predetermined threshold value. If it is larger, the candidate frame is removed as erroneous detection, and a confidence value calculation step for determining that the recognition target is detected in the range of the input image indicated by the candidate frame for the candidate frame that has not been removed And.

  Thereby, it is possible to realize three-dimensional object recognition that is faster and more accurate than the conventional method.

  According to the present invention, it is possible to provide an object recognition apparatus and an object recognition method capable of realizing three-dimensional object recognition that is faster and more accurate than conventional methods.

1 is a functional block diagram illustrating a configuration of an object recognition device according to Embodiment 1. FIG. 3 is a diagram for explaining a learning model according to Embodiment 1. FIG. 3 is a flowchart showing an outline of a recognition procedure according to the first embodiment. 6 is an image for explaining a feature point position vector according to the first embodiment; 6 is an image for explaining voting of center candidate points according to the first embodiment. 5 is an image for explaining corresponding point search and Voting processing in an input image according to the first embodiment. 4 is an image showing an example of a recall image according to the first embodiment. It is an image for demonstrating the generation method of the recall image which concerns on Embodiment 1. FIG. It is an image for demonstrating the false detection removal using the recall image which concerns on Embodiment 1. FIG. It is an image which shows the recognition result when it has detected correctly according to Embodiment 1. It is an image which shows the recognition result at the time of detecting incorrectly according to Embodiment 1. 6 is a graph showing a class detection result according to the first embodiment. 6 is a graph showing a direction estimation result according to the first embodiment.

Before describing each embodiment of the present invention, the basic configuration of the present invention will be described.
First, the present invention is independent of each learning image for three-dimensional object recognition for a general object, which is a rigid body with relatively little change in shape (an automobile will be described as an example in the embodiments described later). Appearance-based approach to learning and recognition is adopted. Furthermore, as will be described later, in order to confirm the effect of the present invention, the PASCAL VOC 2006 dataset is used to evaluate the class detection accuracy of a three-dimensional object and quantitatively evaluate the direction estimation results. .

  In the present invention, Voting processing with relatively low calculation cost (Masunari Takagi, Hironobu Fujiyoshi, “Traffic road sign recognition using SIFT features,” Denki C, vol.129, no.5, pp.824 -831, 2009.), the presence / absence of the existence of each learned recognition target is determined, and the existence range and the direction of the recognition target are estimated by superimposing the results.

  In the present invention, at the time of recognition, the appearance (appearance) of the recognition target is recalled (a recall image is generated). This is to estimate how the recognition target exists in the recognized range and its appearance, so that the system can visually check how the target is recognized. It becomes possible. In the present invention, the false detection is removed by comparing the recall result and the recognition range as an image, thereby improving the recognition accuracy. In principle, the recognition target according to the present invention is not limited to an automobile as long as it is a rigid body with little shape change, and may be another object.

Embodiment 1
Embodiments of the present invention will be described below with reference to the drawings. In the following, in the description in the text, the symbols described before are used as necessary.

  FIG. 1 is a functional block diagram showing the configuration of the object recognition apparatus according to the present embodiment. The object recognition device 1 includes a database 2 storing learning models, a corresponding point search unit 3, a voting processing unit 4, a voting point clustering unit 5, a candidate frame generation unit 6, a recall image generation unit 7, a trust A value calculation unit 8 and a direction estimation unit 9 are provided.

  The stored image specifying unit 11 includes a function of the corresponding point search unit 3, a function of the voting processing unit 4, a function of the voting point clustering unit 5, and a function of the candidate frame generation unit 6. More specifically, the feature point is extracted from the input image by the function of the corresponding point search unit 3, and the feature point of the stored image stored in the database 2 and the feature point extracted from the input image correspond to each other. Search for feature points. With the function of the voting processing unit 4, the voting points calculated based on the positional relationship information of the feature points of the stored image are voted on the input image for the searched corresponding points. The function of the voting point clustering unit 5 determines that the similarity between the input image and the stored image is higher as the voting points are concentrated and the number of votes is larger. As a result of the determination, a stored image similar to the input image is specified by the function of the candidate frame generation unit 6.

  The determination unit 12 includes a function of the recall image generation unit 7, a function of the confidence value calculation unit 8, and a function of the direction estimation unit 9. More specifically, the function of the recall image generation unit 7 generates a recall image that shows how the input image looks using the stored image specified by the stored image specifying means 11. The function of the confidence value calculation unit 8 compares the generated recall image and the input image to determine whether they are similar to each other, and when it is determined that they are similar, the recognition target is detected in the input image Is determined. The direction of the detected recognition target is estimated by the function of the direction estimation unit 9.

  The database 2 is a storage unit that stores a learning model. In this embodiment, an aggregate of all template models for a certain class is used as a class model, and an aggregate of all class models is handled as a learning model. That is, the learning model includes a plurality of class models, and each class model includes a plurality of template models. In the example illustrated in FIG. 1, the learning model includes a class model 21, and the class model 21 includes template models 211, 212, and 213. In the present embodiment, the learning model is described as having one class model (car class), but may have a plurality of class models (multi-class).

  The object recognition device 1 learns a recognition target in advance and performs recognition using a learning model stored in advance in the database 2. At the time of learning, the object recognition apparatus 1 incrementally learns the recognition target included in the learning image every time the learning image is presented. Learning of the recognition target is performed by constructing a template model for all of the given learning images.

  As shown in FIG. 2, each template model includes a grayscale image (Learning image) (hereinafter sometimes referred to as a stored image) of a region including a recognition target, a feature point, and a position vector ( Feature point & Location vector) and template information (Template information) are included. Template information includes the average scale size of feature points (Average scale size of feature points) and the value obtained by normalizing the size of the stored image with the average scale size (Template size (width, height)) (hereinafter referred to as template size) And orientation label (Pose label).

  When a learning image is given, the object recognition device 1 extracts a local feature amount (hereinafter sometimes referred to as a feature point) in a region where a recognition target exists in the learning image, and extracts each feature point. A position vector is calculated for. Details of the position vector will be described later. In the present embodiment, the object recognition apparatus 1 calculates the average scale size of the feature points and the template size, and the orientation label is given by the user.

  Next, the outline of the recognition procedure by the object recognition apparatus 1 will be described with reference to FIG. The object recognition apparatus 1 recognizes an object using a learning model that is built in advance. The recognition procedure is roughly divided into two stages of a candidate frame detection procedure (S102) and an erroneous detection removal procedure (S103). First, in the candidate frame detection procedure (S102), a range in which the recognition target is considered to exist in the input image (hereinafter sometimes referred to as a candidate frame) is detected. In the erroneous detection removal procedure (S103), candidate frames that are considered to be erroneous detection are removed from the detected candidate frames. More specifically, each includes the following procedures.

S101: The object recognition apparatus 1 acquires an input image (input image acquisition process).
S111: Corresponding point search unit 3 searches for a correspondence between feature points between the learning model and the input image (corresponding point search process).
S112: The voting processing unit 4 uses the positional relationship information of the feature points in the learning model for the feature points of the input image that correspond to the learning model, and determines center candidate points for the feature points of the input image. Vote (Voting process).
S113: The voting point clustering unit 5 clusters the central candidate points voted on the input image (hereinafter sometimes simply referred to as voting points), and obtains a cluster in which the number of votes exceeding a predetermined threshold is collected (voting). Point clustering process).
S114: The candidate frame generation unit 6 generates a rectangular area (candidate frame) surrounding the cluster center (candidate frame generation process).
S116: For each candidate frame in the input image, the recall image generation unit 7 generates a recall image using the stored image of the learning model corresponding to the candidate frame (appearance recall process).
S117: The reliability value calculation unit 8 calculates the similarity between the recall image and the real image (the image in the candidate frame in the input image) as a reliability value, and removes the erroneous detection candidate frame (reliability value calculation process). ).
S117: The direction estimation unit 9 estimates the direction of the detected recognition target from the direction label of the template model used for calculating the candidate frame (direction estimation process).
S104: The object recognition apparatus 1 outputs the presence / absence of the recognition target, the existing range, and the direction (recognition result output process).
The details of the processing by each unit will be specifically described below.

  The corresponding point search unit 3 extracts feature points from the input image, and performs matching between all feature points in the learning model and feature points extracted from the input image. In this embodiment, SURF is used as a feature point extracted from the input image, but the type of feature point extracted from the input image is not limited to this, and other types of feature points used for object recognition such as SIFT are used. May be used.

When the i-th feature point in the learning model is p _i , the j-th feature point in the input image is q _j, and the distance between them is d _ij , the feature point nearest to the feature point p _i is , J _1NN = argmin _j d _ij is a feature point q _j1NN . Here, when there are a plurality of similar feature points in the input image, if the feature point q _j1NN is regarded as a corresponding point as it is, many erroneous correspondences may occur. For this reason, in the present embodiment, only feature points q _j satisfying the following number (1) are used as corresponding points. In _Equation (1), j _2NN represents the index of the second closest feature point, and t is a predetermined threshold value.

The number (1) indicates that the distance d _ij1NN with the nearest neighbor is less than a certain ratio with respect to the distance d _ij2NN with the second closest feature point. By this conditional expression, it is possible to perform corresponding point search with reduced erroneous correspondence.

  The voting processing unit 4 uses the Voting process to determine whether or not there is a recognition target and determine a region where the recognition target exists. The Voting process is useful for improving recognition speed because it calculates the existence range using a process that does not need to be repeated, such as calculating voting points that are uniquely determined for all matched feature points. It is.

  In the Voting process, using the positional relationship information of the feature points in the learning model, the recognition target center candidate points included in the input image are estimated and voted for the corresponding feature points in the input image. This is a technique that applies the generalized Hough transform, and instead of calculating the number of votes for each item in the hash table, the center candidate points voted on the real image area are clustered, and the number of votes for each cluster is calculated. Is. The flow of the voting process for the center candidate point will be described below.

  First, as a preparation, a reference point is given to each stored image of the template model during learning. In the present embodiment, the center of the stored image is set as a reference point, and in each stored image, positional relationship information between the set reference point and each feature point is calculated. Then, the calculated positional relationship information is given to the template model as a position vector for each feature point. In the present embodiment, the center of the stored image is set as the reference point. However, the present invention is not limited to this, and any other position may be set as the reference point.

  For example, as shown in the upper left of FIG. 4, a location vector of a feature point is calculated using the center (Center point) as a reference point. For example, as shown in the lower right of FIG. 4, for the recognition target (automobile) of the stored image, the position vector for each of the three feature points is calculated with reference to the center of the stored image.

Next, based on the feature point of the stored image of the template model and the position vector of the feature point, a reference point for the feature point in the input image corresponding to the feature point, that is, a center candidate point is obtained. Here, the feature points and position vectors in the stored image of the template model and the feature points in the input image are given as follows.
(I) feature points in the stored image of the template model;
Coordinates: (x _temp , y _temp )
Scale: σ _temp
Luminance gradient direction: θ _temp
Position vector: (Δx, Δy).
(Ii) feature points in the input image;
Coordinates: (x _in , y _in )
Scale: σ _in
Luminance gradient direction: θ _in .

Then, the center candidate point (X, Y) in the input image is obtained from the coordinates, scale, brightness gradient direction, and position vector of the feature point of the stored image, and the coordinates, scale, and brightness gradient direction of the feature point of the input image. Can be obtained by the following numbers (2) and (3). Here, θ = arctan (Δy / Δx).

  The above processing is performed for all feature points corresponding to each template model, and the central candidate point is voted for the input image. That is, for one template model, the center candidate point is calculated by using information such as the position vector of the feature point corresponding to the feature point of the stored image corresponding to the feature point of the input image. Then, the calculated center candidate point is voted for the input image.

  When the same or similar recognition target as the stored image exists in the input image (that is, when the same class of recognition target exists), the positional relationship of the feature points with respect to the reference point in the stored image and the center candidate in the input image The positional relationship of the feature points with respect to the points is considered to be similar to each other. In other words, since all the feature points related to the same recognition target in the stored image are defined by the positional vectors from the same reference point, the recognition target of the same class as the stored image is included in the input image. If the center candidate point of the feature point in the input image is estimated using the position vector of the feature point of the corresponding stored image, the center candidate point is the same as the reference point in the stored image. , Likely to concentrate in a specific place. Therefore, if recognition objects of the same class exist in the input image, it is considered that the voted center candidate points are concentrated near the center of the recognition object.

  For example, as shown in the left diagram of FIG. 5, when the center candidate points (voting points) are dispersed, there is a high possibility that the recognition target object does not exist. On the other hand, as shown in the figure on the right side of the figure, when the voting points are concentrated, it is considered that there is a high possibility that the recognition object exists.

  When the central candidate points are voted on the input image, the voting point clustering unit 5 performs clustering of the central candidate points (voting points). The voting point clustering unit 5 collects adjacent voting points in the same cluster, and calculates the number of votes included in the cluster for each cluster. If the number of votes for a cluster is equal to or greater than a predetermined threshold value (hereinafter sometimes referred to as a vote threshold value), it is determined that a recognition object centered on the cluster center exists. .

  For example, FIG. 6 shows an example in which the number of votes in the cluster of the input image (Input image) is greater than or equal to the vote threshold. In this case, the recognition target (automobile) included in the stored image of the template model (Template model) Is determined to exist around the cluster center of the input image.

  In this embodiment, vote point clustering is applied to the TOD (Threshold Order-Dependent) algorithm (M. Friedman and A. Kandel, “Introduction to Pattern Recognition,” pp. 70-73, World Scientific Publishing Company, 1999.). Run based on. The TOD algorithm can process data sequentially and can perform data clustering at high speed due to extremely simple processing. Note that the voting point clustering is not limited to the TOD algorithm, and may be performed based on other known clustering methods.

  Hereinafter, clustering based on the TOD algorithm will be described. When the coordinates of the voting points are v and the vector whose elements are the scale size and the template size of the feature points with respect to the voting points is w, the processing is as follows.

  Step 1: A clustering threshold T is set. This is the maximum distance between voting points for the same cluster. In this embodiment, a maximum distance with respect to the unit scale size is determined in advance, and a value obtained by multiplying the average scale size of all feature points is set as the clustering threshold T for each input image. Thereby, it can set to a relative value according to the magnitude | size of recognition object.

Step 2: Let C be a set of cluster centers, and let the first input c _{0 be} an element of the set C. Also, a set P _c0 of orientation labels of the cluster center c ₀ is created, and the first input direction label is used as this element. Further, the number of votes ε _c0 to the cluster center c ₀ is 1.

Step 3: With c _new as a new input, search for the cluster center c _NN = argmin _cεC ‖v _cnew -v _cと which is the nearest to c _new .

Step 4: When the distance between v _cnew and v _cNN exceeds the clustering threshold T, c _new is added to the set C. Further, P _cnew having the c _new direction label as an element is created, the number of votes ε _cnew for c _new is _set to 1, and the process returns to _{Step 3} .

_{Step5: v} when the distance _cnew and _{v CNN} is less clustering threshold T _increases 1 counting epsilon _CNN is the total number of votes _{c NN, v CNN} and _w values of _CNN following equation (4 ) And the number (5). Further, the c _new direction label is added to the set P _cNN and the process returns to step 3.

  Step 6: Finally, when all the voting points have been input, it is determined whether or not the number of votes is greater than or equal to the voting threshold for all generated cluster centers. As a result of the determination, the cluster center whose number of votes is less than the vote threshold is deleted. In the present embodiment, clustering is performed by first executing the above processing for each template model.

  The vote point clustering unit 5 performs the clustering shown in Step 1 to Step 6 on each template model. Further, clustering is performed again for each class model using the remaining cluster center for each template model as a voting point. The center of the cluster obtained by the second clustering is set as the final clustering result. The second clustering is for collecting recognition results for each template model, and the cluster center is not deleted based on the number of votes. The clustering threshold T used in the second clustering is a value proportional to the clustering threshold used in the first clustering.

  As a result of clustering, each generated cluster corresponds to one or more template models. Therefore, a template model (stored image) similar to the input image is specified by the above processing. The degree of similarity between the input image and the stored image is determined as the similarity between the input image and the stored image of the template model increases as the number of votes increases in the input image and the number of votes increases.

  The candidate frame generation unit 6 generates, as a candidate frame, a rectangular area centered on the coordinates of each cluster generated as a result of the clustering process. The rectangular area is created based on the average scale size and template size of one or more template models corresponding to each cluster. More specifically, w updated during clustering as a parameter for generating a rectangular area includes the width and height (that is, the template size) of the rectangular area. For this reason, during the first and second clustering processes, w is sequentially updated according to the number (5), so that a desired rectangular area size is recorded with respect to the final cluster center. Is called. That is, in the process of clustering, the template size held by the cluster center is continuously updated, and finally a value such as the average of the voting points belonging to the cluster is obtained, and this is used as the size of the rectangular area. . Thereby, a candidate frame in which a recognition target object is thought to exist is generated in the input image. As a result of the candidate frame generation, each candidate frame corresponds to one cluster center. Each candidate frame corresponds to one or a plurality of template models.

  The recall image generation unit 7 recalls the appearance of each candidate frame generated as an image by using the stored image of the template model corresponding to the candidate frame. Here, the recalled image (hereinafter sometimes referred to as the recalled image) is a visual representation of how the system recognizes the object and how the object looks. It is an estimate of whether or not it exists. For example, for the candidate frame shown in the left diagram of FIG. 7, a recall image as shown in the right diagram of FIG. 7 is generated.

  The method for generating the recall image is simple. First, when generating a candidate frame, the number of votes obtained from which template model for the template model corresponding to the cluster is stored as information for the cluster center of the candidate frame. Then, for each template model, the stored image is resized to the size of the candidate frame, and further, the brightness is reduced, and then the stored images are superimposed. At this time, the brightness of each stored image to be reduced is adjusted according to the weighting according to the clustering result. In the present embodiment, the calculation is performed by multiplying the luminance value of all the pixels of each stored image by the weighting according to the clustering result. Here, the ratio of the number of votes of the template model (stored image) to the number of votes stored at the center of the cluster is the above weighting. Thereby, one recall image is finally generated for one candidate frame.

  For example, in the example illustrated in FIG. 8, a candidate frame (Candidate window detection) illustrated in the upper right diagram is generated from the voting result (Voting result) illustrated in the upper left diagram in the input image. The memory images (Learning image of template model) shown in the lower left figure are then resized and weighted to create a recall image shown in the lower right figure. did. In the example shown in the figure, the lightness weighting is calculated based on the total number of votes stored in the center of the cluster (10 votes in the figure), and the number of votes of each template model (3 votes in the figure, 5 votes, 2 votes) (0.3, 0.5, 0.2).

  The confidence value calculation unit 8 calculates the confidence value of the candidate frame by comparing the generated recall image and the actual image (the image of the candidate frame in the input image), and erroneously detects based on the calculated confidence value. The candidate frame is removed. That is, when the recalled image and the actual image are compared and these images are similar, it is considered that the candidate frame has been correctly detected. On the other hand, if the two images are not similar, it is considered that the candidate frame was erroneously detected. The reliability value calculation unit 8 removes the candidate frame that is regarded as being detected in error, and as a result, removes the recognition target (the template model corresponding to the candidate frame) in the range of the input image indicated by the candidate frame that is not removed. Class) is detected.

  In the present embodiment, the degree of difference between the recall image and the actual image is calculated as an average value of the luminance value differences of all the pixels, and if this value is larger than a predetermined threshold value, it is determined as a false detection. The candidate frame is removed. As a candidate frame removal method, an erroneous candidate frame can be removed based on various evaluation criteria other than the average value of the luminance value differences of all pixels.

  In the present embodiment, for each candidate frame after false detection removal based on the above-mentioned difference degree, in addition to the difference degree between the recalled image and the actual image (average value of luminance value differences of all pixels), voting The confidence value of the candidate frame that takes the number into consideration is calculated, and if this value is smaller than a predetermined threshold, the candidate frame is removed as a false detection. This is because it is preferable to make a comparison based on the similarity as an image without considering the number of votes in the determination of erroneous detection. However, when it is determined that the detection is correct, the similarity as an image is the same. This is because the higher the number of votes, the higher the degree of similarity with the target.

The confidence value is calculated by the following number (6). However, the horizontal and vertical widths of the candidate frame are w and h, the luminance value of the actual image of the candidate frame is I _est , the luminance value of the recall image is I _rec, and the total number of votes of the candidate frame is ε. By using this value, the reliability of each candidate frame can be compared.

  For example, in the example illustrated in FIG. 9, as a result of comparison with the recall image for three candidate frames as illustrated in the upper right diagram, candidate frames other than the one candidate frame illustrated in the lower left diagram are removed as erroneous detection.

The direction estimation unit 9 estimates the direction of the recognition target for each candidate frame. P is the set of orientation labels, A _class (p) is the total number of template models with orientation label p∈P in the class model to be recognized, and the number of votes whose orientation label matches p among all the voting results of the candidate frames Is A _est (p), the orientation of the recognition target is estimated by the following number (7). In other words, the orientation label that has been voted the most as a percentage of the total number of orientation labels learned is the estimation result.

  Next, the effect by this Embodiment is demonstrated. In order to show the effectiveness according to the present embodiment, an experiment of class detection and direction estimation by the object recognition apparatus 1 was performed on a real environment image obtained by photographing a real world scene.

  In the experiment, PASCAL Challenge Visual Object Class ("PASCAL Challenge" .http: //www.pascal-network.org/challenges/VOC/) is a data set that is often used for evaluation experiments of 3D object recognition methods. .) Was used. In this embodiment, the PASCAL VOC 2006 dataset (M. Everingham, A. Zisserman, CKI Williams, and L. Van Gool, "The PASCAL Visual Object Classes Challenge 2006 (VOC2006) Results" , "Technical report, PASCAL Network, 2006.)" was evaluated for the "car" class.

  In the case of the “car” class data, 854 cars shown in 553 out of all 2,618 learning images are given as learning targets, and 854 cars shown in 544 out of all 2,686 test images. It is a detection target. All the images are taken from a variety of real environments, and there are various types of automobiles to be detected, and their orientations and sizes vary.

  As with other 3D object recognition methods, in addition to the above learning data, 3D objects dataset (S. Savarese and L. Fei-Fei, "3d generic object categorization, localization and pose estimation," IEEE "Car" class data of Int. Conf. Comput. Vision, pp.1-8, 2007.) was also used. In this example, 48 images taken on 8 directions × 2 altitudes × 3 scales, shooting position information, and target area information are given for 10 types of automobiles.

  10 and 11 show examples of processing results for the above data set. FIG. 10 is an image showing an example of a case where the detection is correctly performed. A small image in the upper left or lower right of each image indicates a recall image for each detection result. It can be seen that even when there are some occlusions in the recognition target or when a plurality of recognition targets are included, the recognition target is correctly recognized. FIG. 11 is an image showing an example of erroneous detection. It can be seen that the region size is not appropriate or that a clear misrecognition has occurred.

  Next, an automobile class detection experiment was performed using the test data, and the results are shown in FIG. FIG. 12 shows a comparison result between the present embodiment and Sun & Su CVPR09 (Non-Patent Document 2). The evaluation was performed with a precision-recall curve and its AP (Average Precision).

  As can be seen from FIG. 12, regarding the detection accuracy, the AP of the existing method (Sun & Su CVPR09) is 0.310, whereas the AP of the present embodiment (Our method) is 0.323, Higher detection accuracy can be achieved. Regarding the processing time required for recognition, in this embodiment, the recognition time per image including the feature extraction time is 12.6 seconds (3.2 GHz, Matlab) on average. On the other hand, according to personal communication, the recognition time of the existing method is about 300 seconds (2.2 GHz, Matlab) per image. Therefore, not only the detection accuracy but also the processing speed was confirmed to be effective according to the present embodiment.

  In FIG. 12, Liebelt CVPR08 (J. Liebelt, C. Schmid, and K. Schertler, “Viewpointindependent object class detection using 3d feature maps,” IEEE Int. Conf. Comput. Vision and Pattern Recognit. , pp.1-8, 2008.) and Su & SunICCV09 (H. Su, M. Sun, L. Fei-Fei, and S. Savarese, "Learning a dense multi-view representation for detection, viewpoint classification and synthesis of object The results of four teams participating in categories, "IEEE Int. Conf. Comput. Vision, 2009.) and PASCAL VOC 2006 are also shown in the graph. However, Liebelt CVPR08 uses its own CG model, and Su & SunICCV09 uses its own prepared video clip in addition to the learning data used in this experiment. Note that the data is the same, but the data used for learning is different. In addition, the method of the team that participated in PASCAL VOC 2006 is only for the purpose of target detection, and the direction estimation is not considered.

  Next, the direction of the recognition target was estimated for the detection results for the test data that were correctly recognized. FIG. 13 shows the result of direction estimation according to the present embodiment and the result of comparison (Sun & Su CVPR09). However, the direction of the direction is estimated in the PASCAL VOC 2006dataset only in four directions. For this reason, in this Embodiment, it was estimated to which of the four directions it belongs. Note that the estimation result is not considered when the target in the test image does not belong to any of the four directions.

  As shown in FIG. 13, in the present embodiment, it was possible to estimate with higher accuracy in any direction as compared with the comparison target (Sun & Su CVPR09). Regarding the average accuracy, the existing method is 62%, while it is 86% in the present embodiment, and the high accuracy of the direction estimation according to the present embodiment was confirmed.

  As described above, in this embodiment, a relatively high-speed and high-accuracy appearance-based three-dimensional object recognition method based on two-stage processing of candidate frame detection by voting processing and false detection removal by appearance recall. It was realized. In addition, in order to confirm the effect of this embodiment, we performed experiments using the PASCAL VOC 2006 dataset and showed results superior to the existing method in the direction estimation accuracy and recognition time. The result was equivalent or better.

  In the present embodiment, by adopting an appearance-based method that independently learns and recognizes a given appearance itself as compared to the conventional model-based method, it is possible to easily compensate for the disadvantages of the conventional method. It was possible.

  Further, in the present embodiment, by using the Voting process, it is possible to recognize the target at the class level by superimposing the results while learning and recognizing various appearances of the target independently. Here, the present embodiment is characterized in that appearance is recalled with respect to the recognition result obtained by the Voting process, and false recognition is eliminated by comparing the recall result with the recognition result. We were able to improve more.

  Furthermore, in this embodiment, since learning is performed independently from each learning image, there is an advantage that the description method of the recognition target is simple and that additional learning can be easily performed.

Note that the present invention is not limited to the above-described embodiment, and can be changed as appropriate without departing from the spirit of the present invention.
For example, in the above-described embodiment, the hardware configuration has been described. However, the present invention is not limited to this, and arbitrary processing may be realized by causing a CPU (Central Processing Unit) to execute a computer program. Is possible. The program may be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROMs (Read Only Memory), CD-Rs, CD-R / W and semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (random access memory)) are included. The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

1 object recognition device,
2 database,
3 Corresponding point search part,
4 voting processing section,
5 vote point clustering unit,
6 candidate frame generator,
7 Recall image generator,
8 confidence value calculator,
9 orientation estimation unit,
11 stored image specifying means,
12 determination means,
21 class model,
211, 212, 213 Template model

Claims (10)

A memory for storing a plurality of stored images each including a recognition target, feature points extracted in advance from each of the plurality of stored images, and positional relationship information of the extracted feature points with respect to a reference position in the stored image Means,
A feature point is extracted from the input image, and a corresponding feature point is searched between the feature point of the stored image stored in the storage unit and the feature point extracted from the input image. The voting points calculated based on the positional relationship information of the feature points of the stored image are voted on the input image, and the similarity between the input image and the stored image increases as the voting points concentrate and the number of votes increases. A stored image specifying means for determining that the stored image is similar to the input image as a result of the determination;
Whether or not the stored image specified by the stored image specifying means is used to generate a recall image indicating how the input image is seen, and the generated recall image and the input image are compared to determine whether they are similar to each other Determining means for determining that the recognition target is detected in the input image when it is determined that they are similar;
An object recognition apparatus comprising: The determination means includes
The said recall image is produced | generated by synthesize | combining the said some memory image specified by the said memory | storage image specification means according to the similarity with each said input image. Object recognition device. The determination means includes
When the recall image and the input image are compared to determine whether they are similar to each other, the larger the total number of voting points of the input image used for generating the recall image, the more the recall image and The object recognition apparatus according to claim 1, wherein it is determined that the input image is more similar. The storage means
Storing the direction information of the recognition target in association with the stored image;
The determination means includes
The recall image and the input image are compared to determine whether they are similar to each other. If it is determined that they are similar, it is determined that the recognition target is detected in the input image, and the recall The object recognition apparatus according to any one of claims 1 to 3, wherein the detected orientation of the recognition target is estimated based on orientation information of the stored image used for generating an image. A stored image including a recognition target, feature points extracted in advance from the stored image, positional relationship information of the extracted feature points with respect to a reference position in the stored image, an average scale size of the feature points, and the storage A database that stores a plurality of template models for each class model, including a template size that is a value obtained by normalizing the size of the image with the average scale size;
A feature point is extracted from the input image, and matching is performed between the feature point of the stored image stored in the database and the feature point extracted from the input image, thereby corresponding to the feature point of the stored image A corresponding point search unit that searches for feature points of the input image as corresponding points;
With respect to corresponding points of the input image searched by the corresponding point search unit, voting points are calculated based on positional relationship information of feature points of the stored image stored in the database, and the voting points are voted on the input image. A voting processor to
When the voting points in the input image voted by the voting processing unit are clustered and the number of voting points included in the same cluster is equal to or greater than a predetermined threshold, the recognition target exists around the center of the cluster. A voting point clustering unit to determine,
A rectangular area that is generated based on an average scale size and a template size of the template model corresponding to the cluster center for the cluster center that is determined by the vote point clustering unit to be present in the input image. A candidate frame generation unit that generates a candidate frame indicating a range in which the
A recall image generation unit that generates a recall image indicating how the input image is viewed using the stored image of the template model corresponding to the candidate frame for the candidate frame generated by the candidate frame generation unit;
When the recall image generated by the recall image generation unit and the image of the candidate frame in the input image are compared to calculate the difference, and the calculated difference is greater than a predetermined threshold A reliability value calculation unit that determines that the recognition target is detected in a range of the input image indicated by the candidate frame for the candidate frame that has been removed as erroneous detection, and the candidate frame has not been removed;
An object recognition apparatus comprising: The recall image generation unit
For the candidate frame generated by the candidate frame generation unit, the storage images of the plurality of template models corresponding to the candidate frame are synthesized according to the number of votes in the input image obtained from each of the template models. The object recognition apparatus according to claim 5, wherein the recall image is generated. The confidence value calculation unit
The recall image generated by the recall image generation unit and the image of the candidate frame in the input image are compared to calculate a difference, and the obtained from all the template models corresponding to the candidate frame The total number of votes in the input image is calculated, and if the confidence value calculated based on the reciprocal of the calculated difference and the total number of votes is smaller than a predetermined threshold, the candidate frame is removed as a false detection. The object recognition apparatus according to claim 5, wherein the candidate frame that has not been removed is determined to have detected the recognition target in the range of the input image indicated by the candidate frame. The template model is
It further includes orientation information indicating the orientation of the recognition target,
When it is determined that the recognition target is detected by the confidence value calculation unit, for the detected candidate frame of the recognition target, based on the orientation information of the plurality of template models corresponding to the candidate frame, The object recognition apparatus according to claim 5, further comprising a direction estimation unit that estimates the detected direction of the recognition target. A plurality of stored images each including a recognition target, feature points extracted in advance from each of the plurality of stored images, and positional relationship information of the extracted feature points with respect to a reference position in the stored image are stored in advance. A memory step;
A feature point is extracted from the input image, a corresponding feature point is searched between the feature point of the stored image stored in the storage step and the feature point extracted from the input image, and the searched corresponding point The voting points calculated based on the positional relationship information of the feature points of the stored image are voted on the input image, and the similarity between the input image and the stored image increases as the voting points concentrate and the number of votes increases. A stored image specifying step for determining that the stored image is similar to the input image as a result of the determination;
Whether or not the stored image specified in the stored image specifying step is used to generate a recall image indicating how the input image is seen, and the generated recall image and the input image are compared with each other to determine whether they are similar to each other A determination step for determining that the recognition target is detected in the input image when it is determined that they are similar to each other;
An object recognition method comprising: A stored image including a recognition target, feature points extracted in advance from the stored image, positional relationship information of the extracted feature points with respect to a reference position in the stored image, an average scale size of the feature points, and the storage A storage step of storing a plurality of template models in advance for each class model, including a template size that is a value obtained by normalizing the size of the image with the average scale size;
A feature point is extracted from the input image, and matching is performed between the feature point of the stored image stored in the storage step and the feature point extracted from the input image, thereby corresponding to the feature point of the stored image A corresponding point search step for searching for a feature point of the input image as a corresponding point;
For the corresponding point of the input image searched in the corresponding point search step, calculate a voting point based on the positional relationship information of the feature point of the stored image stored in the storage step, and the voting point in the input image A voting process step to vote;
When the voting points in the input image voted in the voting processing step are clustered, and the number of voting points included in the same cluster is equal to or greater than a predetermined threshold, the recognition target exists around the center of the cluster. A voting point clustering step to determine,
A rectangular area that is generated based on an average scale size and a template size of the template model corresponding to the cluster center for the cluster center determined to be present in the vote point clustering step in the input image. A candidate frame generation step for generating a candidate frame indicating a range in which the
A recall image generation step for generating a recall image indicating how the input image is viewed using the stored image of the template model corresponding to the candidate frame for the candidate frame generated in the candidate frame generation step;
When the recall image generated in the recall image generation step and the image of the candidate frame in the input image are compared to calculate a difference, and the calculated difference is greater than a predetermined threshold A reliability value calculating step for determining that the recognition target is detected in a range of the input image indicated by the candidate frame, with respect to the candidate frame that has been removed as being erroneously detected as the candidate frame;
An object recognition method comprising: