JP6055435B2 - Subject recognition apparatus, subject recognition method, and subject recognition program - Google Patents

Description

  The present invention learns in advance a training image group in which a plurality of subjects are photographed, and presumes which subject in the training image group is photographed by test images photographed at different times and at different positions and angles. The present invention relates to a subject recognition method and a subject recognition program.

  A technique for estimating which test image input by a camera that captures a space is similar to a training image group in which a subject is captured in advance enables subject recognition and detection. For example, in the method described in Non-Patent Document 1, a plurality of local feature amounts called SIFT are extracted in advance for each training image group in which a subject is photographed, and stored in a database (DB) in association with the training image. Next, a plurality of SIFT features are similarly extracted from a test image whose shooting subject is unknown.

  Then, for each SIFT feature, a process of specifying a local feature amount having the smallest inter-vector distance from local feature amounts recorded in association with a certain training image is performed over SIFT features of all test images. . Then, after the provisional corresponding points are determined, the erroneous corresponding points are removed, and the number of remaining corresponding points is recorded. This process is performed on all the training images stored in the database, and the training image having the largest number of corresponding points is set as an image corresponding to the test image. By this process, it is realized to estimate which subject is photographed by the test image.

  5 to 7 are explanatory diagrams showing examples of erroneous corresponding point removal according to the prior art. The removal of the erroneous corresponding point after the provisional corresponding point is determined is performed as follows for speeding up. First, changes in position information (x1, y1), scale (s1), and rotation amount (r1) shown in FIG. 5, which are SIFT feature attribute information held between two certain SIFT features constituting a provisional corresponding point. Assume that the quantities (Δx, Δy, Δs, Δr) are the same at the correct corresponding points.

  Then, a four-dimensional space (four-dimensional space defined by the axes of change amounts Δx, Δy, Δs, and Δr: divided by cells including position information, scale, and rotation amount of the local region over all provisional corresponding points: Vote (see Figure 6). As a result of this voting, only corresponding points that have obtained a certain number of votes or more are adopted to eliminate erroneous correspondence (see FIG. 7). Next, the main two-dimensional affine transformation between the test image and the training image is calculated based on the least square method from the provisional corresponding points belonging to the cells that have obtained a certain number of votes or more. The false corresponding point removal is realized by further removing the provisional corresponding point having attribute information that does not follow the calculated affine transformation x, y, scale, and rotation amount as an incorrect correspondence.

D. Lowe, "Distinctive Image Features from Scale-Invariant Keypoints", International Journal of Computer Vision, Vol. 60 Issue 2, pp.91-110, November 2004.

  By the way, in the conventional method, under the assumption that the attribute information such as the position information (x, y coordinates), the scale, and the rotation amount held by the local feature amount is correct, the calculated affine is calculated in the process of removing the incorrect correspondence. Temporary corresponding points having attribute information that does not follow the conversion x, y, scale, and rotation amount are removed.

  However, among the attribute information of local features, in particular, the scale and the rotation amount cause errors due to noise generated in the image and image distortion generated when the subject is a three-dimensional object, and as a result, it becomes impossible to correctly remove the erroneous correspondence. There is a problem.

  The present invention has been made in view of such circumstances, and even when attribute information other than position information of local features has not been correctly obtained, erroneous correspondence points are removed at high speed and with high accuracy, and which subject the test image shows. It is an object of the present invention to provide a subject recognition device, a subject recognition method, and a subject recognition program that can estimate whether a photograph is taken.

The present invention recognizes a local feature amount extracted from a plurality of training images obtained by imaging a subject and attribute information indicating the attribute of the local feature amount in association with each training image, and recognizes it. a local feature extraction unit configured to extract the attribute information and the local feature amount from the test image obtained by imaging a subject to, and the local feature quantity extracted from the test image, and the local feature amount stored in the image information storage unit by comparing the interim and corresponding points tentative corresponding point determination unit for determining a, to build the attribute information of the type of number Vote separated by size of a given cell of the same number of dimensions and space, the provisional response For all points, a weak geometric verification unit that calculates the amount of change between the attribute information and votes for the corresponding cell, and based on the result of the voting, for all the cells having a predetermined number of votes, The provisional corresponding point Fitting a constant geometric model, by estimating the geometric transformation parameters, the strength geometric verification unit for determining the inliers corresponding points not an outlier, the calculated inliers corresponding points reliability value of the reliability is a predetermined value A reliability estimation unit that rejects a lower inlier point group, and a scoring unit that outputs the sum of the unrejected inlier corresponding points as a score of the training image are provided.

  The present invention provides information indicating which point on the training image corresponds to the point on the training image by obtaining a geometric transformation matrix of affine transformation or homography transformation from the inlier corresponding points that are not rejected. Is further provided with a positive correspondence 2D geometric transformation matrix estimation unit.

  The present invention calculates a basic matrix from the inlier corresponding points that have not been rejected, estimates a focal length of a camera that has captured the test image and the training image, and calculates the basic matrix, the test image, and the training image. The camera further comprises a relative posture estimation unit that calculates and outputs relative translation and rotation of the camera from the focal length of the camera that has captured each of the images.

The present invention includes a subject having an image information storage unit that stores a local feature amount extracted from a plurality of training images obtained by imaging a subject and attribute information indicating attributes of the local feature amount in association with each training image. A method for recognizing a subject performed by a recognition apparatus, wherein a local feature extraction step for extracting a local feature and the attribute information from a test image obtained by imaging a subject to be recognized, the local feature extracted from the test image, by comparing the local feature amount stored in the image information storage unit, the provisional corresponding points and tentative corresponding point determination step of determining the size of the predetermined cells of the same number of dimensions and the number of kinds of the attribute information A weak geometric verification step of constructing a voting space separated by, calculating a change amount between the attribute information for all the provisional corresponding points, and voting the corresponding cell, and a result of the voting Strong geometric verification step for obtaining inlier corresponding points that are not outliers by fitting the provisional corresponding points to a predetermined geometric model and estimating geometric transformation parameters for all cells having a predetermined number or more of votes If the calculated inliers corresponding points reliability, the value of reliability reliability estimating step of rejecting low inliers point group than the predetermined value, the sum of inliers corresponding points that are not the rejected of the training images And a scoring step for outputting as a score.

  The present invention provides information indicating which point on the training image corresponds to the point on the training image by obtaining a geometric transformation matrix of affine transformation or homography transformation from the inlier corresponding points that are not rejected. Is further provided with a positive correspondence 2D geometric transformation matrix estimator step.

  The present invention calculates a basic matrix from the inlier corresponding points that have not been rejected, estimates a focal length of a camera that has captured the test image and the training image, and calculates the basic matrix, the test image, and the training image. The method further comprises a relative posture estimation step of calculating and outputting the relative translation and rotation of the camera from the focal length of the camera that captured each of them.

  The present invention is a subject recognition program for causing a computer to function as the subject recognition device.

  According to the present invention, even when attribute information other than position information of local features is not correctly obtained, it is possible to remove erroneous correspondence points with high speed and accuracy and to estimate which subject the test image has captured with high accuracy. The effect that it can be obtained.

It is a block diagram which shows the structure of the to-be-recognized apparatus by 1st Embodiment of this invention. 3 is a flowchart showing a processing operation of the subject recognition apparatus shown in FIG. 1. It is a block diagram which shows the structure of the to-be-recognized apparatus by 2nd Embodiment of this invention. It is a block diagram which shows the structure of the to-be-recognized apparatus by 1st Embodiment of this invention. It is explanatory drawing which shows the example of the false corresponding point removal by a prior art. It is explanatory drawing which shows the example of the false corresponding point removal by a prior art. It is explanatory drawing which shows the example of the false corresponding point removal by a prior art.

<First Embodiment>
A subject recognition apparatus according to a first embodiment of the present invention will be described below with reference to the drawings. FIG. 1 is a block diagram showing the configuration of the embodiment. As shown in FIG. 1, the subject recognition apparatus 1 includes a database (DB) 20, a training unit 30, and a recognition unit 40. The training unit 30 includes a preprocessing unit 2, a local feature extraction unit 3, and a local feature quantity / attribute information storage unit 4. The recognition unit 40 includes a preprocessing unit 2, a local feature extraction unit 3, a provisional corresponding point determination unit 5, a weak geometric verification unit 6, a strong geometric verification unit 7, a reliability estimation unit 8, and a scoring unit. 9. The preprocessing unit 2 and the local feature extraction unit 3 are the same in the training unit 30 and the recognition unit 40.

  Next, the operation of the training unit 30 shown in FIG. 1 will be described with reference to FIG. The training unit 30 inputs a training image group obtained by photographing different subjects. When a training image group is input to the training unit 30, the following processing is performed sequentially one by one. First, the preprocessing unit 2 enlarges or reduces the input image to a predetermined size. In this process, for example, a smoothing process may be performed in order to reduce image noise.

Next, the local feature extraction unit 3 extracts local feature amounts and attribute information from the enlarged / reduced image. As the local feature, SIFT similar to Non-Patent Document 1 may be used, or other local features such as SURF, BRIEF, BRISK, ORB, and FREAK may be used. As attribute information, position information (x, y), a scale, and a rotation amount may be used as in Non-Patent Document 1, and the rotation direction is acquired for three degrees of freedom by the method of Reference Document 1, and position information (x , Y), scale, rotation (roll, pitch, yaw).
Reference 1: J. Morel, G. Yu, "ASIFT: A New Framework for Fully Affine Invariant Image Comparison", Journal SIAM Journal on Imaging Sciences, Volume 2 Issue 2, pp. 438-469, April 2009

  Next, the local feature quantity / attribute information storage unit 4 records and stores the set of the extracted attribute information and local feature quantity in the database 20 in association with the training image ID uniquely assigned to each training image.

  Next, the processing operation of the recognition unit 40 shown in FIG. 1 will be described with reference to FIG. FIG. 2 is a flowchart showing the processing operation of the recognition unit 40 shown in FIG. The recognition unit 40 inputs an image obtained by photographing an unknown subject. When an image is input, the preprocessing unit 2 reduces the image to a specified size (step S1). Subsequently, the local feature extraction unit 3 extracts local feature amounts and attribute information (step S2). Since the processes of the preprocessing unit 2 and the local feature extraction unit 3 are the same as those of the training unit 30, detailed description thereof is omitted.

  Next, the provisional corresponding point determination unit 5 reads a local feature amount group associated with the training image ID from the database 20 (step S3). Subsequently, the following processing (steps S4 to S12) is performed for each training image ID. First, the provisional corresponding point determination unit 5 compares the local feature amount of a certain local feature Fi of the extracted test image with all the local feature amounts of the read target training image ID, and the distance between the feature amounts is the largest. A local feature Fj of the target training image ID to be reduced is determined, and a pair of Fi and Fj is set as a provisional corresponding point Cp {Fi, Fj} (step S5). This process is performed for all local features extracted from the test image. When the number of local features of the test image is N, the number of provisional corresponding points is also N.

  Next, the weak geometric verification unit 6 constructs a voting space that is divided by a predetermined cell size and has the same number of dimensions as the number of attribute information of local features. For example, if the attribute information of local features is position information (x, y), scale, and rotation amount, a four-dimensional voting space is constructed. If the position information (x, y), scale, and rotation (roll, pitch, yaw), a 6-dimensional voting space is constructed. In addition, it is good also as a 4 dimension by a scale and rotation (roll, pitch, yaw), without using all 6 dimensions here. Here, it is assumed that a four-dimensional voting space of position information (x, y), scale, and rotation amount is constructed.

  Next, the weak geometry verification unit 6 calculates the amount of change between the attribute information for all the provisional corresponding points and votes for the corresponding cell (step S6). In this process, for example, for a certain provisional corresponding point Cp {Fi, Fj}, the attribute information of Fi is {10, 0, 1, 90} in the order of position information (x, y), scale, and rotation amount, and similarly Fj. If the attribute information is {30, 30, 3, 200}, the change amount of Fj with respect to Fi is {20, 30, 3, 110}. Here, the position and rotation components are calculated by the difference amount, and the scale component is calculated by the change ratio. From this variation {20, 30, 3, 110}, the cell having the closest value in the prepared voting space is specified, and the target provisional corresponding point Cp {Fi, Fj} is stored in the cell. The above processing is performed over all provisional corresponding points.

  Next, the subsequent processing (steps S7 to S10) is performed for all cells having a predetermined number of votes. Here, the predetermined number is the minimum number of corresponding points required in the geometric model fitting described later. For example, when the basic matrix F indicating the positional relationship between the imaging camera of the training image and the imaging camera of the test image is used as the geometric model. 7 or 8. For example, 3 is used when affine transformation is used, and 4 is used when projective transformation (homography) is used.

  First, the strong geometric verification unit 7 fits a predetermined geometric model to the provisional corresponding point group belonging to the processing target cell using robust estimation. Here, the robust estimation is a method for estimating a valid parameter based on random sampling of data even if outliers are mixed in the data. Here, the data corresponds to a provisional corresponding point group, the outlier corresponds to an erroneous corresponding point, and the parameter corresponds to a geometric model parameter. In the robust estimation performed by the strong geometric verification unit 7, a corresponding point group determined to be an inlier that is not an outlier among the parameters of the estimated geometric model and the input temporary corresponding point group is obtained. Is output as an inlier corresponding point group (step S8).

  Next, the reliability estimation unit 8 calculates the reliability of the inlier corresponding point group, and rejects the target inlier point group if the reliability value is lower than a predetermined value (step S9). On the contrary, if it is larger than the predetermined value, it is accepted and the number of inlier corresponding points is recorded in the memory, and the process is finished. Here, the calculation of the reliability is performed by mapping the test image on the training image using the estimated geometric transformation parameter described in Non-Patent Document 1, for example, and miscorresponding according to the number of local feature points in the mapping region This can be realized by a method of calculating the probability of occurrence.

  When the processing is completed for all the cells whose number of votes is equal to or larger than the predetermined number, the scoring unit 9 calculates the sum of the inlier corresponding points recorded in the memory, and records it in the memory as the score of the target training image ID (step S11).

  And if processing is completed about all training image IDs, finally, training image ID and a score are linked | related and output, and the process of the recognition part 40 is complete | finished (step S13). The subject can be recognized by regarding the training image having a high score as an image of the subject in the test image.

  With this configuration, even when the attribute information of the local feature is uncertain, it is possible to remove the erroneous corresponding point at high speed and accurately estimate which subject the test image has captured.

Second Embodiment
Next, a subject recognition apparatus according to a second embodiment of the present invention will be described. In the first embodiment described above, geometric verification is performed based on the vote of change amount of attribute information of the local feature, and then geometric verification is performed by robust estimation based only on the position of the local feature. It is possible to accurately estimate which subject is taken by the test image.

  However, there is a problem that it is unclear where the corresponding training image region, that is, the subject region exists on the test image. The second embodiment solves this problem.

  FIG. 3 is a block diagram showing the configuration of the subject recognition apparatus according to the embodiment. In this figure, the same parts as those in the apparatus shown in FIG. The apparatus shown in this figure is different from the apparatus shown in FIG. 1 in that a positive correspondence 2D geometric transformation matrix estimation unit 10 is provided.

Next, the operation of the subject recognition apparatus 1 shown in FIG. 3 will be described. The positive correspondence 2D geometric transformation matrix estimation unit 10 inputs inlier corresponding points used by the scoring unit 9 for score calculation. The inlier correspondence points indicate a probable correspondence of local features between the test image and the corresponding training image. The positive correspondence 2D geometric transformation matrix estimation unit 10 uses the corresponding points to estimate where the region on the test image corresponding to the training image is. Specifically, the geometric transformation from the training image to the test image is affine transformation or homography transformation, and the parameters of the affine transformation or homography transformation are estimated by the least square method. Here, the method for estimating the affine transformation or homography transformation parameters from the corresponding points by, for example, the least square method is the same as the method described in Non-Patent Document 1 for affine transformation and Reference 2 for homography. Therefore, detailed description thereof is omitted here.
Reference 2: Richard Hartley and Andrew Zisserman, "Multiple View Geometry in Computer Vision Second Edition", Cambridge University Press, March 2004. pp.88-91.

  Finally, the correct correspondence 2D geometric transformation matrix estimation unit 10 outputs the estimated geometric transformation matrix as a 2D geometric transformation matrix, and ends the process. Note that the calculated affine transformation or homography transformation indicates where on the test image a certain point on the training image corresponds. For this reason, for example, when using the four corners of the training image as the region indicating the subject region photographed in the training image, the coordinates are converted into the coordinates on the test image using the affine transformation matrix or the homography transformation that estimates the points of the four corners, respectively. By doing so, it is possible to determine the existence area of the subject on the test image.

  With this configuration, it is possible to determine where the corresponding training image region, that is, the region of the subject exists on the test image.

<Third Embodiment>
Next, a subject recognition apparatus according to a third embodiment of the present invention will be described. In the second embodiment, by calculating an affine transformation or homography transformation matrix from inlier corresponding points, it is possible to estimate where the corresponding training image region, that is, the region of the subject exists on the test image.

  However, there is a problem that it is unclear what posture and position the subject of the test image is photographed with respect to the subject of the corresponding training image. The third embodiment solves this problem.

  FIG. 4 is a block diagram showing the configuration of the subject recognition apparatus according to the embodiment. In this figure, the same parts as those in the apparatus shown in FIG. The apparatus shown in this figure is different from the apparatus shown in FIG. 3 in that a relative posture estimation unit 11 is provided.

Next, the operation of the subject recognition apparatus 1 shown in FIG. 4 will be described. The relative posture estimation unit 11 inputs the inlier corresponding point used by the scoring unit 9 for calculating the score. The relative posture estimation unit 11 first calculates a basic matrix using the inlier corresponding points. Next, the relative posture estimation unit 11 estimates the focal length of the camera that captured each of the test image and the training image. Next, the relative translation and rotation of the two cameras are calculated from the focal length of the camera that captured the basic matrix, the test image, and the training image. Here, the basic matrix, the focal length, and the relative translation and rotation can be estimated by the method described in Reference 3, for example.
Reference 3: Yamada, Kanazawa, Kanaya, Sugaya, “Latest algorithm for three-dimensional reconstruction from two images”, IPSJ Research Report, CVIM, [Computer Vision and Image Media] 2009-CVIM-168 (15), 1 -8, 2009-08-24

  Finally, the relative posture estimation unit 11 outputs the estimated relative translation and rotation as a relative posture, and ends the process. In addition, if the relative attitude to the camera that has captured the test image is obtained from the camera that has captured the training image, the three-dimensional coordinates of the inlier corresponding point can also be calculated. The three-dimensional coordinates may be calculated together in the relative posture estimation unit and output in addition to the relative posture.

  With this configuration, it is possible to estimate in what posture and position the subject of the test image was photographed with respect to the subject of the corresponding training image.

  As explained above, after voting all tentative corresponding points to a voting space delimited by cells of a predetermined size, indicating the geometric transformation between the test image and the training image, the provisional correspondence belonging to a cell whose number of votes is a predetermined number or more By applying a predetermined geometric model to the point group using only the position coordinates (x, y) of the corresponding points constituting the provisional corresponding points, the erroneous corresponding points are removed at high speed and accurately. It is possible to estimate which subject is captured by the test image.

  The subject recognition device in the above-described embodiment may be realized by a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  Even when attribute information other than position information of local features is not correctly obtained, it can be applied to a use in which it is indispensable to correctly and correctly estimate which subject has been captured by removing erroneous corresponding points at high speed.

  DESCRIPTION OF SYMBOLS 1 ... Subject recognition apparatus, 2 ... Pre-processing part, 3 ... Local feature extraction part, 4 ... Local feature-value, attribute information storage part, 5 ... Temporary corresponding point determination part, 6. .. Weak geometric verification unit, 7 ... Strong geometric verification unit, 8 ... Reliability estimation unit, 9 ... Scoring unit, 10 ... Positive correspondence 2D geometric transformation matrix estimation unit, 11 ... Relative posture estimation unit, 20 ... database (DB), 30 ... training unit, 40 ... recognition unit

Claims (7)

An image information storage unit that stores local feature amounts extracted from a plurality of training images obtained by imaging a subject and attribute information indicating attributes of the local feature amounts in association with each training image;
A local feature extraction unit that extracts a local feature amount and the attribute information from a test image obtained by imaging a subject to be recognized;
A provisional corresponding point determining unit that compares the local feature amount extracted from the test image with the local feature amount stored in the image information storage unit to determine a provisional corresponding point;
Wherein constructing the attribute information of the type of number Vote separated by size of a given cell of the same number of dimensions and space for said tentative corresponding point all the relevant by calculating the amount of change between the attribute information A weak geometry verifier voting against a cell;
Based on the result of the voting, an inlier corresponding point that is not an outlier is obtained by fitting the provisional corresponding point to a predetermined geometric model and estimating a geometric transformation parameter for all cells having a predetermined number of votes or more. Strong geometry verification part,
Calculating a reliability of the inliers corresponding points, a reliability estimation unit value of the reliability is rejected low inliers point group than a predetermined value,
A scoring unit that outputs the sum of the inlier corresponding points not rejected as a score of the training image. By obtaining a geometric transformation matrix of affine transformation or homography transformation from the inlier corresponding points that have not been rejected, the information indicating which point on the test image corresponds to the point on the training image is output. The subject recognition apparatus according to claim 1, further comprising a corresponding 2D geometric transformation matrix estimation unit. The basic matrix was calculated from the inlier corresponding points that were not rejected, the focal length of the camera that captured each of the test image and the training image was estimated, and each of the basic matrix, the test image, and the training image was captured The subject recognition apparatus according to claim 1, further comprising a relative posture estimation unit that calculates and outputs a relative translation and rotation of the camera from a focal length of the camera. A subject recognition apparatus including an image information storage unit that stores a local feature amount extracted from a plurality of training images obtained by imaging a subject and attribute information indicating an attribute of the local feature amount in association with each training image is performed. An object recognition method,
A local feature extraction step of extracting a local feature amount and the attribute information from a test image obtained by imaging a subject to be recognized;
A provisional corresponding point determination step of determining a provisional corresponding point by comparing the local feature amount extracted from the test image with the local feature amount stored in the image information storage unit;
Wherein constructing the attribute information of the type of number Vote separated by size of a given cell of the same number of dimensions and space for said tentative corresponding point all the relevant by calculating the amount of change between the attribute information Weak geometry verification step for voting against the cell;
Based on the result of the voting, an inlier corresponding point that is not an outlier is obtained by fitting the provisional corresponding point to a predetermined geometric model and estimating a geometric transformation parameter for all cells having a predetermined number of votes or more. A strong geometry verification step;
Calculating a reliability of the inliers corresponding points, a reliability estimation step the value of the reliability is rejected low inliers point group than a predetermined value,
A scoring step of outputting the sum of the inlier corresponding points not rejected as a score of the training image. By obtaining a geometric transformation matrix of affine transformation or homography transformation from the inlier corresponding points that have not been rejected, the information indicating which point on the test image corresponds to the point on the training image is output. The subject recognition method according to claim 4, further comprising a corresponding 2D geometric transformation matrix estimation unit step. The basic matrix was calculated from the inlier corresponding points that were not rejected, the focal length of the camera that captured each of the test image and the training image was estimated, and each of the basic matrix, the test image, and the training image was captured The object recognition method according to claim 4, further comprising a relative posture estimation step of calculating and outputting a relative translation and rotation of the camera from a focal length of the camera.   A subject recognition program for causing a computer to function as the subject recognition device according to any one of claims 1 to 3.

Images