JP6674393B2 - Feature amount registration device, method and program - Google Patents

Description

  The present invention relates to a feature amount registration device, method, and program that can obtain a robust feature amount even when there is a view change.

  As in Non-Patent Documents 1 and 2, there is a technique for performing an object search using local features extracted from an image. In these techniques, local features are extracted from images registered in advance in a database, and a DB is constructed according to a method of creating an arbitrary database (hereinafter, abbreviated as DB). At the time of execution, an image obtained by photographing an object with a photographing means such as a camera is used as a query image to extract local features. By matching these features with features in the DB, the similarity between the query image and the DB image is calculated, and an object search result is obtained.

  In order to speed up the search process and reduce the amount of memory used, the feature vector of the feature point is quantized into a representative vector called a Visual Word (Visual Word; hereinafter, abbreviated as VW). At that time, the quantization is generally quantized to VW which is the nearest neighbor of the feature vector of the feature point. However, the feature vector extracted from the photographed image may change due to a difference in the photographing viewpoint from the time of registration and various factors such as blurring and blurring of the photographed image. Therefore, even a feature vector extracted from the same position of the object registered in the DB may be quantized to a different VW. This is called a quantization error.

  To address this problem, in Non-Patent Document 1, the effect of quantization errors is reduced by assigning feature vectors of feature points to a plurality of VWs near k (Soft Assignment) and registering them in the DB. .

  Also, in Non-Patent Document 2, an original feature vector is converted into a more compact binary vector, and stored together with VW in a database. Then, the distance between the binary vectors of the features having the same VW on the query side and the DB side is measured and used for weighting the similarity score. This distance is also used for determining feature matching. That is, it is determined that the matching is performed only when the distance is within the threshold value or that the matching is performed only when the feature on the database side is located near k with respect to the feature on the query side. Furthermore, by using the orientation and scale of the feature points, matches having no geometric consistency are filtered to improve search accuracy.

Philbin, James, et al. "Lost in quantization: Improving particular object retrieval in large scale image databases." Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 2008. Jegou, Herve, Matthijs Douze, and Cordelia Schmid. "Hamming embedding and weak geometric consistency for large scale image search." Computer Vision-ECCV 2008. Springer Berlin Heidelberg, 2008. 304-317.

  However, in the related arts such as Non-Patent Literatures 1 and 2 described above, no consideration has been given to changes in views on the query side and the DB side. In other words, although the local features have robustness of feature point detection and feature amount robustness when the view changes, no consideration has been given to them.

  Here, a view is an image of an object such as the position and orientation of the camera viewpoint, the light source position and direction, the focus setting, the camera movement (blurring) state, and the like when the object is captured in the query image and the DB image. It refers to any condition that changes the appearance when an image is obtained. In addition, the image is not limited to an image obtained by physically photographing an actual object with a camera (actual photograph), but may be an image synthesized by a computer graphics (hereinafter abbreviated as CG) method, the CG method, and the like. It also includes an image obtained by combining actual photographs.

  In view of the above-described problems of the related art, an object of the present invention is to provide a feature amount registration apparatus, method, and program capable of obtaining a robust feature amount even when there is a view change.

  In order to achieve the above object, the present invention is a feature amount registration apparatus for obtaining a registered feature set as a registration target from an input image, wherein the input image and / or one or more obtained by processing the input image are obtained. A pseudo-query feature generation unit that generates a pseudo-query feature set from the processed image, a registered feature candidate generation unit that generates a registered feature candidate set, and each element of the pseudo-query feature set and each element of the registered feature candidate set A similarity score for obtaining a sum of similarity scores between the elements of the pseudo-query feature set for each element of the registered feature candidate set based on a matching result obtained by the feature matching unit. Determining an element of the registered feature set using an element of the registered feature candidate set selected based on the calculated similarity score sum; Characterized by and a registered feature generation unit for obtaining the registration feature sets.

  The present invention is also a feature registration method for obtaining a registered feature set as a registration target from an input image, wherein a pseudo query feature is obtained from the input image and / or one or more processed images obtained by processing the input image. A pseudo query feature generating step for generating a set, a registered feature candidate generating step for generating a registered feature candidate set, and a feature for performing matching between each element of the pseudo query feature set and each element of the registered feature candidate set. A similarity score calculating step of calculating a sum of similarity scores between the elements of the pseudo query feature set for each element of the registered feature candidate set based on a matching result obtained by the feature matching step; By determining the elements of the registered feature set using the elements of the registered feature candidate set selected based on the sum of similarity scores, the registration is performed. Characterized in that it comprises a registration feature generation determining a feature set, the. Furthermore, the present invention is characterized in that it is a program that causes a computer to function as the feature amount registration device.

  According to the present invention, a robust feature can be obtained even when there is a view change.

FIG. 2 is a functional block diagram of a feature amount registration device according to one embodiment. 6 is a flowchart of an operation of the feature amount registration device according to one embodiment. FIG. 4 is a functional block diagram illustrating details of a pseudo query feature generation unit according to one embodiment. FIG. 7 is a diagram for explaining a relationship between distractors in a case where images have the same object geometric transformation relationship (and a relationship between the same object and the same portion as the reverse).

  FIG. 1 is a functional block diagram of a feature registering apparatus according to one embodiment. As shown in the drawing, the feature amount registration device 10 includes a pseudo query feature generation unit 1, a registered feature candidate generation unit 2, an iterative processing unit 3, and a feature amount storage unit 4, and the iterative processing unit 3 further includes a feature matching unit 31, A similarity score calculator 32 and a registered feature generator 33 are provided.

  First, the overall operation of the feature amount registration apparatus 10 is to read an image as an input and output a registered feature set suitable for the image in which a view change is assumed to have occurred. According to the present invention, a registered feature set is output as a feature set that maximizes a similarity score obtained through feature matching even when a change in view is expected between the query side and the DB side. It is possible.

  For example, if the view change is specifically a change in the position and orientation of the camera at the time of capturing an image, the registered feature set obtained by the feature amount registration device 10 indicates the position of the camera at the time of capturing the image. Even if the posture changes in various ways, the overall optimization is performed over the fluctuation range, and the feature matching can be robustly performed regardless of the position and the posture when the image is taken.

  Furthermore, according to an embodiment of the present invention, it is possible to output a registered feature set assuming that a similarity score is maximized on the assumption that a Distractor (distractor, an option of an incorrect answer) exists. It is. Here, in the feature search problem, generally, an incorrect answer option for the feature on the query side, that is, a distractor is unavoidable, but in the present invention, the similarity score on the assumption of the presence of the distractor is used. It is also possible to output a registered feature set that aims to maximize. Details will be described later.

  Note that the registered feature set output by the feature amount registration apparatus 10 means a feature set suitable for registering the feature set of the image in the DB from the above viewpoint, and is not necessarily registered as a use (DB or other (Permanently or temporarily recorded on any medium).

The framework (main processing flow) of the feature amount registration device 10 is as follows. That is, the set of N DB features (registered features) to be obtained is X = {x ₁ , x ₂ ,..., X _N }, and the set of any M pseudo query features is Q = {q ₁ , q ₂ , …, Q _M }, and let D = {d ₁ , d ₂ ,…, d _L } be a set of any L distractors and obtain the maximum similarity score from Q under any constraints It is a framework for generating X that can be done. (Note that, as described above, the use of the distractor set D may be omitted in some embodiments.)

  The above is the overall outline. Regarding the flow of data exchange between the functional blocks shown in FIG. 1, the outline of the processing contents of each unit is as follows.

  The pseudo-query feature generation unit 1 reads an image as an input to the feature amount registration device 10, generates a pseudo-query feature set Q, and outputs it to the feature matching unit 31. In some embodiments, the pseudo-query feature generation unit 1 may further output the generated pseudo-query feature set Q to the registered feature candidate generation unit 2 as shown by a line L21, or as shown by a line L31. Alternatively, distractor information (described later) obtained at the time of generation may be output to the iterative processing unit 3.

  The registered feature candidate generation unit 2 generates a registered feature candidate set to be used as a candidate for obtaining a registered feature set as a final output from the feature amount registration device 10, and outputs the set to the feature matching unit 31. The input for generating the registered feature candidate set can be any of the embodiments as indicated by lines L21, L22, L23, and the pseudo feature obtained from the pseudo query feature generation unit 1 as indicated by the line L21. A query feature set may be used, a registered feature set obtained from the registered feature generating unit 33 may be used as shown by a line L22, or the feature amount may be stored in advance in the feature amount storage unit 4 as shown by a line L23. The feature set that has been set may be used.

  The iterative processing unit 3 executes the framework described above, and receives a pseudo query feature set obtained from the pseudo query feature generation unit 1 and a registered feature candidate set obtained from the registered feature candidate generation unit 2 as inputs. Then, a predetermined number N of registered feature sets are output as a final result by an iterative process. Although details will be described later with reference to FIG. 2, each element (qi) of the pseudo-query feature set and each element (yj) of the registered feature candidate set in the feature matching unit 31 and the similarity score calculation unit 32 as a first iteration process. ), A matching result and a similarity score are obtained, and a second repetition process based on the obtained matching result and the similarity score is performed by the registered feature generation unit 33 to obtain N matching results. Is obtained. In each of the second iterations, an instruction to update the matching result and the similarity score obtained as shown in FIG. 1 is output from the registered feature generation unit 33, and the matching result and the similarity score are updated. Is performed.

  Each of the units 31, 32, and 33 executes the element processing in each iteration of the iterative processing unit 3, respectively. The feature matching unit 31 performs matching between the pseudo query feature set obtained from the pseudo query feature generation unit 1 and the registered feature candidate set obtained from the registered feature candidate generation unit 2, and calculates a matching result as a similarity score calculation. Output to the unit 32. The similarity score calculation unit 32 calculates a similarity score based on the obtained matching result and outputs the similarity score to the registered feature generation unit 33. Based on the obtained similarity score, the registered feature generation unit 33 determines a total of N elements of the registered feature set as the final output, one by one. As described above, the feature matching unit 31 and the similarity score calculating unit 32 perform the first iterative process, and the registered feature generating unit 33 performs the second iterative process.

  In one embodiment, as shown by the line L22, the registered feature set obtained by the iterative processing unit 3 may be further output to the registered feature candidate generating unit 2. In this case, the repetition processing is performed a predetermined number of times (A times) in the entire feature amount registration apparatus 10, and the repetition processing unit 3 registers the a-th time (a = 1, 2,..., A-1). By using the registered feature set output from the feature generating unit 33 as an input to the next (a + 1) th registered feature candidate generating unit 2 (and the feature matching unit 31), the registered feature generation is finally performed at the Ath time. The registered feature obtained from the unit 33 is used as an output of the feature amount registration device 10. Therefore, in FIG. 1, when the repetition is not performed (when A = 1), the line L22 is omitted.

  In some embodiments, the iterative processing unit 3 also receives the distractor information obtained from the pseudo query feature generation unit 1 as indicated by the line L31 or the distractor information obtained from the feature amount storage unit 4 as indicated by the line L32. By using tractor information, a registered feature set in consideration of a distractor may be generated.

  The feature amount storage unit 4 is equivalent to a so-called general DB at the time of image search, and stores a feature amount of each image in which each object is captured in advance as a result of a manual operation or the like. In one embodiment, as shown by the line L23, the feature amount is provided to the registered feature candidate generating unit 2 for reference. The feature amount storage unit 4 may further store the destructor information for each of the feature amounts to be stored, as well as those obtained in advance by manual work, etc., as shown in a line L32 in one embodiment. In this manner, the distractor information may be provided to the repetition processing unit 3 for reference.

  FIG. 2 is a flowchart of the operation of the feature amount registration device 10 according to one embodiment. Hereinafter, the details of the processing of each unit in FIG. 1 will be described while describing each step in FIG. 2.

In step S1, a pseudo query feature set Q = {q ₁ , q ₂ ,..., Q _M } composed of M elements is generated from the image obtained as an input by the pseudo query feature generation unit 1, and then, in step S2 Proceed to.

  FIG. 3 is a functional block diagram showing details of the pseudo query feature generation unit 1. As shown, the pseudo query feature generation unit 1 includes an image processing unit 11 and a generation unit 12. The image P0 (to be described as an image P0 for explanation) input to the pseudo query feature generation unit 1 is input to the generation unit 12 and the image processing unit 11. The image processing unit 11 obtains processed images P1, P2,..., Pb as those obtained by subjecting the image P0 to b processing processes pr1, pr2,. Is output.

  For each of the b processings in the image processing unit 11, a predetermined processing is determined in advance by giving parameters such as the type of the processing and the strength of the type of processing. In the processing, a predetermined process may be determined in advance to cover a view change expected when performing an image search or the like based on a registered feature set output by the feature amount registration device 10.

  For example, types of processing may include those that perform geometric transformation such as similarity transformation, affine transformation, and projective transformation. In the geometric transformation processing, when a 3D model of an object appearing in the input image P0 is further provided, an image in which a view is changed after attaching a texture to the model may be generated. That is, a 3D model of an object including texture information in each of the element polygons may be given as predetermined data and generated as a CG model (generation by rendering) which is an existing technology. Further, when noise, blur, occlusion, and the like are assumed as the view change, any image processing such as noise addition, smoothing, and masking may be employed as the processing.

  Also, the parameters of the processing can be given as a transformation matrix for performing the geometric transformation in the case of geometric transformation, and as the strength of the imparted noise in the case of adding noise. When obtaining one processed image, two or more processing processes may be combined.

The generation unit 12 extracts a feature amount from the input image P0 and a series of processed images P1, P2, ..., Pb obtained from the image processing unit 11, and extracts all of the extracted feature amounts. Output as a pseudo-query feature set Q = {q ₁ , q ₂ ,..., Q _M }. As the extracted feature amount, a predetermined type of local feature amount (a feature amount calculated near a local point of a feature point (key point)) such as a well-known SIFT feature amount or SURF feature amount can be adopted.

In the generator 12, the input image P0 is the processed image P1, P2 without, ..., pseudo query feature set by extracting a feature from Pb only _{Q = {q 1, q 2} , ..., q _M } may be output. Similarly, instead of processing one input image P0 as input using the image processing unit 11, a plurality of images obtained by actually photographing the same object under various viewing conditions are generated. May be input.

  In step S1, the pseudo query feature generation unit 1 further performs processing including geometric transformation in the image processing unit 11; distractor information (incorrect information) (or equivalent processing when there is a geometric transformation relationship). As the information, the same target / same place information) can also be output. In the present invention, the distractor means all features other than the correct DB feature for one query feature (qi). Specifically, (1) an image (1) irrelevant to the query image (input image P0) (2) All the features extracted from an image of a target different from the image P0) and (2) a captured image of the same target as the query image (ie, processed images P1, P2,..., Pn) Are defined as a set D (qi) (a set determined for each query feature qi) that combines all features that do not satisfy the matching constraint conditions extracted from. Here, the DB feature that can be said to be “correct” for a certain query feature is a local feature that describes the same part of the same object, but is not exactly the same due to a change in imaging conditions (view). Therefore, features that are sufficiently “similar” are considered to be correct, but the “similar” criteria are matching constraints. A specific example of the matching constraint condition will be described later in the description of the feature matching unit 31.

  With respect to the destructor defined for each query feature qi as both (1) and (2) above, (1) is evident and a schematic example of (2) of the destructors with reference to FIG. Show. In FIG. 4, n = 100 processed images P1 to P100 are generated as a result of geometrically transforming the image P0 as shown on the right side with respect to the query image P0 on the left side and are used as DB features (registered feature candidates). An example is shown. With respect to the features qi0 extracted from the position p0 of the query image P0 indicated by 〇 (the position of the top of the roof of “house” as a target example), those that satisfy the matching constraints are also indicated by 〇 with respect to the images P1 to P100. A feature yj1 extracted from the corresponding position p1 in the image P1, a feature yj2 extracted from the corresponding position p2 in the image P2, a feature yj3 extracted from the corresponding position p3 in the image P3,..., A corresponding position p100 in the image P100. , And 100 features. Therefore, all the features extracted from the processed images P1 to P10 except for the 100 features satisfying the matching constraint condition (for example, the feature yj1e at the position p1e in the image P1), which are indicated by Δ in FIG. In FIG. 4, only a part of the destructor is illustrated as a mark, and only a part of the code is given.) Is the distractor of the feature qi0 of the query image P0.

  When the image processing unit 11 generates a processed image Pk (k = 1, 2,...) From the input image P0 by geometric transformation Hk as in the example of FIG. Whether the characteristic qi0 of the image Pk and the characteristic yk1 of the position pk of the image Pk correspond to each other is determined by “|| pk−Hk * p0 || <TH” (where |||| is the Euclidean distance If (absolute value) and TH are threshold values, it can be determined to be a corresponding location (the same target has the same location). That is, when it is determined that the position Hk * p0 where the position p0 of the image P0 is mapped by the geometric transformation Hk expressed by the homography matrix or the like is sufficiently close to the position pk, it can be determined to be the corresponding position. Conversely, the feature amount extracted at any position other than the sufficiently close position is a distractor for the feature qi0 at the position p0 of the image P0. Therefore, when a processed image is obtained by performing a transformation including a geometric transformation in the image processing unit 11, the distractor information can be output by specifying an object having no correspondence between the same object and the same portion in this manner. .

In step S2, the registration feature candidate set registered feature candidate generating unit 2 is composed of K components _{Y = {y 1, y 2} , ..., y K} from the generate, the process proceeds to step S3. Note that the registered feature candidate set Y is such that by selecting N elements from among them, N registered feature sets X = {x ₁ , x ₂ ,..., X _N } are obtained. Therefore, generation is performed so that K> N with respect to the number K of elements.

  In the step S2, the following embodiments are possible as a method of generating the information.

  In one embodiment, as shown by the line L21 in FIG. 1, the pseudo query feature set Q generated in step S1 is adopted as it is as the registered feature candidate set Y, that is, generated as Y = Q.

  In one embodiment, as shown by a line L22 in FIG. 1, the registered feature set generated by the registered feature generating unit 33 is adopted as a registered feature candidate set Y. This embodiment is equivalent to repeating the entire flow chart of FIG. 2 a predetermined number of times and updating the registered feature set obtained in step S10 a predetermined number of times as a registered feature set as a final output.

  In one embodiment, as shown by a line L23 in FIG. 1, a predetermined number K of feature quantities extracted from the feature quantity storage unit 4 is adopted as a registered feature candidate set Y. The extraction may be performed at random, but it is preferable to extract the feature amount so as not to be biased in the feature space.

  Since the feature amount storage unit 4 is a so-called general DB, the feature amount of an arbitrary image (unknown image) input to the pseudo-query feature generation unit 1 is stored in the destructor information without prior knowledge. It cannot be defined as a feature value in the storage unit 4. As described above with reference to FIG. 4, when there is a relationship of geometric transformation, the distractor information is defined as having no correspondence between the same object and the same place. However, if there is such prior knowledge, the distractor information may be defined in at least a part of the feature amount stored in the feature amount storage unit 4. The distractor information can be provided to the iterative processing unit 3 as shown by a line L32 in FIG.

  In step S3, the similarity score calculation unit 32 sets zero as an initial value of the similarity score sum score (yj) calculated for each registered feature candidate yj by the following double loop processing S4 to S9, Proceed to S4. (If the value of the similarity score sum score (yj) is determined to be matching in step S6 in the double loop processing, the value for the corresponding yj is added and updated, thereby exiting the double loop processing. At this point (at the time when step S10 is reached), the value is obtained.)

  Step S4 is a dummy step indicating that the following steps S5 to S9 are repeatedly performed for each pseudo query feature qi generated in step S1, and the process proceeds to step S5 after setting the current processing target qi. Step S5 is a dummy step indicating that the following steps S6 to S8 are repeatedly performed for the registered feature candidate yj generated in step S2, and the process proceeds to step S6 after setting the current processing target yj.

  In step S6, the feature matching unit 31 determines whether or not the pseudo query feature qi set as the processing target matches the registered feature candidate yj, and then proceeds to step S7. In step S7, when a matching determination is obtained in the latest step S6, the similarity score calculation unit 32 calculates the current similarity score sum score (yi) for the registered feature candidate yi set as the processing target. After adding and updating the value, the process proceeds to step S8. (If a matching determination is not obtained in the latest step S6, addition and updating are not performed in step S7, and the process directly proceeds to step S8.)

  It should be noted that details of each of the processes in steps S6 and S7 in the double loop will be described after the description of the double loop process.

  In step S8, it is determined whether or not the processing in steps S5 to S7 has been completed for all the registered feature candidates yj.If the processing has been completed, the process proceeds to step S9.If not completed, the process returns to step S5. An unprocessed registered feature candidate yj is set as a processing target, and the above is repeated. In step S9, it is determined whether or not the processing in steps S4 to S8 has been completed for all the pseudo query features qi.If the processing has been completed, the process proceeds to step S10.If not completed, the process returns to step S4. The unprocessed pseudo query feature qi is set as a processing target and the above is repeated.

  As described above, at the time when the double loop processing for each qi and each yj in steps S4 to S9 is completed and the process reaches step S10, a series of matching results by the feature matching unit 32 and each yj by the similarity score calculation unit 32 The final similarity score sum score (yj) has been found. Hereinafter, details of the matching processing and the similarity score calculation will be described in order.

First, with regard to the matching processing in step S6, the feature matching unit 31 determines, for each element qi of the pseudo query feature set Q, the element yj of the registered feature candidate set Y based on a constraint condition (matching determination condition) specified in advance. Find the match (qi) that matches from among them. In other words, if the set elements are formally described by listing them with the symbol “{}”, the following matching result will be obtained when the double loop processing is completed. This is called result list 1.
match (q ₁ ) = {y ₁₁ , y ₁₂ , ...}, match (q ₂ ) = {y ₂₁ , y ₂₂ , ...}, ...
match (q _i ) = {y _i1 , y _i2 , ...}, ...., match (q _N ) = {y _N1 , y _N2 , ...}… (result list 1)

In step S6, various types of constraint conditions for the matching determination are possible, for example, the following. Any one or an arbitrary combination of the respective methods may be used as the constraint condition.
(Method 1) The feature (qi) on the query side and the feature (yj) on the DB side are quantized to the same VW.
(Method 2) In the feature vector space in which the distractor exists, the DB-side feature vector (yj) is located near k with respect to the query-side feature vector (qi).
(Method 3) The distance between the query-side feature vector (qi) and the DB-side feature vector (yj) is within a threshold.
(Method 4) The difference between the query-side features and the DB-side features of the orientation is within a threshold value.
(Method 5) The ratio between the scale of the query side and the scale of the DB side is within a threshold value.

  Note that, in the second method, the vicinity of k means a method of matching qi with yj included in the k-th place having a higher similarity to qi. That is, in the method 2, the k-th higher-order similarity with the query-side feature vector (qi) is obtained as the matching result match (qi).

  Note that the registered feature candidate set Y may include a plurality of features y extracted from the same position as the feature (yj) of the DB currently focused on, It may be excluded from the ranking of time. Specifically, the following [1] to [3] may be performed.

[1] A list of other y's (coordinates of the same object and the same portion) whose coordinates match yj is made in advance. For example, if the pseudo query feature set is extracted from the images P1 to P100 (excluding the image P0) in the example of FIG. 4, list (yj1) = {yj2, yj3, ..., yj100}. . Similarly, list (yj2) = {yj1, yj3, yj4, ..., yj100}. list (yj3) and the like can be similarly defined.
[2] Measure the distance between the feature vectors of qi and each y, and create a ranking sorted in ascending order. At this time, the elements of list (yj) are not included in the ranking. (Includes yj itself.) That is, a series of y (list (yj) having the same subject and the same location as each of the currently focused yj is excluded from the ranking. (In other words, only the currently focused yj and the other y not belonging to list (yj) are subject to similarity ranking creation between qi.)
[3] If yj is included in the top k items in the ranking obtained as described above, it is regarded that the constraint condition is satisfied, and qi and yj are matched.

  Through the above procedures [1] to [3], it is possible to reproduce a situation where the top k cases contain only one true answer at most. This corresponds to an exceptional process caused by including a large number of images obtained by converting the same image in the matching target in the present invention.

  The “feature orientation difference and / or scale ratio” described above and in the methods 4 and 5 means a difference and a ratio regarding the orientation and scale obtained when a feature vector is obtained as a SIFT feature amount or the like.

  Also, in step S7, the similarity score calculation unit 32 adds and updates the similarity score sum score (yj) for the registered feature candidate yj as a contribution of the qi when a matching determination is obtained in the latest step S6. As a result, a state is obtained in which the final similarity score sum score (yj) is obtained when the double loop processing is completed.

First, the obtained similarity score sum score (yj) is obtained by using the result list 1 obtained for each pseudo query feature qi described above as a matching result match (yj) viewed from each registered feature candidate yj as follows. After rewriting the result list 2, the similarity score sum of qi belonging to the result list 2 is obtained. Note that the result list 1 and the result list 2 are equivalent in content.
match (y ₁ ) = {q ₁₁ , q ₁₂ , ...}, match (y ₂ ) = {q ₂₁ , q ₂₂ , ...}, ...
match (y _j ) = {q _j1 , q _j2 , ...}, ...., match (y _K ) = {q _K1 , q _K2 , ...}… (result list 2)

Specifically, in the processing for obtaining the similarity score sum in the loop processing, the element qi belonging to the result list 2 to be obtained based on the calculation method specified in advance (the element qi determined to be matched in step S6) ), The similarity score sum score (y _j ) of each registered feature candidate y _j (the sum of similarity scores between each element q in the list match (y _j ) and the registered feature candidate y _j ) Will be calculated. Various weights can be used for the addition, for example, the following.

(Calculation method 1) “1” (fixed value) is used as a weighting value. That is, by adding “1” to the score sum score (yi) each time yj and qi are determined to be matched in step S6, at the time when the loop processing is completed, the following score sum score (yj) Is determined. That is, as the similarity score sum is large number of elements match (yj) in the result list 2 score (y _j) is obtained as large.
score (y _j ) = number of elements belonging to list match (y _j ).

(Calculation method 2) As a weighting value, tf-idf (Term Frequency-Inverse Document Frequency) which is an existing method is used. That is, when the tf-idf value of the feature vector q belonging to the list match (y _j ) is expressed as tfidf (q), the similarity score sum score (y _j ) Is obtained. Note that the value tf-idf (q) can be obtained according to an existing method after quantizing the feature vector q into VW.

(Calculation method 3) The expression “kthd ² -d ² ” is used as a weighting value. Here, d is the distance between the query-side and reference-side feature vectors, and kthd is the distance between the feature vectors of the query-side feature and the k-th closest feature. That is, if the value of “kthd ² -d ² ” obtained for the feature vector q belonging to the list match (y _j ) is written as f (q), as in the case of tf-idf above, Will be asked.

Here, “kthd ² -d ² ” is a score function when an image classification method called Local Naive-Bayes Nearest-Neighbor disclosed in Non-Patent Document 3 below is applied to image search. kthd ² -d ² may be calculated using a feature vector even after quantization to VW, or using a distance between vectors called Hamming embedding which is a compact conversion of the feature vector as in Non-Patent Document 1. You may.
[Non-Patent Document 3] McCann and DG Lowe, "Local naive bayes nearest neighbor for image classification," in Proc. Of CVPR, 2012.

  In calculation method 3, as described in (method 2) of the feature matching unit 31, a ranking is created by steps [1] to [3], and a k-th feature vector in the ranking is calculated. You may ask for it.

  As described above, the similarity score sum score (yj) can be calculated using not only the calculation methods 1 to 3 but also any other existing score function (weighting function) for image search.

  In step S10, the registered feature generating unit 33 sets an empty set (as an initial value of a registered feature set for obtaining a final result (the final result is output in step S15) in the following N repeated steps S11 to S14). After setting (φ), the process proceeds to step S11. Step S11 is a dummy step for indicating that the following steps S12 to S14 are repeated N times, and the process directly proceeds to step S12.

In step S12, the registered feature generating unit 33 uses the following (procedure 1) and (procedure 2) to add a new element x _n (n = 1, 2,,,, N) to the registered feature collection, and It represents the current number of repetitions of steps S12 to S14.), And then proceeds to step S13.

(Procedure 1) From among the registered feature candidates y _j , a registered feature candidate y _max having the maximum similarity score score (y _j ) is selected. That is, the index max is determined as follows in the expression.
max = argmax (score (y _j ))
(Procedure 2) A feature vector _xn to be added is obtained by averaging the feature vectors of the pseudo query feature set match (y _max ) associated with the registered feature candidate y _max .

In the same manner further in step 2 above, not only the feature vector x _n only, orientation o, scale s, and respectively averaged wherein at least one of the coordinates u, may be registered feature f _n. For example, when all of these and registration features registered feature is obtained as those listed _{f n = (x n, o} n, s n, u n) the respective amounts as.

  In the above procedure 2, the following additional embodiment is also possible. That is, by averaging the features in step 2, the sum of the similarity scores obtained from the pseudo query feature set may be reduced as a result. Therefore, the processes of the feature matching unit 31 and the similarity score calculating unit 32 are performed again using the features averaged in the procedure 2, and if the sum of the similarity scores is small as a result, the averaging is not performed. May be determined.

That is, the score (y _max) selected in the initial (Step 1), and score (x _n) by x _n obtained by averaging in (Step 2), by comparing the "score (y _max) > Score ( _xn ) ", the averaged _xn may not be adopted as a registration target. Conversely, when the similarity score sum increases, _xn averaged as described above may be employed as an element of the registered feature set. When calculating score (x _n ) using the averaged features, a provisional feature in which a plurality of averaged features are replaced with one averaged feature in the registered feature candidate set Y The processing of the feature matching unit 31 and the similarity score calculation unit 32 may be performed on the set Y ′ (that is, using the provisional set Y ′ instead of the set Y as input). For example, when the registration features are obtained by averaging _xn = (y1 + y2 + y3) / 3 and three, y3, y1, y2, and y3 may be replaced with one _xn . If it is determined that averaging is not performed, _ymax may be registered as _xn without averaging.

In step S13, the registration feature generation unit 33 of the immediately preceding step S12 in brute cord to y _max determined in (Step 1) Pseudo query feature set match (y _max) (matching result list pseudo query feature for y _max) Updates are performed to exclude each element from the pseudo query feature set match (y _j ) = {q _j1 , q _j2 , ...} associated with all other yj (including y _max itself), and match ( After updating the subtraction of the contribution to the score sum given from each element of y _max ) from the other similarity score sum score (y _j ), the process proceeds to step S14. The update to be subtracted means using `` match (yi) ＼ match (y _max ) '' excluding elements belonging to match (y _max ) from match (yi) for calculating score sum score (yi) , Which is equivalent to recalculating the score sum score (yi). “＼” Is a difference set operation, and the same applies to the following.

By the update in step S13, the registered feature candidate set Y is updated as follows, and the matching result and the similarity score sum obtained when the double loop processing of steps S4 to S9 is performed again are assumed to be substantially the same. The updated matching result and the similarity score sum can be obtained at high speed without performing the double loop processing again.
Y = Y＼match (y _max )

Assuming that the number of repetitions of steps S11 to S14 is the n-th, the pseudo query feature set match (y _j ) = (q _j1 , q _j2 ,... Updated in the n-th step S13 as described above. .} (Ie, the updated result list 2) and the updated similarity score sum score (y _j ) will be used in the next (n + 1) _-th determination in step S12. (It is assumed selected as y _max (y _[Selected] at least once), the score value score (y _[Selected]) because is zero are updated value in the iterative process since In addition, the list match (y _[selected] ) as the matching result is also an empty set, and is therefore excluded from the selection in (Procedure 1).)

  In step S14, the registered feature generation unit 33 determines whether or not a registered feature set including N elements has been obtained at the time, and if not, returns to step S11 to add a new element. The above is repeated, and if it is obtained, the process proceeds to step S15. In step S15, the registered feature generation unit 33 outputs the registered feature set including the N elements obtained in the above-described repeated steps S11 to S14 as a final result, and the flow in FIG. 2 ends.

  As described above, according to the present invention, a robust feature can be obtained even when there is a view change. In addition, in the feature vector space in which the distractor exists, a registered feature set that is highly likely to match query features in various views can be generated. Further, by preferentially selecting a registered feature that gives a higher similarity score, it is possible to improve the accuracy of image search for a large-scale database. The supplementary items are described below.

  (1) An embodiment in which the entire flow in FIG. 2 is repeated (an embodiment in which the input of the line L22 in FIG. 1 is performed in the second and subsequent iterations), that is, a registered feature set obtained by one process is newly registered. An embodiment in which the process is regarded as a feature candidate and the process is repeated until the end condition is satisfied can be, for example, as follows.

  For example, the first registered feature candidate is set to a pseudo query feature set (M> N), and M ′ (M> M ′> N) registered feature sets are obtained. The process of obtaining the M ″ (M ′> M ″> N) registered feature sets by regarding them as new registered feature candidates may be repeated until N registered features are obtained. Alternatively, the process may be performed by regarding the N registered features obtained in the first process as new registered feature candidates, and the process may be repeated until the sum of the resulting similarity scores no longer changes.

  (2) The effects of the present invention are expected as the matching constraints, the similarity score calculation method, the pseudo query feature set, and the destructor in the present invention conform to the search processing at the time of execution. In other words, the matching constraint conditions and the similarity score calculation method match the matching method and the similarity score calculation method used at the time of execution, and the pseudo query feature set is similar to the one obtained from the image actually taken by the user. However, it is desirable that the content of the element set to which the destructor relation is given and the number of the elements be similar to the actual search database.

  (3) The present invention can also be provided as a program that causes a computer to function as the feature amount registration device 10. The computer may have a well-known hardware configuration such as a CPU (Central Processing Unit), a memory, and various I / Fs, and the CPU executes an instruction corresponding to a function of each unit of the feature amount registration device 10. Will be done.

  10: feature registration device, 1: pseudo query feature generation unit, 2: registered feature candidate generation unit, 31: feature matching unit, 32: similarity score calculation unit, 33: registered feature generation unit, 4: feature storage unit

Claims (13)

A feature amount registration device for obtaining a registered feature set as a registration target from an input image,
A pseudo query feature generation unit that generates a pseudo query feature set from the input image and / or one or more processed images obtained by processing the input image;
A registered feature candidate generating unit that generates a registered feature candidate set;
A feature matching unit that performs matching between each element of the pseudo query feature set and each element of the registered feature candidate set based on the similarity between local feature amounts that do not include position information of the elements ,
For each element of the registered feature candidate set, a similarity score for obtaining a sum of similarity scores between the elements of the pseudo query feature set and at least the number of matches determined by the matching result by the feature matching unit is reflected A score calculator,
A registered feature generation unit for determining the registered feature set by determining an element of the registered feature set using an element of the registered feature candidate set selected based on the obtained similarity score sum. A feature registering apparatus characterized in that: Between at least some of the elements of the registered feature candidate set, the same target same location information defining that the feature is extracted from the same location of the same target is given in advance,
The said characteristic matching part and / or the said similarity score calculation part each perform the said matching process and / or the process which calculates | requires the said similarity score sum using the said same object same location information, respectively. 2. The feature amount registration device according to 1. The feature matching unit determines, for each first element of the pseudo query feature set, whether or not to match with each second element of the registered feature candidate set, by determining a similarity degree with the first element. When the ranking is within a predetermined upper rank in the registered feature candidate set, it is determined that the matching is performed,
When making a determination as to whether or not it is within the upper predetermined position , the determination target of the registered feature candidate set having a relationship of the same target same location information with a second element that is the target of the determination 3. The feature amount registration apparatus according to claim 2, wherein the second element other than the second element is excluded from the targets that can belong to the upper predetermined position . 4. Wherein the similarity score calculation unit, the feature quantity registration device according to claim 2 or 3, characterized in that obtaining the similarity score sum by a predetermined type of weighted sum. The feature registration device according to any one of claims 1 to 4 , wherein the registered feature generation unit selects the element of the registered feature candidate set that maximizes the obtained similarity score sum. The registered feature generation unit determines an element of the registered feature set based on each element matched by the feature matching unit with respect to an element of the selected registered feature candidate set. feature quantity registration device according to any one of claims 1 to 5. The registered feature generation unit determines the registered feature set element by averaging each element matched by the feature matching unit with respect to the selected registered feature candidate set element. The feature registering device according to any one of claims 1 to 6 , wherein A feature amount registration device for obtaining a registered feature set as a registration target from an input image,
A pseudo query feature generation unit that generates a pseudo query feature set from the input image and / or one or more processed images obtained by processing the input image;
A registered feature candidate generating unit that generates a registered feature candidate set;
A feature matching unit that performs matching between each element of the pseudo query feature set and each element of the registered feature candidate set,
A similarity score calculation unit that calculates a sum of similarity scores between elements of the pseudo query feature set for each element of the registered feature candidate set based on a matching result by the feature matching unit;
A registered feature generation unit that determines the registered feature set by determining an element of the registered feature set by using an element of the registered feature candidate set selected based on the obtained similarity score sum. ,
In the registered feature generation unit, for the elements of the selected registered feature candidate set, an average of each element matched by the feature matching unit is obtained, and the registered feature candidate set is determined by the averaged element. A similarity score sum calculated for the averaged element by applying the processing by the feature matching unit and the similarity score calculation unit to the provisionally updated set that has been provisionally updated, and the sum of similarity scores calculated for the selected registered feature candidate set If the element is smaller than the sum of the similarity scores of the elements, the element of the selected registered feature candidate set is adopted as the element of the registered feature set as it is, and if not smaller, the averaged element is the element of the registered feature set. A feature amount registration apparatus characterized in that the feature amount registration apparatus is adopted. A feature amount registration device for obtaining a registered feature set as a registration target from an input image,
A pseudo query feature generation unit that generates a pseudo query feature set from the input image and / or one or more processed images obtained by processing the input image;
A registered feature candidate generating unit that generates a registered feature candidate set;
A feature matching unit that performs matching between each element of the pseudo query feature set and each element of the registered feature candidate set,
A similarity score calculation unit that calculates a sum of similarity scores between elements of the pseudo query feature set for each element of the registered feature candidate set based on a matching result by the feature matching unit;
A registered feature generation unit that determines the registered feature set by determining an element of the registered feature set by using an element of the registered feature candidate set selected based on the obtained similarity score sum. ,
In the registered feature generation unit, the elements of the registered feature set are determined one by one, and each time the determination is performed, the contribution by the element of the pseudo query feature set corresponding to the determined element is determined from the similarity score sum. The similarity score sum is updated by subtraction, and the elements of the pseudo query feature set corresponding to the determined elements based on the updated similarity score sum are subsequently registered feature sets. A feature registration apparatus characterized in that the determination of the next element of the registered feature set is repeated a predetermined number of times by excluding the element from the reference target for determining the element. The registered feature candidate generating unit adopts a pseudo query feature set generated by the pseudo query feature generating unit or a registered feature already generated by the registered feature generating unit as the registered feature candidate set. The feature registering device according to any one of claims 1 to 9 , wherein Wherein each of the processed image as obtained by geometric transformation of the input image, and / or, as having been subjected to the rendering based on a predetermined 3D data including texture of the object is reflected in the input image, the resulting image The feature registration device according to any one of claims 1 to 10 , wherein A feature amount registration method for obtaining a registered feature set as a registration target from an input image,
A pseudo query feature generation step of generating a pseudo query feature set from the input image and / or one or more processed images obtained by processing the input image;
A registered feature candidate generating step of generating a registered feature candidate set;
A feature matching step of performing a matching between each element of the pseudo query feature set and each element of the registered feature candidate set based on the similarity between local feature amounts not including positional information of the elements ,
For each element of the registered feature candidate set, a similarity score for obtaining a sum of similarity scores between the elements of the pseudo query feature set and at least the number of matches determined by the matching result in the feature matching step is reflected Score calculation stage,
Using the elements of the registered feature candidate set selected based on the obtained similarity score sum to determine the registered feature set elements, thereby obtaining the registered feature set. A feature registration method. A program for causing a computer to function as the feature amount registration device according to any one of claims 1 to 11 .