JP2018120395A - Apparatus, method and program for registering feature amount - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an apparatus for registering a feature amount capable of acquiring a robust feature amount even in a case where there is a view change.SOLUTION: An apparatus 10 for registering a feature amount that determines a registration feature set as a registration object from an input image comprises: a pseudo-query feature generation unit 1 that generates a pseudo-query feature set from the input image and/or one or more processed images obtained by processing the input image; a registration-feature candidate generation unit 2 that generates a registration-feature candidate set; a feature matching unit 31 that carries out matching between each element of the pseudo-query feature set and each element of the registration-feature candidate set; a similarity score calculation unit 32 that determines a similarity score sum to elements of the pseudo-query feature set over each element of the registration-feature candidate set on the basis of a matching result from the feature matching unit 31; and a registration-feature generation unit 33 that determines the registration feature set by determining an element of the registration feature set using the element of the registration-feature candidate set selected based on the determined similarity score sum.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a feature amount registration apparatus, method, and program capable of obtaining a robust feature amount even when there is a view change.

  As in Non-Patent Documents 1 and 2, there is a technique for performing object search using local features extracted from an image. In these techniques, local features are extracted from images registered in advance in a database, and a DB is constructed according to an arbitrary database (hereinafter abbreviated as DB) creation method. At the time of execution, an image obtained by photographing an object with a photographing means such as a camera is used as a query image, and local features are extracted. An object search result is obtained by calculating the similarity between the query image and the DB image by matching these features with the features in the DB.

  In order to speed up the search process and reduce the amount of memory used, the feature vector of the feature point is quantized into a representative vector called Visual Word (visual word, hereinafter abbreviated as VW). In that case, it is generally quantized to VW, which is the nearest feature vector of the feature point. However, the feature vector extracted from the photographed image may change due to various factors such as a difference in photographing viewpoint from that at the time of registration, blurring and blurring of the photographed image. For this reason, even feature vectors extracted from the same position of an object registered in the DB may be quantized to different VWs. This is called a quantization error.

  To deal with this problem, Non-Patent Document 1 alleviates the influence of quantization errors by assigning feature vectors of feature points to a plurality of VWs near k (Soft Assignment) and registering them in the DB. .

  In Non-Patent Document 2, the original feature vector is converted into a more compact binary vector and stored in the database together with the VW. The distance between the binary vectors of the features having the same VW on the query side and the DB side is measured and used to weight the similarity score. In addition, this distance is also used to determine feature matching. That is, it is determined that matching is performed only when the distance is within the threshold value, or matching is performed only when the database-side feature is located in the vicinity of k for the query-side feature. Furthermore, by using the orientation and scale of the feature points, matches with no geometric consistency are filtered to improve the search accuracy.

Philbin, James, et al. "Lost in quantization: Improving particular object retrieval in large scale image databases." Computer Vision and Pattern Recognition, 2008. CVPR 2008. IEEE Conference on. IEEE, 2008. Jegou, Herve, Matthijs Douze, and Cordelia Schmid. "Hamming embedding and weak geometric consistency for large scale image search." Computer Vision-ECCV 2008. Springer Berlin Heidelberg, 2008. 304-317.

  However, in the conventional technologies such as Non-Patent Documents 1 and 2 described above, no consideration is given to changes in view on the query side and the DB side. That is, although local features have robustness of feature point detection and feature amount robustness when the view changes, they have not been considered.

  Here, the view refers to shooting an object such as the position and orientation of the camera viewpoint when shooting the object in the query image and DB image, the light source position and direction, the focus setting, the camera movement (blur) state, etc. An arbitrary condition for changing the appearance when an image is obtained is assumed. In addition, images are not limited to images obtained by physically shooting an actual object with a camera (actual shooting), but images synthesized by a computer graphics (hereinafter abbreviated as CG) method, the CG method, It also includes images obtained by combining live-action shots.

  In view of the above-described problems of the conventional technology, an object of the present invention is to provide a feature amount registration apparatus, method, and program capable of obtaining a robust feature amount even when there is a view change.

  In order to achieve the above object, the present invention is a feature amount registration apparatus for obtaining a registered feature set as a registration target from an input image, wherein the input image and / or one or more obtained by processing the input image A pseudo query feature generation unit that generates a pseudo query feature set from a processed image, a registered feature candidate generation unit that generates a registered feature candidate set, each element of the pseudo query feature set, and each element of the registered feature candidate set And a similarity score for calculating a similarity score sum between the elements of the pseudo-query feature set for each element of the registered feature candidate set based on a matching result obtained by the feature matching unit Determining an element of the registered feature set using a calculation unit and an element of the registered feature candidate set selected based on the calculated similarity score sum Characterized by and a registered feature generation unit for obtaining the registration feature sets.

  The present invention is also a feature registration method for obtaining a registered feature set as a registration target from an input image, wherein the pseudo query feature is obtained from the input image and / or one or more processed images obtained by processing the input image. A pseudo query feature generation stage for generating a set, a registered feature candidate generation stage for generating a registered feature candidate set, and a feature for performing matching between each element of the pseudo query feature set and each element of the registered feature candidate set A similarity score calculation step for obtaining a similarity score sum between each element of the registered feature candidate set and an element of the pseudo-query feature set based on a matching result obtained by the feature matching step; By determining the elements of the registered feature set using the elements of the registered feature candidate set selected based on the sum of similarity scores. Characterized in that it comprises a registration feature generation determining a feature set, the. Furthermore, the present invention is a program for causing a computer to function as the feature amount registration apparatus.

  According to the present invention, it is possible to obtain a robust feature amount even when there is a view change.

It is a functional block diagram of the feature-value registration apparatus which concerns on one Embodiment. It is a flowchart of operation | movement of the feature-value registration apparatus which concerns on one Embodiment. It is a functional block diagram which shows the detail of the pseudo query characteristic production | generation part which concerns on one Embodiment. It is a figure for demonstrating the relationship of the destructor in case there exists the geometric transformation relationship of the same object between images (and the relationship of the same object same location as the contrary).

  FIG. 1 is a functional block diagram of a feature amount registration apparatus according to an embodiment. As shown in the figure, the feature amount registration device 10 includes a pseudo query feature generation unit 1, a registered feature candidate generation unit 2, an iterative processing unit 3, and a feature amount storage unit 4, and the iterative processing unit 3 further includes a feature matching unit 31, A similarity score calculation unit 32 and a registered feature generation unit 33 are provided.

  First, the overall operation of the feature amount registration apparatus 10 is to read an image as an input and output a registered feature set suitable for the image in which a view change is assumed to occur. According to the present invention, the registered feature set is output as a feature set that maximizes the similarity score obtained through feature matching even when a view change is assumed between the query side and the DB side. It is possible.

  For example, if the view change is specifically a change in the position and orientation of the camera at the time of image capture, the registered feature set obtained by the feature amount registration device 10 is the position of the camera at the time of image capture. Even if the posture changes variously, it is optimized as a whole over the fluctuation range, and it is possible to perform robust feature matching regardless of the position and orientation of the image.

  Furthermore, according to an embodiment of the present invention, it is possible to output a registered feature set for the purpose of maximizing the similarity score on the assumption of the presence of a Distractor (distractor, incorrect answer option). It is. Here, in general, in the feature search problem, an incorrect answer option for a query-side feature, that is, a distractor is unavoidable, but in the present invention, a similarity score based on the existence of the distractor is assumed. It is also possible to output a registered feature set that aims to maximize. Details will be described later.

  Note that the registered feature set output by the feature amount registration apparatus 10 means a feature suitable for registration in the DB as a feature set of the image from the above viewpoint, and is not necessarily registered as a use (DB or other It is not required to be recorded permanently or temporarily on some medium.

Further, the framework (main processing flow) of the feature amount registration apparatus 10 is as follows. That is, a set of N DB features (registered features) to be obtained is X = {x ₁ , x ₂ ,…, x _N }, and a set of arbitrary M pseudo-query features is Q = {q ₁ , q ₂ , …, Q _M }, and let D = {d ₁ , d ₂ ,…, d _L } be a set of any L distractors and obtain the maximum similarity score from Q under any constraints It is a framework that generates X that can. (As described above, use of the distractor set D may be omitted depending on the embodiment.)

  The above is the overall outline. Regarding the flow of data exchange between the functional blocks shown in FIG. 1, the outline of the processing contents of each part is as follows.

  The pseudo query feature generation unit 1 reads an image as an input to the feature amount registration apparatus 10, generates a pseudo query feature set Q, and outputs it to the feature matching unit 31. Depending on the embodiment, the pseudo query feature generation unit 1 may further output the generated pseudo query feature set Q to the registered feature candidate generation unit 2 as indicated by a line L21, or as indicated by a line L31. In addition, distractor information (described later) obtained at the time of generation may be output to the iterative processing unit 3.

  The registered feature candidate generation unit 2 generates a registered feature candidate set to be used as a candidate for obtaining a registered feature set as a final output from the feature amount registration apparatus 10 and outputs the registered feature candidate set to the feature matching unit 31. The input for generating the registered feature candidate set can be any one of the embodiments shown by lines L21, L22, and L23, and the pseudo-feature obtained from the pseudo-query feature generating unit 1 as shown by the line L21. A query feature set may be used, a registered feature set obtained from the registered feature generation unit 33 as indicated by a line L22 may be used, or a feature amount storage unit 4 is stored in advance as indicated by a line L23. A feature set may be used.

  The iterative processing unit 3 executes the above-described framework. The pseudo query feature set obtained from the pseudo query feature generation unit 1 and the registration feature candidate set obtained from the registration feature candidate generation unit 2 are input. A predetermined number N of registered feature sets are output as a final result by iterative processing. Details will be described later with reference to FIG. 2. As a first iterative process, each element (qi) of the pseudo query feature set and each element (yj) of the registered feature candidate set are used in the feature matching unit 31 and the similarity score calculation unit 32. ) To obtain a matching result and a similarity score, and the registered feature generation unit 33 performs a second iterative process based on the obtained matching result and the similarity score. A registered feature set is obtained. In each round of the second iterative process, an instruction to update the matching result and similarity score obtained as shown in FIG. 1 is output from the registered feature generation unit 33, and the matching result and similarity score are updated. Is done.

  The units 31, 32, and 33 execute the element processing in each iteration of the iterative processing unit 3, respectively. The feature matching unit 31 performs matching between the pseudo query feature set obtained from the pseudo query feature generation unit 1 and the registered feature candidate set obtained from the registered feature candidate generation unit 2, and the matching result is used to calculate a similarity score. Output to part 32. The similarity score calculation unit 32 calculates a similarity score based on the obtained matching result and outputs the similarity score to the registered feature generation unit 33. Based on the obtained similarity score, the registered feature generation unit 33 determines N elements in a registered feature set as a final output one by one. As described above, the feature matching unit 31 and the similarity score calculation unit 32 perform the first iteration process, and the registered feature generation unit 33 performs the second iteration process.

  In one embodiment, as shown by a line L22, the registered feature set obtained by the iterative processing unit 3 may be further output to the registered feature candidate generating unit 2. In this case, that is, the entire feature quantity registration device 10 performs a predetermined number of repetitions (assuming A times), and registration of the iterative processing unit 3 is performed a-th (a = 1, 2,..., A-1) By using the registered feature set output from the feature generation unit 33 as the input to the next a + 1-th registered feature candidate generation unit 2 (and the feature matching unit 31), finally the registered feature generation at the Ath time The registered feature obtained from the unit 33 is used as the output of the feature registration device 10. Accordingly, in FIG. 1, the line L22 is omitted when the repetition is not performed (when A = 1).

  Depending on the embodiment, the iterative processing unit 3 may also use the distractor information obtained from the pseudo query feature generation unit 1 as indicated by the line L31 or the distracter information obtained from the feature amount storage unit 4 as indicated by the line L32. Using the tractor information, a registered feature set considering the distractor may be generated.

  The feature amount storage unit 4 corresponds to a so-called general DB at the time of image search, and stores the feature amount of each image obtained by photographing each object in advance as a manual operation or the like. In addition, as shown by a line L23, the feature amount is provided to the registered feature candidate generation unit 2 for reference in one embodiment. The feature quantity storage unit 4 may further store the distractor information for each feature quantity to be stored in the same manner as previously obtained by manual work or the like, which is indicated by a line L32 in one embodiment. Thus, the distractor information may be used for reference of the iterative processing unit 3.

  FIG. 2 is a flowchart of the operation of the feature amount registration apparatus 10 according to an embodiment. In the following, details of the processing contents of each unit in FIG. 1 will be described while explaining each step in FIG. 2.

In step S1, a pseudo query feature set Q = {q ₁ , q ₂ ,..., Q _M } composed of M elements is generated from the image obtained as input by the pseudo query feature generation unit 1, and then step S2 Proceed to

  FIG. 3 is a functional block diagram showing details of the pseudo query feature generation unit 1, and the pseudo query feature generation unit 1 includes an image processing unit 11 and a generation unit 12 as shown. The image P0 input to the pseudo query feature generation unit 1 (for the sake of explanation, the image P0) is input to the generation unit 12 and the image processing unit 11. The image processing unit 11 obtains the processed images P1, P2,..., Pb on the assumption that the image P0 has been subjected to the predetermined types b of processings pr1, pr2,. Is output.

  Each of the b pieces of processing in the image processing unit 11 is determined in advance by giving parameters such as the type of processing and the strength of the processing of the type. A predetermined process may be determined in advance as a process that covers view changes assumed when an image search or the like is further performed using the registered feature set output by the feature amount registration apparatus 10.

  For example, the types of processing may include those that perform geometric transformation such as similarity transformation, affine transformation, and projective transformation. In the geometric transformation process, when a 3D model of an object shown in the input image P0 is further included, an image in which a view is changed after applying a texture to the model may be generated. That is, a 3D model of an object including texture information in each element polygon may be given as predetermined data and generated as a CG model that is an existing technology (generated by rendering). Further, when noise, blur, occlusion, or the like is assumed as the view change, for example, arbitrary image processing such as noise addition, smoothing, and masking may be employed as the processing process.

  Further, the parameters of the processing process can be given as a transformation matrix for performing the geometric transformation in the case of geometric transformation, and as the strength of the imparted noise in the case of noise addition or the like. When obtaining one processed image, two or more processing processes may be combined.

The generation unit 12 extracts feature amounts from the input image P0 and the series of processed images P1, P2, ..., Pb obtained from the image processing unit 11, and extracts all the extracted feature amounts. Output as pseudo query feature set Q = {q ₁ , q ₂ ,..., Q _M }. As the feature quantity to be extracted, a predetermined type of local feature quantity (a feature quantity calculated in the vicinity of a feature point (key point)) such as a well-known SIFT feature quantity or SURF feature quantity can be employed.

In addition, the generation unit 12 does not use the input image P0, but extracts the feature amount from only the processed images P1, P2,..., Pb, so that the pseudo query feature set Q = {q ₁ , q ₂ ,. _M } may be output. Similarly, instead of processing by the image processing unit 11 using one input image P0 as an input, a plurality of images obtained by actually photographing the same object under various view conditions are generated by the generation unit 12. May be input.

  In step S1, the pseudo-query feature generation unit 1 further performs processing including geometric transformation in the image processing unit 11, and the distractor information (incorrect information) (or equivalent if there is a geometric transformation relationship) As the information, the same target and same part information) can also be output. In the present invention, a distractor means all the features other than the correct DB feature for one query feature (qi). Specifically, (1) an image that is not related to the query image (input image P0) ( All the features extracted from the image that is taken from a different object than the image P0), and (2) the taken image that is the same subject as the query image (ie, processed images P1, P2,..., Pn) Are defined as a set D (qi) (a set determined for each query feature qi) that combines both of the features that do not satisfy the matching constraint condition. Here, a DB feature that can be said to be “correct” for a certain query feature is a local feature that describes the same part of the same object, but is not exactly the same due to changes in shooting conditions (views). Therefore, a feature that is sufficiently “similar” is regarded as a correct answer, but the “similar” criterion is a matching constraint. A specific example of the matching constraint condition will be described later in the description of the feature matching unit 31.

  With respect to the destructor defined for each query feature qi as both (1) and (2) above, (1) is obvious and with reference to FIG. 4, a schematic example of (2) of the destructors Show. In FIG. 4, n = 100 processed images P1 to P100 are generated and used as DB features (registered feature candidates) as a result of geometric transformation or the like of the image P0 as shown on the right side with respect to the query image P0 shown on the left side. An example is shown. For the feature qi0 extracted from the position p0 of the query image P0 indicated by a circle (the leading edge position of the roof of the “house” as an example), those that satisfy the matching constraint are also marked with a circle for the images P1 to P100. The feature yj1 extracted from the corresponding position p1 in the image P1, the feature yj2 extracted from the corresponding position p2 in the image P2, the feature yj3 extracted from the corresponding position p3 in the image P3, and the corresponding position p100 in the image P100. 100 features yj100 extracted from Therefore, all of the features extracted from the processed images P1 to P10 indicated by Δ in FIG. 4 other than 100 features satisfying these matching constraint conditions (for example, the feature yj1e at the position p1e in the image P1). In FIG. 4, only a part of the destructors are illustrated as Δ marks, and the reference numerals are also given to only a part of them.) Is the destructor of the feature qi0 of the query image P0.

  In the case where the processed image Pk (k = 1, 2,...) Is generated by the geometric transformation Hk for the input image P0 in the image processing unit 11 as in the example of FIG. 4, the position p0 of the image P0. Whether or not the feature qi0 of the image Pk and the feature yk1 of the position pk of the image Pk are the corresponding parts is determined by “|| pk-Hk * p0 || <TH” (where || || is the Euclidean distance When (absolute value), TH is a threshold value), it can be determined that it is a corresponding location (the same target has the same location). That is, when it is determined that the position Hk * p0 obtained by mapping the position p0 of the image P0 by the geometric transformation Hk expressed by a homography matrix or the like is sufficiently close to the position pk, it can be determined as a corresponding part. Conversely, the feature quantity extracted at an arbitrary location other than the sufficiently close position becomes a distractor for the feature qi0 at the position p0 of the image P0. Therefore, when the image processing unit 11 performs conversion including geometric conversion to obtain a processed image, in this way, it is possible to output distractor information by specifying those that do not correspond to the same target and the same location. .

In step S2, the registered feature candidate generation unit 2 generates a registered feature candidate set Y = {y ₁ , y ₂ ,..., Y _K } composed of K elements, and then proceeds to step S3. The registered feature candidate set Y is to obtain N registered feature sets X = {x ₁ , x ₂ ,..., X _N } by selecting N elements from among them. Therefore, K> N is generated for the number of elements K.

  The following embodiments can be used as the method of generating in step S2.

  In one embodiment, as indicated by a line L21 in FIG. 1, the pseudo-query feature set Q generated in step S1 is directly adopted as the registered feature candidate set Y, that is, generated as Y = Q.

  In one embodiment, the registered feature set generated by the registered feature generation unit 33 is adopted as the registered feature candidate set Y as indicated by a line L22 in FIG. The embodiment corresponds to repeating the entire flowchart of FIG. 2 a predetermined number of times and updating the registered feature set obtained in step S10 a predetermined number of times as a registered feature set as a final output.

  In one embodiment, as indicated by a line L23 in FIG. 1, a predetermined number K of feature quantities extracted from the feature quantity storage unit 4 is adopted as the registered feature candidate set Y. The extraction may be performed at random, but it is preferable to perform extraction so that the feature amount is not biased in the feature space.

  Note that since the feature quantity storage unit 4 is a so-called general DB, the destructor information is used as feature quantity without prior knowledge for the feature quantity of an arbitrary image (unknown image) input to the pseudo-query feature generation unit 1. It cannot be defined as a feature quantity in the storage unit 4. As described above with reference to FIG. 4, when there is a geometric transformation relationship, distractor information is defined as having no correspondence between the same target and the same location. However, when there is prior knowledge, distractor information may be defined in at least a part of the feature quantity stored in the feature quantity storage unit 4. The distracter information can be used for reference to the iterative processing unit 3 as indicated by a line L32 in FIG.

  In step S3, the similarity score calculation unit 32 sets zero as the initial value of the similarity score sum score (yj) calculated for each registered feature candidate yj by the following double loop processing S4 to S9, Proceed to S4. (The value of the similarity score sum score (yj) is passed through the double loop process when the value of the corresponding yj is added and updated when the matching determination is made in step S6 in the double loop process. (At the time (when step S10 is reached), the value has been obtained.)

  Step S4 is a dummy step indicating that the following steps S5 to S9 are repeatedly performed for each pseudo query feature qi generated in step S1, and after setting qi to be processed this time, the process proceeds to step S5. Step S5 is a dummy step indicating that the following steps S6 to S8 are repeatedly performed for the registered feature candidate yj generated in step S2, and the process proceeds to step S6 after setting the current processing target yj.

  In step S6, the feature matching unit 31 determines whether or not the pseudo query feature qi set as the processing target matches the registered feature candidate yj, and the process proceeds to step S7. In step S7, when a matching determination is obtained in the latest step S6, the similarity score calculation unit 32 calculates the current similarity score sum score (yi) for the registered feature candidate yi set as the processing target. After adding and updating the value, the process proceeds to step S8. (If the matching determination is not obtained in the latest step S6, the addition update is not performed in step S7, and the process proceeds to step S8 as it is.)

  Note that the details of the processes of steps S6 and S7 in the double loop will be described after the description of the double loop process.

  In step S8, it is determined whether or not the processing in steps S5 to S7 has been completed for all registered feature candidates yj. If completed, the process proceeds to step S9, and if not completed, the process returns to step S5. The unprocessed registered feature candidate yj is set as a processing target and the above is repeated. In step S9, it is determined whether or not the processing of the above steps S4 to S8 has been completed for all the pseudo query features qi. If completed, the process proceeds to step S10, and if not completed, the process returns to step S4. The above processing is repeated with the unprocessed pseudo query feature qi set as the processing target.

  As described above, when the double loop process for each qi and each yj in steps S4 to S9 is completed and the process reaches step S10, a series of matching results by the feature matching unit 32 and each yj by the similarity score calculation unit 32 The final similarity score sum score (yj) is obtained. Hereinafter, details of the matching process and the similarity score calculation will be described in order.

First, with respect to the matching processing in step S6, the feature matching unit 31 sets each element yj of the registered feature candidate set Y for each element qi of the pseudo-query feature set Q based on a previously specified constraint condition (matching determination condition). Find match (qi) from among the matches. That is, if form elements are listed by enumerating them with the symbol “{}”, the following matching results are obtained when the double loop processing is completed. This is called a result list 1.
match (q ₁ ) = {y ₁₁ , y ₁₂ , ...}, match (q ₂ ) = {y ₂₁ , y ₂₂ , ...}, ...
match (q _i ) = {y _i1 , y _i2 , ...}, ...., match (q _N ) = {y _N1 , y _N2 , ...}… (Result list 1)

Various constraints can be used for the matching determination in step S6. Examples include the following. Any or any combination of the methods may be used as the constraint condition.
(Method 1) The query-side feature (qi) and the DB-side feature (yj) are quantized to the same VW.
(Method 2) In the feature vector space where the distractor exists, the feature vector (yj) on the DB side is located in the vicinity of k for the feature vector (qi) on the query side.
(Method 3) The distance between the query-side feature vector (qi) and the DB-side feature vector (yj) is within a threshold.
(Method 4) The difference between the orientation of the query-side feature and the DB-side feature is within the threshold.
(Method 5) The ratio of the scale of the query-side feature to the DB-side feature is within the threshold.

  In the method 2, the k neighborhood means a method of matching qi with yj that is included in the k-th rank having the highest similarity with respect to qi. That is, in Method 2, k pieces having higher similarity to the query-side feature vector (qi) are obtained as matching results match (qi).

  Note that the registered feature candidate set Y may include multiple features y extracted from the same position as the feature (yj) on the DB that is currently focused on. You may exclude it from the ranking of time. Specifically, the following [1] to [3] may be performed.

[1] A list of other ys whose coordinates coincide with yj (those having the same target and the same location) are listed in advance. For example, in the example of FIG. 4, if a pseudo query feature set is extracted from images P1 to P100 (excluding image P0), list (yj1) = {yj2, yj3, ..., yj100} . Similarly, list (yj2) = {yj1, yj3, yj4, ..., yj100}. List (yj3) etc. can be defined in the same way.
[2] Measure the distance between feature vectors of qi and each y, and create a ranking sorted in ascending order. At this time, the element of list (yj) is not included in the ranking. (Yj itself is included.) That is, a series of yεlist (yj) having the same target and the same location as yj currently focused on is excluded from the ranking. (In other words, only yj that is currently focused on, and other y that does not belong to list (yj), are the targets of similarity ranking creation between qi.)
[3] If yj is included in the top k rankings obtained as described above, it is considered that the constraint condition is satisfied, and qi and yj are matched.

  By the above steps [1] to [3], it is possible to reproduce a situation in which only one true correct answer is included in the top k cases. This corresponds to an exceptional process caused by including a large number of images obtained by converting the same image in the subject of the present invention.

  Note that the “feature orientation difference and / or scale ratio” described above and in the methods 4 and 5 mean the difference and ratio regarding the orientation and scale obtained when obtaining a feature vector as an SIFT feature amount or the like.

  In step S7, the similarity score calculation unit 32 adds and updates the similarity score score score (yj) related to the registered feature candidate yj as the contribution of the qi when the matching determination is obtained in the latest step S6. Thus, when the double loop processing is finished, the final similarity score sum score (yj) is obtained.

First, the obtained similarity score sum score (yj) is obtained by using the result list 1 obtained for each pseudo query feature qi described above as a matching result match (yj) viewed from each registered feature candidate yj as follows: After rewriting to the result list 2, the similarity score sum of qi belonging to the result list 2 is obtained. The result list 1 and the result list 2 are equivalent in content.
match (y ₁ ) = {q ₁₁ , q ₁₂ , ...}, match (y ₂ ) = {q ₂₁ , q ₂₂ , ...}, ...
match (y _j ) = {q _j1 , q _j2 , ...}, ...., match (y _K ) = {q _K1 , q _K2 , ...}… (Result list 2)

Specifically, in the process of calculating the similarity score sum in the loop process, the element qi belonging to the result list 2 to be obtained based on the calculation method specified in advance (the element qi determined to be matched in step S6) ), The similarity score sum score (y _j ) of each registered feature candidate y _j (score sum of the similarity between each element q in the list match (y _j ) and the registered feature candidate y _j ) Will be calculated. Various weightings can be used for the addition, and examples include the following.

(Calculation method 1) “1” (fixed value) is used as a weighting value. That is, every time yj and qi are determined to be matched in step S6, “1” is added to the score sum score (yi), thereby completing the following score sum score (yj) Will be required. That is, the similarity score sum score (y _j ) is increased as the number of elements of match (yj) in the result list 2 increases.
score (y _j ) = number of elements belonging to the list match (y _j ).

(Calculation method 2) The existing method tf-idf (Term Frequency-Inverse Document Frequency) is used as the weighting value. That is, if the tf-idf value of the feature vector q belonging to the list match (y _j ) is expressed as tfidf (q), the similarity score sum score (y _j ) Is obtained. The value tf-idf (q) can be obtained according to an existing method after quantizing the feature vector q into VW.

(Calculation method 3) The expression “kthd ² -d ² ” is used as a weighting value. Here, d is the distance between the feature vectors on the query side and the reference side, and kthd is the distance between the feature vectors with the k-th feature closest to the feature on the query side. That is, if the value of `` kthd ² -d ² '' obtained for the feature vector q belonging to the list match (y _j ) is written as f (q), as in the case of tf-idf, It will be sought.

Here, “kthd ² -d ² ” is a score function when an image classification method called Local Naive-Bayes Nearest-Neighbor disclosed in Non-Patent Document 3 described below is applied to image retrieval. kthd ² -d ² may be calculated using feature vectors even after being quantized to VW, and as in Non-Patent Document 1, the distance between vectors called Hamming embedding, which is a compact transformation of feature vectors, is used. May be.
[Non-Patent Document 3] McCann and DG Lowe, "Local naive bayes nearest neighbor for image classification," in Proc. Of CVPR, 2012.

  In calculation method 3, as described in (Method 2) of feature matching unit 31, a ranking is created by procedures [1] to [3], and the k-th feature vector in the ranking is calculated. You may make it ask.

  The similarity score sum score (yj) can be calculated using not only the calculation methods 1 to 3 but also any other existing score function (weighting function) for image search.

  In step S10, the registered feature generation unit 33 uses an empty set (as an initial value of a registered feature set for obtaining a final result (the final result is output in step S15) in the following N repetition steps S11 to S14). After setting φ), the process proceeds to step S11. Step S11 is a dummy step indicating that the following steps S12 to S14 are repeated N times, and the process directly proceeds to step S12.

In step S12, the registered feature generation unit 33 adds new elements x _n (n = 1, 2,,,, N to the registered feature list by the following (procedure 1) and (procedure 2), and n is the repetition Step S12 to S14 represent the number of repetitions at the present time), and the process proceeds to Step S13.

(Procedure 1) The registered feature candidate y _max having the maximum similarity score score (y _j ) is selected from each registered feature candidate y _j . That is, when expressed in an expression, the index max is determined as follows.
max = argmax (score (y _j ))
(Procedure 2) A feature vector x _{n to} which an average of the feature vectors of the pseudo-query feature set match (y _max ) associated with this registered feature candidate y _max is added.

In the procedure 2 described above, similarly, not only the feature vector x _n but also a feature obtained by averaging at least one of the orientation o, the scale s, and the coordinate u may be used as the registered feature f _n . For example, when all of these and registration features registered feature is obtained as those listed _{f n = (x n, o} n, s n, u n) the respective amounts as.

  In the above procedure 2, the following additional embodiment is also possible. In other words, the feature averaging in the procedure 2 may result in a smaller sum of similarity scores obtained from the pseudo-query feature set. Therefore, the feature matching unit 31 and the similarity score calculation unit 32 are processed again using the features averaged in step 2, and as a result, if the sum of the similarity scores is small, no averaging is performed. You may make a judgment.

That is, the score (y _max) selected in the initial (Step 1), and score (x _n) by x _n obtained by averaging in (Step 2), by comparing the "score (y _max) If “> score (x _n )”, the averaged x _n may not be adopted as a registration target. On the other hand, when the similarity score sum increases, _xn averaged as described above may be adopted as an element of the registered feature set. When calculating the score (x _n ) using the averaged feature, the provisional feature is obtained by replacing a plurality of features to be averaged with one averaged feature in the registered feature candidate set Y. The processing of the feature matching unit 31 and the similarity score calculation unit 32 may be performed on the set Y ′ (that is, using the temporary set Y ′ instead of the set Y as an input). For example, when a registered feature is obtained by averaging x _n = (y1 + y2 + y3) / 3, three y1, y2, and y3 may be replaced with one _xn . If it is determined that averaging is not performed, _ymax may be registered as _xn as it is without averaging.

In step S13, the registration feature generation unit 33 of the immediately preceding step S12 in brute cord to y _max determined in (Step 1) Pseudo query feature set match (y _max) (matching result list pseudo query feature for y _max) Updates that exclude each element from the pseudo query feature set match (y _j ) = {q _j1 , q _j2 , ...} associated with all other yj (including y _max itself) and match ( After updating to subtract the contribution to the score sum given from each element of y _max ) from the other similarity score sum score (y _j ), the process proceeds to step S14. The update to be subtracted is, in other words, using “match (yi) \ match (y _max )” that excludes elements belonging to match (y _max ) from match (yi) for calculating the score sum score (yi) This is equivalent to recalculating the score sum score (yi). “\” Is a difference set operation, and the same applies to the following.

As a result of the update in step S13, the registered feature candidate set Y is updated as follows, and the matching result and the similarity score sum obtained when the double loop processing of steps S4 to S9 is performed again are substantially the same. The updated matching result and similarity score sum can be obtained at high speed without performing the double loop process again.
Y = Y \ match (y _max )

Also, assuming that the number of iterations of steps S11 to S14 is n, the pseudo-query feature set match (y _j ) = {q _j1 , q _{j2 ,.} .} (That is, the updated result list 2) and the updated similarity score sum score (y _j ) are used in the next n + 1 determination in step S12. (Note that if y _max is selected even once (assuming y _[selected] ), the score value score (y _[selected] ) will be updated to zero in subsequent iterations. In addition, the list match (y _[selected] ) as the matching result is also an empty set, and is therefore excluded from the selection in (procedure 1).)

  In step S14, the registered feature generation unit 33 determines whether or not a registered feature set including N elements has been obtained at that time. If not, the process returns to step S11 to add a new element. The above is repeated, and if it is obtained, the process proceeds to step S15. In step S15, the registered feature generation unit 33 outputs a registered feature set composed of N elements obtained in the above repeated steps S11 to S14 as a final result, and the flow of FIG. 2 ends.

  As described above, according to the present invention, it is possible to obtain a robust feature amount even when there is a view change. In addition, it is possible to generate a registered feature set that is highly likely to match query features in various views in a feature vector space in which a distractor exists. Furthermore, by preferentially selecting a registered feature that gives a higher similarity score, it is possible to improve the accuracy of image search for a large-scale database. The supplementary items are described below.

  (1) An embodiment in which the entire flow of FIG. 2 is repeated (an embodiment in which the input of the line L22 in FIG. 1 is performed after the second iteration), that is, a registered feature set obtained by a single process is newly registered. An embodiment in which processing is repeated until it is regarded as a feature candidate and the end condition is satisfied can be performed as follows, for example.

  For example, the first registered feature candidate is set to a pseudo query feature set (M> N), and M ′ (M> M ′> N) registered feature sets are obtained. The process of obtaining M ″ (M ′> M ″> N) registered feature sets by regarding it as a new registered feature candidate may be repeated until N registered features are obtained. Alternatively, the N registered features obtained in the first process may be regarded as new registered feature candidates, the process may be performed, and the process may be repeated until there is no change in the resulting similarity score sum.

  (2) The matching constraint, the similarity score calculation method, the pseudo query feature set, and the destructor according to the present invention are expected to be more effective as the search process at the time of execution is followed. That is, the matching constraints and the similarity score calculation method are content-matching with the matching method and similarity score calculation method used at the time of execution, and the pseudo-query feature set is similar to that obtained from the image actually captured by the user However, it is desirable that the contents of the element set to which the distractor relationship is given and the number of elements are similar to those in the actual search database.

  (3) The present invention can also be provided as a program that causes a computer to function as the feature amount registration apparatus 10. The computer can employ a known hardware configuration such as a CPU (Central Processing Unit), memory, and various I / Fs, and the CPU executes instructions corresponding to the functions of each part of the feature value registration device 10. Will be.

  DESCRIPTION OF SYMBOLS 10 ... Feature quantity registration apparatus, 1 ... Pseudo query feature generation part, 2 ... Registered feature candidate generation part, 31 ... Feature matching part, 32 ... Similarity score calculation part, 33 ... Registration feature generation part, 4 ... Feature quantity storage part

Claims (14)

A feature amount registration device for obtaining a registered feature set as a registration target from an input image,
A pseudo query feature generation unit that generates a pseudo query feature set from the input image and / or one or more processed images obtained by processing the input image;
A registered feature candidate generator for generating a registered feature candidate set;
A feature matching unit that performs matching between each element of the pseudo-query feature set and each element of the registered feature candidate set;
Based on the matching result by the feature matching unit, for each element of the registered feature candidate set, a similarity score calculation unit that calculates a similarity score sum between the elements of the pseudo query feature set;
A registered feature generation unit for obtaining the registered feature set by determining an element of the registered feature set using an element of the registered feature candidate set selected based on the obtained similarity score sum A feature amount registration apparatus characterized by that. Between at least some elements of the registered feature candidate set, the same target same part information that prescribes that the features are extracted from the same part of the same target is given in advance.
The feature matching unit and / or the similarity score calculation unit respectively perform the matching process and / or the process of obtaining the similarity score sum using the same target same part information. The feature amount registration device according to 1.   In the feature matching unit, for each first element of the pseudo-query feature set, when determining whether to match with each second element of the registered feature candidate set, The feature amount registration according to claim 2, wherein the other second element of the registered feature candidate set having the relationship of the same target and the same location information is excluded from the material for determining whether or not to match. apparatus.   In the feature matching unit, for each first element of the pseudo-query feature set, a determination as to whether or not to match with each second element of the registered feature candidate set is based on the similarity to the first element. 4. The feature amount registration apparatus according to claim 3, wherein when the rank is within a predetermined upper rank, it is determined that the matching is performed.   5. The feature amount registration apparatus according to claim 2, wherein the similarity score calculation unit obtains the similarity score sum by a predetermined type of weighted sum.   6. The feature amount registration apparatus according to claim 1, wherein the registered feature generation unit selects the element of the registered feature candidate set that maximizes the calculated similarity score sum.   The registered feature generation unit determines an element of the registered feature set based on each element matched by the matching unit with respect to an element of the selected registered feature candidate set. Item 7. The feature registration device according to any one of Items 1 to 6.   The registered feature generation unit determines the elements of the registered feature set by averaging the elements matched by the matching unit with respect to the elements of the selected registered feature candidate set. The feature amount registration apparatus according to any one of claims 1 to 7.   In the registered feature generation unit, an element obtained by averaging the elements matched by the matching unit is obtained for the elements of the selected registered feature candidate set, and the registered feature candidate set is provisionally determined based on the averaged element. The similarity score sum calculated for the averaged element by applying the processing by the feature matching unit and the similarity score calculation unit for the updated provisional update set is an element of the selected registered feature candidate set. If it is smaller than the similarity score sum in, the element of the selected registered feature candidate set is directly adopted as the element of the registered feature set, and if not smaller, the averaged element is used as the element of the registered feature set 8. The feature amount registration apparatus according to claim 1, wherein the feature amount registration device is employed.   In the registered feature generation unit, each element of the registered feature set is determined one by one, and each time the determination is made, the contribution by the element of the pseudo query feature set corresponding to the determined element is determined from the similarity score sum. The similarity score sum is updated by subtraction, and the elements of the pseudo-query feature set corresponding to the determined element based on the updated similarity score sum are subsequently used as the registered feature set. 10. The method according to any one of claims 1 to 9, wherein the determination of the next one of the elements of the registered feature set is repeated a predetermined number of times by excluding them from the reference object for determining the elements of The feature quantity registration device described.   The registration feature candidate generation unit employs the pseudo query feature set generated by the pseudo query feature generation unit or the registration feature already generated by the registration feature generation unit as the registration feature candidate set. The feature amount registration apparatus according to claim 1.   Each of the processed images is an image obtained as a result of geometric transformation of the input image and / or as a result of performing a Lindering based on predetermined 3D data including a texture of an object shown in the input image. The feature amount registration device according to claim 1, wherein the feature amount registration device is a feature amount registration device. A feature amount registration method for obtaining a registered feature set as a registration target from an input image,
A pseudo query feature generation step of generating a pseudo query feature set from the input image and / or one or more processed images obtained by processing the input image;
A registered feature candidate generation stage for generating a registered feature candidate set;
A feature matching stage for matching between each element of the pseudo-query feature set and each element of the registered feature candidate set;
A similarity score calculation step for obtaining a similarity score sum between elements of the pseudo-query feature set for each element of the registered feature candidate set based on a matching result obtained by the feature matching step;
A registered feature generation step of determining the registered feature set by determining an element of the registered feature set using an element of the registered feature candidate set selected based on the calculated similarity score sum. A feature amount registration method characterized by that.   A program for causing a computer to function as the feature amount registration apparatus according to any one of claims 1 to 12.