JP6541226B2 - Information terminal device and program - Google Patents

Description

  The present invention relates to an information terminal apparatus, program and data structure for recognizing an object imaged by an imaging unit, and more particularly to an information terminal apparatus, program and data structure for recognizing similar objects with high accuracy by selecting feature information. .

  The technology for recognizing an object from an image can convert analog information described in a medium that can be easily distributed or presented into digital information, and can improve user convenience.

  In Non-Patent Document 1, as an example of the technique, a feature point is detected from an image, a local image feature amount is calculated from the periphery of the feature point, and then matched with a local image feature amount accumulated in advance. Recognize the type of object.

  Also, as a technique for improving the accuracy of recognition of similar objects, the followings have been disclosed.

  Patent Document 1 discloses a method of identifying similar images by selecting image features to be used for matching in accordance with position information and orientation information of an object. Patent Document 2 discloses a method of discriminating similar images by bidirectional template matching of partial images even when the position or the luminance is shifted in template matching. In bi-directional template matching, first, a partial image of the reference image that matches the partial image of the query image is detected, and then a partial image of the query image that matches the detected result is detected. The accuracy is improved by determining In Non-Patent Document 2, first, candidates are narrowed down using the method of Non-Patent Document 1, and secondly, a target with a small difference is determined by template matching of a difference area.

JP 2014-134858 A Patent 5375823

D. G. Lowe, '' Object recognition from local scale-invariant Features, '' Proc. Of IEEE International Conference on Computer Vision (ICCV), pp. 1150-1157, 1999. Mase, Iwamoto, '' Article inspection system using two-step analysis with local feature matching and image subtraction, '' Information Science and Technology Forum Proceedings 13 (3), 103-104, 2014.

  However, in the prior art as described above, when there are multiple similar objects, there is a problem that the similar objects can not be distinguished and recognized, or some difficulty is caused even if they can be recognized. Here, the similar target is, for example, the cover G1 of the upper volume and the cover G2 of the lower volume in the book published separately in the upper volume and the lower volume, as schematically illustrated in FIG. , G2 differ only in the letter part P1 of the "upper volume" and the letter part P2 of the "lower volume" which are only a very small part of them, and the remaining most of the title "Novel Kai" (a fictional example for illustration The patterns of the cover binding and the like are common.

  That is, in Non-Patent Document 1, when there are a plurality of similar objects as exemplified in FIG. 1, the local image feature quantities extracted from each of the similar objects become almost the same among the similar objects. Therefore, there is a problem that similar objects can not be distinguished from each other, or even if they can be distinguished from each other, the accuracy is poor.

  In Patent Document 2, it is claimed that similar images such as twins can be distinguished by bidirectional template matching of partial images including acquired facial moles, stains, freckles and the like. However, because normalization (position adjustment, rotation, scaling, etc.) required as preprocessing for template matching is based on global positional relationships such as eyes, nose, ears, etc., processing targets are limited to the same shape Have the problem of being

  In Patent Document 1, it is possible to identify a plurality of landmarks as an example that can be a plurality of similar images by using position information and orientation information in combination, but can not be applied to objects that do not depend on position information or orientation information. There is a problem of

  In Non-Patent Document 2, since the template matching of the difference area is used, it is possible to partially solve the above problem (in particular, the problem in Non-Patent Document 1). However, since it is necessary to designate the difference area in advance, there is a problem that it takes time and effort to substantially deal with the case where there are many recognition targets.

  Furthermore, in both Patent Document 2 and Non-Patent Document 2, the value of template matching changes significantly due to a slight positional deviation, so there is a problem that the recognition accuracy is not sufficient. FIGS. 2 and 3 each show an example of a similar object in which a slight positional deviation occurs which greatly changes the value of template matching.

  FIG. 2 is a diagram showing soft packages G3 and G4 in which the same kind of liquid detergents common to manufacturers and brands are enclosed as an example of similar objects, and detergent names such as "detergent α" Most of them are common to each other, for example, package patterns and the like, but they are different from each other only in the description portion P3 of the description portion of the fragrance and the description portion P4 of the "no fragrance combination".

  The soft packages G3 and G4, which are similar objects to each other, are formed of a plastic film or the like which has flexibility and is easily deformed. The enclosed detergent flows according to the direction of (or the like) so as to reduce the potential energy and is easily deformed according to the flow. In addition, even if the direction of placement is the same, for example, even if the product shelf is placed vertically, since the surface has a cylindrical shape by enclosing the detergent, it is always possible to image the product shelf from the front Even if you use it, the way you look easily changes depending on the installation direction. For example, the appearance changes depending on whether the product shelf is installed to the right with respect to the front of the product shelf or to the left with respect to the front of the product shelf. Since the positional deviation on the captured image is easily generated due to the easy deformation and / or the easy change of appearance, the value of the template matching also largely fluctuates.

  FIG. 3 shows, as an example of similar objects, packages G5 and G6 of the same game package provided by the same game vendor and provided for different game machines (hardware devices for executing the game). Although the packages G5 and G6 are entirely similar by being similar to each other, there are slight differences as follows. That is, the description portion P51 indicating that the target game device is "game device X" and the description portion P61 indicating that "game device Y" are different from each other. In addition, a predetermined standard logo indicating, for example, that it is "adult" for the age to be played in the game is provided by a logo P52 of a predetermined shape provided by the vendor of "game machine X" and a vendor of "game machine Y". They differ from one another in that they are logos P62 of another predetermined shape.

  Furthermore, parts P53 and P63 of the game title "adventure game" and package illustrations (which are fictitious examples for illustration) are identical to each other as "pictures" but only their sizes (and / or aspect ratios) Are slightly different in that they are slightly different. (In addition, in order to promote understanding, in Fig. 3, differences such as size etc. are drawn emphatically with respect to parts P53 and P63.)

  In this way, with respect to the portions P53 and P63 whose sizes (and / or aspect ratios) are different even if they are the same as a picture, this is an example in which slight positional deviation occurs, and it is well Even if you try to match the other using a template that matches, it will hardly match.

  An object of the present invention is to provide an information terminal device, a program, and a data structure capable of recognizing with high accuracy even if there is a similar target as a recognition target, in view of the problems of the prior art described above. .

  In order to achieve the above object, the present invention is characterized by the following (1) to (12).

  (1) An imaging unit that captures an imaging target and obtains a captured image, a calculation unit that calculates a feature point and a local image feature amount near the feature point from the captured image, and outputs it as feature information; For each recognition target feature information obtained by accessing a database storing the same type of feature information as the calculation unit calculates from the image, and feature information calculated from the captured image The first matching unit obtains candidate information by evaluating similarity and listing recognition targets determined to be similar, and feature information between the recognition targets listed in the candidate information Match is determined, and the feature information for which the match is not determined is determined as partial feature information for each recognition target listed in the candidate information, and the obtained partial feature information for each recognition target and the captured image A second matching unit that evaluates the similarity with the calculated feature information and determines a recognition target corresponding to partial feature information determined to be similar as one corresponding to an imaging target in the captured image; An information terminal apparatus comprising:

  (2) When the second collating unit determines that the feature information matches each other among the recognition targets listed in the candidate information, each of the feature information includes a predetermined number or more of other recognition targets. When it is determined that the feature information is similar to the feature information, a match is determined for the feature information.

  (3) When the second matching unit evaluates the similarity between the partial feature information of each of the recognition objects obtained and the feature information calculated from the captured image, a distance between the partial feature information and the feature information And / or evaluation is performed based on the coincidence between the partial region near the feature point of the partial feature information and the partial region near the feature point of the feature information.

  (4) The second collation unit evaluates the similarity of the feature information of each recognition object in the candidate information with respect to the recognition objects listed in the candidate information for which the partial feature information is to be obtained. When there is a recognition target determined to be dissimilar to another recognition target similar to each other, the recognition target determined to be dissimilar is excluded from the targets for which the partial feature information is sought. It is characterized by

  (5) The second collating unit normalizes the resolution of the feature information of each recognition object in advance when determining the coincidence of the feature information among the recognition objects listed in the candidate information. And determining the match.

  (6) When normalizing the resolution, the second matching unit determines a relationship for converting the coordinates of the feature point in the feature information of each recognition target into the coordinates of the feature point in the feature information of the captured image, It is characterized in that normalization is performed based on the relationship to be converted.

  (7) In the database, for each of a plurality of recognition objects, similar object enumeration information in which other recognition objects similar to itself are listed in addition to the same kind of feature information calculated by the calculation unit from the image. Are stored, and the first matching unit evaluates the similarity and obtains candidate information as a list of recognition targets determined to be similar, and is further listed in the candidate information. For each recognition target, the similar target enumeration information is referred to, and when there is a deviation recognition target as a recognition target not included in the candidate information, the deviation recognition target is compared with the candidate information. It is characterized by adding.

  (8) In the database, similar target enumeration information in which other recognition targets similar to itself are listed in addition to the same kind of feature information calculated by the calculation unit from the image for each of a plurality of recognition targets. Is stored, and the first matching unit evaluates the similarity, and when obtaining candidate information as a list of recognition objects determined to be similar, the evaluation target of the similarity, It is determined that the similarity is made with respect to at least one of the recognition objects of the part after limiting to only a part of a series of recognition objects which are made similar to each other by the similar object enumeration information in the database In the case where the candidate information is obtained, the candidate information is obtained as a list of a series of recognition objects that are considered to be similar to each other by the similar object enumeration information. And butterflies.

  (9) In the database, in addition to the same kind of feature information calculated by the calculation unit from the image, text information on the recognition object is stored for each of the plurality of recognition objects, The collation unit evaluates the similarity, obtains candidate information as a list of recognition targets determined to be similar, and further, obtains text information about each recognition target listed in the candidate information. By searching the database using the above, the deviation recognition target as a recognition target not included in the candidate information and found to be similar to the text information is found from the database In this case, the deviation recognition target is added to the candidate information.

  (10) The calculation unit calculates the feature information using only a specific color channel designated in advance, and the first matching unit evaluates the similarity using only the specific color channel, The second matching unit is characterized in that the match is determined using only the specific color channel, and the similarity is evaluated.

  (11) A program which causes a computer to function as the information terminal device.

  (12) A data structure in which the characteristic information calculated from the image and the similar target enumeration information in which other recognition targets similar to itself are listed for each of a plurality of recognition targets, The data structure for each of a plurality of recognition targets, an imaging unit for capturing and acquiring a captured image, a calculation unit for calculating a feature point and a local image feature quantity near the feature point from the captured image and outputting as feature information Similarity is evaluated between the feature information of each recognition target obtained by accessing the database storing the matched data and the feature information calculated from the captured image, and the recognition determined to be similar The first matching unit for obtaining candidate information as a list of objects, and the recognition objects listed in the candidate information, determine whether the feature information matches, and the determination of the match is not made. Characteristic information is determined as partial feature information in each recognition target listed in the candidate information, and the similarity between the determined partial feature information of each recognition target and the feature information calculated from the captured image is evaluated, and similarity is obtained. And a second collating unit that determines a recognition target corresponding to partial feature information determined to be correct as a target to be imaged in the captured image. The matching unit evaluates the similarity, obtains candidate information as a list of recognition targets determined to be similar, and further, for each recognition target listed in the candidate information, the similarity When there is a deviation recognition target as a recognition target not included in the candidate information with reference to the target enumeration information, the deviation recognition target is added to the candidate information, or The first matching unit evaluates the similarity, and when obtaining candidate information as a list of recognition targets determined to be similar, the target for evaluating the similarity is listed in the database as the similar target. If it is determined that the similarity is at least one of the recognition targets of the part after limiting to only a part of the series of recognition targets that are similar to each other according to the information The candidate information is obtained as a list of a series of recognition objects which are considered to be similar to each other by similar object enumeration information.

  According to the feature (1) or (11), even if there are similar objects similar to each other among the recognition objects, the second matching is performed after the similar matching is listed as candidate information in the first matching unit. Since information on a difference between similar objects is automatically extracted as partial feature information in the unit and recognition of the imaging target in the captured image is performed based on the partial feature information, the imaging target is recognized with high accuracy. be able to.

  According to the feature of the above (2), the feature information which is similar to other feature information incidentally by making the match determination by obtaining the similarity determination of the predetermined number or more is the match determination Can be used to sort out appropriate partial feature information.

  According to the feature (3), the imaging target is determined based on the distance between the feature information and / or the coincidence between the partial areas in the vicinity of the feature point as the imaging subject and the feature on the image of the recognition target. It is possible to evaluate the similarity between the feature information of and the partial feature information of the recognition target.

  According to the feature (4), when there is a recognition target that is different from other recognition targets accidentally in the candidate information, the different recognition targets are excluded from the candidate information. As a result, it becomes possible to sort out appropriate partial feature information by candidate information composed only of similar objects.

  According to the feature (5), partial feature information can be appropriately sorted by unification by prior resolution normalization. According to the feature (6), appropriate normalization can be performed by implementing the normalization as one that applies a conversion relationship between coordinates.

  According to the feature (7) (or (12)), by using similar target enumeration information for each recognition target provided in advance in the database (or data structure), other similar targets can be captured images Since it is possible to automatically add deviation recognition targets that are accidentally judged to be dissimilar to the captured image despite being judged to be similar to the candidate information, a plurality of similar targets Even in some cases, high-accuracy recognition is possible.

  According to the feature (8) (or (12)), by using similar target enumeration information for each recognition target provided in advance in the database (or data structure), a large number of similar targets as recognition targets Even if it exists, high-speed and highly accurate recognition is possible. In addition, it is also possible to suppress the amount of data communication between the database and the first collation unit.

  According to the feature (9), as in the feature (7), even if the similar target enumeration information is not provided in advance to the database, it corresponds to the similar target enumeration information dynamically based on the similarity of the text information By creating the same, it is possible to do the same as in the case of the feature (7). That is, it is possible to automatically add, to the candidate information, a deviation recognition target which is accidentally determined to be dissimilar to the captured image although the other similar objects are determined to be similar to the captured image. Since this becomes possible, highly accurate recognition is possible even when there are multiple similar objects.

  According to the feature (10), when there is a similar target such that only the color is different in the recognition target, it is possible to distinguish and recognize.

It is a figure which shows the cover of the book which is mutually similar as a typical example of similar object. FIG. 6 shows detergent packages similar to one another as a schematic example of similar objects. It is a figure which shows the game package which is mutually similar as a typical example of similar object. It is a functional block diagram of the information terminal device concerning one embodiment. It is a figure which shows a typical series of examples which show the processing content of each part of an information terminal device. It is a figure which shows a typical series of examples which show the processing content of each part of an information terminal device.

  FIG. 4 is a functional block diagram of the information terminal device according to an embodiment. The information terminal device 1 includes an imaging unit 2, a calculation unit 3, a storage unit 4, a first comparison unit 5, and a second comparison unit 6.

  The hardware configuration of the information terminal device 1 for realizing the respective units 2 to 6 may be the configuration of a desktop computer, laptop computer, or other common computer, as well as a portable terminal such as a smartphone or a tablet terminal. it can. That is, data communication between the CPU (central processing unit), a temporary storage device for providing a work area to the CPU, a secondary storage device for storing data such as programs, and various input / output devices And a general computer hardware configuration that includes a carrying bus. Each unit 2 to 6 is realized by the CPU reading and executing a program stored in the secondary storage device. As various input / output devices, a camera for acquiring an image, a display for displaying, a touch panel or keyboard for receiving user input, a microphone / speaker for inputting / outputting voice, a communication interface for communicating with the outside in a wired or wireless manner From among such things, you can use the one according to the required function.

  In addition, although the information terminal device 1 includes all of the units 2 to 6 in FIG. 4, an arbitrary part of this may be an external device (one or more external devices such as a server separate from the information terminal device 1). ) May be realized. For example, the storage unit 4 may be realized by a server as an external device. In this case, transmission and reception of necessary information may be performed via a network or the like between the information terminal device 1 and the external device (including two or more external devices, including between external devices). . The hardware configuration of the external device may adopt a general computer configuration as well as the information terminal device 1.

  Hereinafter, the processing content of each part 2 to 6 in FIG. 4 will be described. 5 and 6 are diagrams showing a series of typical examples showing the processing contents of the respective parts 2 to 6, and will be referred to as appropriate in the following description.

  The imaging unit 2 captures an imaging target and outputs the captured image to the calculation unit 3. Here, under the operation of the user or the like, the imaging unit 2 performs an operation such that the captured image includes a known imaging target (target matching the recognition target stored in the storage unit 4 described later) in advance. It is assumed that imaging is performed. The imaging target may be, for example, a marker, a printed matter, a three-dimensional object, or the like having a pattern with known features and the like. As a hardware configuration responsible for the imaging process of the imaging unit 2, for example, a digital camera that is standardly installed on a general portable terminal of recent years can be used.

  The calculation unit 3 first detects a feature point of an imaging target from a captured image captured by the imaging unit 2. As the feature point to be detected, a feature point such as a corner in the recognition target can be used. As a detection method, an existing method for detecting characteristic points such as Scale-Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF) can be used.

  Next, the calculation unit 3 calculates a local image feature amount from the captured image captured by the imaging unit 2 with the detected feature point coordinates as a center. The plurality of feature points and local image feature amounts calculated by the calculation unit 3 are output to the first matching unit 5 as feature information. As a method of calculating the local image feature amount, an existing method of calculating a characteristic amount such as Scale-Invariant Feature Transform (SIFT) or Speeded Up Robust Features (SURF) can be used.

  In the example of FIG. 5, the captured image P1 obtained by the imaging unit 2 is shown in [1], and the feature information F1 calculated in the calculation unit 3 from the captured image P1 is shown in [2]. The feature information F1 is configured as a plurality of pieces of feature information f1 to f9 obtained by calculating local image feature amounts at a plurality of (nine in the example) feature points.

  The storage unit 4 stores, for each of a plurality of predetermined recognition targets, feature information calculated in the same processing as the calculation unit 3 from the image of the recognition target. The predetermined plurality of recognition targets can include two or more similar targets similar to each other as exemplified in FIGS. 1 to 3. The process of storing the feature information for each recognition target may be performed in advance by a manager or the like using a manual or the like. Here, in order to speed up the processing in the first collating unit 5 and the second collating unit 6 to be described next, the feature information may be summarized by vector quantization, a hash function or the like and then stored.

  The first comparison unit 5 evaluates the similarity between the feature information of the imaging target input from the calculation unit 3 and the feature information of each recognition target stored in the storage unit 4 and determines similarity based on a preset threshold value. If there is a high recognition target, the feature information is linked for each recognition target, and candidate information as information in which the recognition target exceeding the threshold is listed is output to the second collating unit 6.

  Here, for the evaluation of the similarity, an existing method using Hamming distance, Euclidean distance, Mahalanobis distance or the like between the pieces of feature information can be used. Further, the correspondence between the most similar feature information is determined based on the distance and the like between individual feature information between the recognition target and the captured image, and then the sum total of the similarities in the decided correspondence is determined. Alternatively, the general correspondence relationship and similarity between feature information between a recognition target and a captured image may be determined by using a well-known RANSAC (Random Sample Consensus).

  In the example of FIG. 5, the feature information determined to be similar to the feature information F1 of the captured image P1 of [2] in the threshold determination is [3-1], [3-2], and [3-3]. , F32 and F33. Here, assuming that the feature information F31, F32, and F33 are the feature information of the recognition objects R1, R2, and R3 (not shown) stored in the storage unit 4, respectively, they are similar to the imaging objects of the captured image P1. {R1, R2, R3} listing the three recognition targets determined to be the candidate information. Here, the candidate information is configured after the feature information F31, F32, and F33 are associated with the respective recognition targets R1, R2, and R3 that configure the candidate information.

  As shown in the figure, the feature information F31 of the recognition target R1 is composed of a plurality of (9 in total) pieces of feature information f11 to f19 calculated from each feature point of the image (a schematic example is not shown; the same applies hereinafter). The feature information F32 of the recognition target R2 is composed of a plurality of (nine in total) feature information f21 to f29 calculated from each feature point of the image, and the feature information F33 of the recognition target R3 is from each feature point of the image A plurality of (total 7 pieces) of feature information f30, f32, f33, f35, f36, f38 and f39 are calculated.

  The second verification unit 6 receives candidate information from the first verification unit 5 and selects one of the recognition targets stored in the storage unit 4 as the imaging target in the captured image as the recognition target selected from the candidate information. Output as collation information representing the result of collation. Here, when only one recognition target is included in the candidate information output from the first collation unit 5, the second collation unit 6 does not need to perform the sorting, and the one recognition target is collated as it is. Output as information.

  On the other hand, when a plurality of (two or more) recognition targets are included in the candidate information, the second collation unit 6 outputs collation information by the following first process and second process.

  As a first process, among the feature information in each recognition target listed in the candidate information, a collation for specifying which one is determined to match between the things belonging to different recognition targets is used. It carries out, and the characteristic information judged not to be matched by the collation concerned is sorted out as partial characteristic information about each recognition object.

  As the second process, the similarity between partial feature information for each recognition target and feature information for imaging target is evaluated, and if there is partial feature information having a similarity higher than a preset threshold value, the partial feature information The recognition target corresponding to is output as collation information.

  Here, for both the coincidence determination in the first process and the evaluation of the similarity in the second process, as in the case of the first matching unit 5, the existing Hamming distance, Euclidean distance, Mahalanobis distance, etc. between feature information You can use the method. In addition, after the correspondence between the most similar pieces of feature information is determined based on the distance or the like among the individual pieces of feature information, the sum of the similarities in the determined correspondence may be determined, or may be known. The overall correspondence relationship and similarity between feature information may be determined by RANSAC. Also in the first process, whether or not the pieces of feature information match each other may be determined by threshold determination of the degree of similarity between the pieces of feature information.

  In the example of FIG. 5, [4-1] to [4-3] of FIG. 5 show an example of matching of feature information in the first process, and [3-1] to [3-3] Among the pieces of feature information F31 to F33 of the respective recognition objects R1 to R3 constituting the same candidate information as shown in, a line is drawn between the pieces of feature information determined to be coincident by matching. ing.

  That is, it is determined that the feature information f11 and f21 match each other between the recognition objects R1 and R2 as indicated by the line L1, and the feature information f12 and f2 between the recognition objects R1, R2 and R3 as indicated by the lines L21 and L22. f22 and f32 are determined to match each other, and as shown by lines L31 and L32, feature information f13, f23 and f33 are determined to match each other between recognition objects R1, R2 and R3, as shown by line L4. It is determined that the feature information f14 and f24 match each other between the recognition targets R1 and R2, and the feature information f15, f25 and f35 match between the recognition targets R1, R2 and R3 as indicated by lines L51 and L52. It is determined that the feature information f16, f26 and f36 match each other between the recognition objects R1, R2 and R3 as shown by lines L61 and L62, and between the recognition objects R1 and R2 as shown by line L7. The feature information f17 and f27 are determined to match each other.

  Next, in the example of FIG. 6, it is determined that the information items [5-1] to [5-3] match depending on the comparison of the pieces of feature information of [4-1] to [4-3] of FIG. Partial feature information PF31 to PF33 in each recognition object R1 to R3 obtained in the first process as the missing feature information is shown.

  That is, as shown in [5-1], the partial feature information PF31 of the recognition target R1 is composed of the feature information f18 and f19, and as shown in [5-2], the partial feature information PF32 of the recognition target R2 is a feature As shown in [5-3], the partial feature information PF33 of the recognition target R3 is composed of feature information f30, f38 and f39. The partial feature information PF31 to PF33 did not match the feature information of other recognition targets in the matching of the feature information shown by the lines between [4-1] to [4-3] in FIG. 5. Because of that, it is selected as partial feature information.

  Furthermore, in the example of FIG. 6, examples of outputting the verification information by the second process are shown in [6-1] to [6-3] and [7]. That is, in [6-1] to [6-3], partial feature information PF31 to PF33 in each recognition target R1 to R3 similar to those shown in [5-1] to [5-3] [7] is a target of similarity evaluation in processing, and [7] is a target of similarity evaluation in second processing for feature information F1 in captured image P1 similar to that shown in [2] of FIG. 5. As shown in

  As indicated by lines L8 and L9 between [6-1] to [6-3] and [7] in FIG. 6, partial feature information PF31 to PF33 in recognition targets R1 to R3 and the captured image P1 As a result of the similarity evaluation with the feature information F1 of the imaging target, the final result (the recognition target R1 as matching information) is obtained as follows.

  That is, first, f8 and f9 of the feature information F1 of the imaging target are determined to be similar to the feature information f18 and f19 of the partial feature information PF31, respectively, and it is determined that the other partial feature information is particularly similar. Since there is no such thing, it can be determined that what is similar to the feature information F1 is the partial feature information PF31. Therefore, recognition object R1 corresponding to partial feature information PF31 judged to be similar among candidate information {R1, R2, R3} as collation information in the second process corresponds to an imaging object of captured image P1. The end result is obtained.

  As described above, the process of the second matching unit 6 has been described with the schematic examples in FIGS. 5 and 6, and the respective units 2 to 6 in FIG. 1 have been described. Next, the effects of the present invention as described above will be described.

  First, although it is clearer than the above description, the significance of the partial feature information automatically calculated in the second matching unit 6 is as follows. That is, specifically, the partial feature information is configured by a plurality of recognition objects which are similar to each other among the plurality of recognition objects stored in the storage unit 4 and the candidate information output from the first matching unit 5 In the case where it has been performed, the information corresponds to (expected) feature information calculated from only mutually different parts of the recognition objects that are similar to each other.

  For example, when the similar objects G1 and G2 in FIG. 1 described above are both stored in the storage unit 4 as recognition objects, and the captured image captured by the imaging unit 2 is one obtained by capturing either G1 or G2. It is assumed that candidate information {G1, G2} is output without being distinguishable by the first matching unit 5 because these are similar. In this case, as the partial feature information, feature information (referred to as PF (P1)) calculated from the portion P1 having a slight difference from the similar object G2 is obtained for the recognition object G1, and for the recognition object G2 Feature information (referred to as PF (P2)) calculated from the portion P2 which is slightly different from the similar object G1 is obtained.

  Thus, in the present invention, only partial feature information PF (P1) and PF (P2) calculated from only the portion having a slight difference when comparing the similar objects G1 and G2 are to be compared (reference) As the second matching unit 6 again performs matching using the feature information of the captured image as a query, it is determined with high accuracy which of the similar objects G1 and G2 the captured images are similar to each other. can do.

  In particular, the partial feature information PF (P1) and PF (P2) correspond to the "slight difference" assumed to be the cause of indistinguishability between similar recognition objects G1 and G2 in the first collation unit 5. However, by automatically extracting only the "slight difference" portion in the second matching unit 6 again, when it is used as a single matching object, it becomes a "significant difference" between each other (partial features It is expected that the information PF (P1) and PF (P2) are significantly different). Therefore, in the present invention, even when there are a plurality of similar objects, they can be distinguished, and highly accurate recognition can be realized.

  In addition, even if it is difficult for the first matching unit 5 to distinguish the minute differences between similar objects and to specify which one of the captured images corresponds to, it is possible to distinguish for each group of similar objects as preprocessing. In addition, it plays a role of identifying which group the captured image corresponds to. For example, the first similar target group {G1, G2} in FIG. 1 described above, the second similar target group {G3, G4} in FIG. 2 and the third similar target group {G5, G6} in FIG. In the case where it is stored in the storage unit 4, the first matching unit 5 may specify as the candidate information whether the imaging target of the captured image corresponds to any of the three groups (or does not correspond to any of the three groups). it can. Therefore, even if a large number of groups constituting similar objects exist as recognition objects stored in storage unit 4 and a large number of similar objects in each group also exist, the first comparison in the present invention The part 5 and the second collation part 6 enable highly accurate recognition.

  Hereinafter, supplementary items such as each other embodiment of the present invention will be described as (Supplementary Note 1) to (Supplementary Note 7). The respective supplementary items can be implemented in any combination. Here, the embodiment described above will be referred to as a first embodiment.

(Supplementary Note 1)
Regarding the process (second process described above) of obtaining matching information by evaluating the similarity between the partial feature information of each recognition target belonging to candidate information and the feature information of the captured image in the second matching unit 6 In one embodiment, the feature information is evaluated by the distance or the like to calculate the similarity. Instead of or in addition to this, it is possible to consider the area similarity evaluated by pixel information as follows.

  That is, the area similarity between a partial area formed as a predetermined vicinity area of a feature point in partial feature information of a recognition target in candidate information and a partial area formed as a predetermined vicinity area of a feature point in feature information of a captured image The degree may be evaluated by the sum of squared differences between corresponding pixel values, and the similarity may be determined in the form of an overall score as the similarity based on the feature information in the first embodiment, etc. Instead of the similarity based on the feature information in the first embodiment, it may be used.

  When calculating the area similarity as described above, the storage unit 4 stores, for each recognition target, in addition to the feature point and the local image feature amount, pixel information of a predetermined proximity area of the feature point I shall keep it. In the second matching unit 6, each feature point of the captured image obtained by referring to the stored pixel information of each feature point to be recognized and the pixel information of the captured image acquired from the imaging unit 2 The above-described area similarity can be evaluated using the pixel information in the vicinity of and.

  The shape or the like of the predetermined neighboring area regarding the feature point for defining the partial area as described above may be set in advance such as a rectangular area of a predetermined size centering on the feature point.

(Supplementary Note 2)
In the first matching unit 5, candidate information is obtained on the assumption that a list of recognition objects similar to each other can be obtained. However, when the candidate information is obtained, a recognition target that is determined to be similar to the captured image due to the influence of noise or the like but is dissimilar to other recognition targets (dissimilar recognition There is also a possibility that the target has been included.

  For example, there is a possibility that the recognition objects G1 and G2 similar to each other in FIG. 1 described above and the recognition object G3 in FIG. 2 dissimilar to these may be obtained as candidate information {G1, G2 and G3}. In such a case, if the second matching unit 6 selects partial feature information according to the first embodiment by using the dissimilarity recognition target G3 included in the candidate information {G1, G2, G3} as it is, according to the present invention There is a risk that the intended matching result may not be obtained.

  Therefore, the first matching unit 5 first determines candidate information (referred to as first candidate information) in the same manner as in the first embodiment, and then, as additional processing, the dissimilarity recognition in the first candidate information The second candidate information may be output to the second collating unit 6 by specifying what is considered to be a target, and excluding the specified non-similar recognition target. Specifically, for example, the dissimilarity recognition target may be specified as any one of the following (Method 1) to (Method 3).

  (Method 1) For each recognition target included in the first candidate information, count the number of feature information matching with other recognition targets, and when it is determined that the number is small (a predetermined threshold value) If it is determined that the following is true), the recognition target is specified as a non-similar recognition target.

  At the time of the counting, attention is paid to each feature information of the recognition target, and (1) when there is even one feature information in another recognition object which is determined to match the feature information, the noted feature information As the number, “1” may be additionally counted, or (2) when a predetermined number or more of feature information in another recognition target determined to match the feature information is concerned The feature information may additionally count “1” as the number.

  (Method 2) The first candidate information includes the similarity and the similarity between the feature information of the captured image and the feature information of the recognition target calculated when the first matching unit 5 calculates the first candidate information. Calculation is performed between all pairs of objects, and for each recognition object, a recognition object whose total sum of similarity calculated with other recognition objects is equal to or less than a threshold is specified as a dissimilarity recognition object.

  (Method 3) Each recognition target included in the first candidate information is clustered based on its feature information, and a recognition target belonging to a cluster determined to have a small number of components by threshold determination is specified as a dissimilarity recognition target Do. In addition, recognition targets belonging to the same cluster that are not determined to be small are set as second candidate information. For clustering, a known method such as k-means (k-means method) can be used.

(Supplementary Note 3)
When selecting partial feature information in the second matching unit 6, partial feature information is obtained more reliably as selection of only appropriate feature information that corresponds to a slight difference between recognition targets of candidate information, The following additional processing may be performed in the first embodiment in order to prevent other unnecessary feature information from being erroneously selected as constituting partial feature information.

  That is, the second matching unit 6 performs “normalization of resolution” as an additional process in advance between the recognition targets belonging to the candidate information, and then the feature information as described in the first embodiment. The match determination between the two is performed, and the feature information not matching is selected as partial feature information of each recognition target.

  The normalization of the resolution may be, for example, as follows. First, as the correspondence relationship already obtained when the candidate information is determined in the first matching unit 5, the correspondence relationship between the feature point coordinates (group) of the captured image and the feature point coordinates (group) of the recognition target A homography matrix representing a correspondence relationship between the coordinate groups (mapping relationship between coordinates) is obtained for each recognition target. Next, feature point coordinates of the recognition target may be transformed into a coordinate system in the captured image using the homography matrix. Thus, the normalization of resolution is completed by unifying the feature point coordinates of each recognition target of the candidate information with those represented in the coordinate system in the captured image.

  As for the homography matrix to be obtained, if the similarity is evaluated using RANSAC when the candidate information is obtained in the first matching unit 5, the homography matrix already obtained at this time is used. You should do it.

  In the example of FIG. 5, the coordinate systems of the feature points in the feature information F31, F32 and F33 of the recognition targets R1, R2 and R3 constituting the candidate information are respectively the coordinates of the feature points in the feature information F1 of the captured image The homography matrices to be converted into a system are determined as H31, H32, and H33, and the feature information H31 (F31), H32 (F32), and H33 (F33) in the converted coordinate system are obtained as [4-1] in FIG. , [4-2], and [4-3].

  In addition, when storing in the storage unit 4, feature information may be stored after performing the equivalent of the above-described resolution normalization among recognition objects that are similar to each other. In this case, it is not necessary to perform resolution normalization in the second matching unit 6. The following first method or second method can be used as a method of normalization when storing in the storage unit 4.

  In the first method, one of the recognition objects similar to each other is selected as a reference recognition object for unifying the coordinate system, and the image of the reference recognition object is regarded as corresponding to the above-mentioned captured image, Can be normalized by the same processing as in (processing of obtaining correspondence between feature point coordinates of recognition objects, obtaining a homography matrix representing the correspondence, and mapping with the matrix) and storing the result in the storage unit 4 . The manager or the like may give prior knowledge as to which recognition objects are similar to each other.

  In the second method, for example, when similar objects G1 and G2 exist as recognition targets for the upper and lower covers of the same book shown in FIG. As an image at the time of obtaining feature information of both covers as a recognition target, an image obtained by imaging both covers with the same arrangement in the same camera may be used.

  When the second matching unit 6 normalizes the resolution, the feature information of the recognition target stored in the storage unit 4 as in the first method and the second method described above is There is an effect that the time and effort to prepare as "data" is omitted.

(Supplement 4)
In the first embodiment, when the second matching unit 6 obtains partial feature information, it is determined whether the feature information matches the feature information of another recognition target for each recognition target in the candidate information, Other than the feature information for which the coincidence determination was obtained was selected as partial feature information. With regard to the coincidence determination, (1) the coincidence determination may be made when the information matches the feature information of at least one other recognition object, or (2) the features of other recognition objects of a predetermined number or more If the information matches, the match determination may be made. In addition, the example of the above-mentioned FIG.5 and FIG.6 is an example which made the coincidence judgment by the said former (1).

(Supplementary Note 5)
In the first embodiment, a plurality of recognition targets similar to the recognition target of the captured image among the plurality of recognition targets stored in the storage unit 4 by obtaining the candidate information in the first matching unit 5 are: So to speak, "dynamically", the process of sorting was carried out. Instead of this, when so-called “static” is stored in advance in the storage unit 4, information as to which one of the plurality of recognition objects is a similar object is also attached to each recognition object as a data structure It may be added and stored.

  Specifically, as additional processing to the first embodiment, information (listed as similar target enumeration information) listing other similar recognition targets as data structures for each recognition target stored in the storage unit 4 It is good to attach it. The first matching unit 5 can efficiently obtain candidate information by the following (Process 1) or (Process 2) by using the similar target enumeration information given by the data structure.

  (Process 1) Similar to the first embodiment, the candidate information is determined, and then the similar target enumeration information on each recognition target in the candidate information is referred to, and the recognition is not included in the recognition target in the candidate information. When there is a target (departure recognition target), the deviation recognition target is added to the candidate information, and then output to the second matching unit 6.

  For example, when the similar recognition target is the cover of each volume of the entire collection of books (such as the cover of each volume of the encyclopedia), if at least any one volume is obtained as the candidate information, all the volumes are automatically candidate information It can be done.

  (Process 2) Contrary to the above, first, the following limitations are added to the first embodiment, and then candidate information is obtained. That is, the feature information of the recognition target read out from the storage unit 4 as a reference to the feature information of the captured image which is a query is limited to the recognition target of only a predetermined part of the similar target enumeration information. Ask for information. Next, an additional process similar to (Process 1) is performed on the obtained candidate information, and the additional information is output to the second collating unit 6.

  For example, when the similar recognition target is the cover of each volume of the entire volume of the book, only one of them is read out from the storage unit 4 and one volume read out is similar to the captured image. If obtained as candidate information, the entire volume can be automatically set as candidate information.

  Note that which of the recognition objects similar to each other may be read out from the storage unit 4 may be set in advance. For example, information on whether or not to be read out is associated with each recognition target in the storage unit 4, and the first collation unit 5 may read out only the recognition target set as the read target.

  According to the above (Process 1), originally, a series of similar objects should all be obtained as candidate information, but a deviation recognition object erroneously deviated from the candidate information not judged as an effect of noise or the like in the captured image , Can be automatically added to the candidate information. Further, according to the above (Process 2), even if a large number of recognition objects similar to each other are stored in the storage unit 4, it is sufficient to evaluate the similarity for only a part of them. The amount of calculations in 5 can be reduced. Furthermore, communication between the storage unit 4 and the first collating unit 5 is performed via the network, for example, because the storage unit 4 is constructed on a server separate from the information terminal device 1 as in the case of reduction of the calculation amount. In the case where is performed, the amount of transmission can be suppressed.

(Supplement 6)
In the above (supplement 5), it is assumed that the similar target enumeration information is stored in advance in the storage unit 4 as so-called "static" in each recognition target. Instead of storing the similar target enumeration information in the storage unit 4, the first matching unit 5 determines a deviation recognition target as equivalent to the similarity recognition target information as “dynamic” as it is. Then, the candidate information may be added similarly to (Process 1) or (Process 2) of (Supplementary 5) above.

  Therefore, the storage unit 4 stores text information on each recognition target instead of the similar target enumeration information for each recognition target. For example, if the recognition target is a product, text describing the product or one or more keywords (tags) related to the product can be used as text information. Then, the first matching unit 5 can dynamically generate itself equivalent to the similar target enumeration information as follows.

  That is, in the same manner as in the first embodiment, the candidate information is determined, and for each recognition target listed in the candidate information, the text information is used as a search key, and the recognition target stored in the storage unit 4 is Among them, those which are determined to be similar in text information are searched. Then, if there is any recognition information stored in the storage unit 4 found by the search for the recognition target of the candidate information and is not included in the candidate information, this will be described in (Supplementary 5). It is regarded as "the deviation recognition object". Furthermore, (Process 1) of (Supplementary 5) may be performed with respect to the deemed “departure recognition target”.

  In the above search, existing technology in the natural language processing field or the like may be used to determine whether the text information is similar or not.

(Supplement 7)
With regard to the color channels of the image for calculating feature information as feature points and local image feature quantities in the calculation unit 3, when the image is configured with a plurality of predetermined color channels such as RGB, only a part of them is used You may do so. In this case, even when the first matching unit 5 and the second matching unit 6 evaluate the similarity between the pieces of feature information, a part of the color channel similar to that used in the calculating unit 3 is used.

  That is, when the image is configured by a predetermined plurality of color channels such as RGB, the entire first embodiment may be implemented using only a part (for example, only the R channel). As a result, it becomes possible to distinguish and handle a portion having the same shape and the like but different only in color as a slight difference portion in the similar object. The user or the like may designate which channel among the predetermined color channels to limit.

  DESCRIPTION OF SYMBOLS 1 ... Information terminal device, 2 ... Imaging part, 3 ... Calculation part, 4 ... Storage part, 5 ... 1st collation part, 6 ... 2nd collation part

Claims (11)

An imaging unit that captures an imaging target and obtains a captured image;
A calculation unit that calculates a feature point and a local image feature amount near the feature point from the captured image and outputs the feature amount as feature information;
For each of a plurality of recognition objects, feature information of each recognition object obtained by accessing a database storing feature information of the same type as that calculated by the calculation unit from the image, and a feature calculated from the captured image A first collating unit that evaluates the similarity between information and obtains candidate information as a list of recognition objects determined to be similar;
Among the recognition objects listed in the candidate information, the matching of the feature information is determined, and the feature information for which the determination of the matching is not made is a partial feature in each of the recognition objects listed in the candidate information It obtains as information, evaluates the similarity between the determined partial feature information of each recognition target and the feature information calculated from the captured image, and recognizes the recognition target corresponding to the partial feature information determined to be similar, An information terminal device, comprising: a second collating unit that determines the subject as an imaging target in the captured image. When the second matching unit determines that the feature information matches with each other among the recognition targets listed in the candidate information,
The information terminal device according to claim 1, wherein when it is determined that each feature information is similar to feature information in a predetermined number or more of other recognition targets, a match is determined with respect to the feature information. . When the second matching unit evaluates the similarity between the partial feature information of each of the obtained recognition objects and the feature information calculated from the captured image,
Distance between the partial feature information and the feature information, and / or evaluation based on the consistency, the vicinity of the partial area of the feature point of this partial feature vicinity of the partial area and the feature information of the feature point information The information terminal device according to claim 1 or 2, characterized in that: The second collation unit relates to a recognition target listed in the candidate information as a target for which the partial feature information is to be obtained.
When there is a recognition target determined to be dissimilar to another recognition target similar to each other by evaluating the similarity of the feature information of each recognition target in the candidate information,
The information terminal device according to any one of claims 1 to 3, wherein the recognition target determined to be non-similar is excluded from the target for which the partial feature information is required. The second matching unit determines in advance whether the feature information matches with each other among the recognition objects listed in the candidate information.
The information terminal device according to any one of claims 1 to 4, wherein the coincidence is determined after normalizing the resolution of the feature information of each recognition object. The second matching unit normalizes the resolution by:
A feature is to obtain a relationship in which the coordinates of the feature point in the feature information of each recognition target are converted into the coordinates of the feature point in the feature information of the captured image, and perform normalization based on the conversion relationship. The information terminal device according to. In the database, similar target enumeration information in which other recognition targets similar to itself are listed for each of a plurality of recognition targets is stored in addition to the same kind of feature information calculated by the calculation unit from the image. Yes,
The first matching unit evaluates the similarity, and obtains candidate information as a list of recognition targets determined to be similar, and further,
For each recognition target listed in the candidate information, the similar target enumeration information is referred to, and when there is a deviation recognition target as a recognition target not included in the candidate information, the deviation recognition target is The information terminal device according to any one of claims 1 to 6, wherein the information terminal device is added to the candidate information. In the database, similar target enumeration information in which other recognition targets similar to itself are listed for each of a plurality of recognition targets is stored in addition to the same kind of feature information calculated by the calculation unit from the image. Yes,
The first matching unit evaluates the similarity and obtains candidate information as a list of recognition targets determined to be similar:
The targets for which the similarity is to be evaluated are limited to only a part of a series of recognition targets made similar to each other by the similar target enumeration information in the database, and at least one of the recognition targets of the part The candidate information is obtained as a list of a series of recognition objects that are considered to be similar to each other by the similar object enumeration information when it is determined that the similarity is the same. The information terminal device according to any one of 6. In the database, for each of a plurality of recognition objects, text information on the recognition objects is stored in addition to the same kind of feature information calculated by the calculation unit from the image,
The first matching unit evaluates the similarity, and obtains candidate information as a list of recognition targets determined to be similar, and further,
By searching the database using the text information for each recognition target listed in the candidate information, it is a deviation recognition target as a recognition target not included in the candidate information, and the text information is similar The information according to any one of claims 1 to 6, characterized in that when a deviation recognition target determined to be determined is found from the database, the deviation recognition target is added to the candidate information. Terminal equipment. The calculation unit calculates the feature information using only a specific color channel designated in advance.
The first matching unit evaluates the similarity using only the specific color channel,
The information terminal apparatus according to any one of claims 1 to 9, wherein the second matching unit determines the match using only the specific color channel and evaluates the similarity.   A program that causes a computer to function as the information terminal device according to any one of claims 1 to 10.