JP5347897B2 - Annotation apparatus, method and program - Google Patents

Description

  The present invention relates to an annotating apparatus, method, and program for analyzing an image characteristic and adding an annotation.

  An image annotation that gives information (annotation (annotation), tag, label) indicating the content of an image to an image is generally a manual operation, and thus has a heavy load, wastes time, and is expensive. In order to solve such problems, research has been conducted to automatically execute image annotation. In the automatic image annotation system, a single learning device is used to learn semantic categories (annotations), and thus various image feature amounts are combined into one feature vector (for example, Non-Patent Document 1). Then, the learning device learns the relationship between the feature vector including the feature amount of the learning image and the annotation given to the learning image using the learning image to which the annotation is given. By referring to the learning result, it is possible to determine an appropriate annotation for the image to be annotated.

  However, in such a method, since the relationship between the feature amount of the image and the specific annotation is weak, there is a problem that an appropriate annotation may not be given to the given image.

  The present invention has been made in view of the above, and an object thereof is to provide an annotation providing apparatus, method, and program capable of more appropriately determining an annotation to be added to an image.

  In order to solve the above-described problems and achieve the object, the present invention provides an index storage unit that stores an index in which a plurality of predetermined images and feature amounts of the images are associated with each other, and the predetermined image. A correspondence storage unit that stores the plurality of annotations in association with each other, a division unit that divides the target image to be annotated into a plurality of divided images, and analyzes the divided images to extract feature amounts of the divided images A feature extraction unit, a search unit that searches the index storage unit for a predetermined number of images having a high degree of similarity between the corresponding feature amount and the extracted feature amount, and the searched image For each of the associated annotations, the smaller the appearance frequency of the annotation in the correspondence storage unit is, and the larger the similarity of the corresponding image is, the larger the correspondence is. A calculation unit that calculates a score of the annotation that has a larger value as the feature quantities of the plurality of images are similar to each other, and determines the annotation to be given to the target image with priority on the annotation having the higher score A first determination unit that performs the annotation.

  The present invention also relates to a method and a program that can be executed by the above apparatus.

  According to the present invention, it is possible to more appropriately determine the annotation to be added to the image.

FIG. 1 is a block diagram illustrating a configuration example of an information processing system including an annotation assigning apparatus according to the present embodiment. FIG. 2 is a diagram for explaining an outline of a function for determining a classification code from a plurality of searched images. FIG. 3 is a flowchart showing the overall flow of the annotation determination process in the present embodiment. FIG. 4 is a flowchart showing the overall flow of the score calculation process. FIG. 5 is a flowchart showing the overall flow of the element a calculation process. FIG. 6 is a flowchart showing the overall flow of the element b calculation process. FIG. 7 is an explanatory diagram showing an example of image pattern detection processing. FIG. 8 is a diagram illustrating an example of the data structure of the trust table. FIG. 9 is a diagram illustrating an example of ranks, similarities, and associated classification codes for each learning image. FIG. 10 is a diagram illustrating a calculation example of the mutual similarity between learning images. FIG. 11 is a diagram illustrating an example of a hardware configuration of the annotation assigning apparatus according to the present embodiment.

  Exemplary embodiments of an annotation assigning apparatus, method, and program according to the present invention will be described below in detail with reference to the accompanying drawings.

  The annotation giving apparatus according to the present embodiment determines an annotation for a target image to be annotated with reference to a learning result obtained by learning using learning data in which a learning image is associated with a plurality of annotations. . The annotation giving apparatus according to the present embodiment can be applied to, for example, an apparatus for giving an annotation corresponding to “Wien classification” defined for classifying graphic elements of a mark on a target image representing a trademark. .

  The Vienna classification has a hierarchical structure in which all graphic elements are divided into a large classification, a medium classification, and a small classification. A number according to a predetermined code system (hereinafter referred to as a classification code) is assigned to each of the major classification, middle classification, and minor classification. For example, the classification code = “1.1” is defined to represent “star, comet”.

  Hereinafter, an example will be described in which the annotation adding apparatus according to the present embodiment is configured to determine the classification code of the Vienna classification to be given to the target image. If the classification code is determined, the content (annotation) corresponding to the classification code can be determined. Note that applicable annotations are not limited to annotations corresponding to the Vienna classification code. An arbitrary annotation can be applied as long as it can be given to the learning image.

  The annotation giving apparatus according to the present embodiment first divides a target image into a plurality of divided images. Next, the top k learning images whose feature amounts are similar to the divided images are obtained, and a classification code corresponding to the learning image is acquired. Next, for the obtained classification code, a score representing the priority of the classification code is calculated according to the appearance frequency of the classification code in the learning data, the similarity between the divided image and the searched image, and the like. . Then, a predetermined number of classification codes having higher scores are determined as classification codes to be given. Thereby, it is possible to appropriately determine a classification code having a strong relationship with the feature amount of the image.

  FIG. 1 is a block diagram illustrating a configuration example of an information processing system including an annotation assigning apparatus according to the present embodiment. As illustrated in FIG. 1, the information processing system has a configuration in which an annotation assigning device 100 and a plurality of user terminals 200 a and 200 b are connected via a network 300. The network 300 can employ any network configuration such as a LAN (Loacal Area Network) and the Internet. The user terminals 200a and 200b are terminal devices used by a user such as a PC (personal computer). In addition, since the user terminals 200a and 200b have the same configuration, the user terminals 200 may be simply referred to as the user terminal 200 below. Further, the number of user terminals 200 is not limited to two.

  The user terminal 200 transmits a target image representing an image to which a classification code is assigned to the annotation giving apparatus 100. Further, the user terminal 200 receives the classification code assigned by the annotation assigning device 100 to the target image from the annotation assigning device 100 and displays it on a display unit (not shown) such as a display.

  The annotation assigning device 100 determines a classification code to be assigned to the designated target image, and outputs the determined classification code to the user terminal 200 or the like. Details of the classification code determination method will be described later.

  The system configuration is not limited to the configuration shown in FIG. For example, it is configured to accept the designation of the target image in the annotation giving apparatus 100 and output the classification code determined for this target image to a display unit (not shown) provided in the annotation giving apparatus 100 or the like. Also good.

  Next, the functional configuration of the annotation assigning apparatus 100 will be described. As shown in FIG. 1, the annotation assigning apparatus 100 includes an index storage unit 151, a correspondence storage unit 152, a rule storage unit 153, an image reception unit 101, a division unit 111, a feature extraction unit 112, and a search unit. 113, a calculation unit 114, a first determination unit 115, a classification unit 116, a second determination unit 117, a pattern detection unit 118, a third determination unit 119, an output unit 121, a communication unit 122, It has.

  The index storage unit 151 stores an index in which a plurality of learning images are associated with feature amounts of the learning images. The index stored in the index storage unit 151 is referred to when the search unit 113 searches for a learning image similar to the target image. The index is configured so that a plurality of learning images whose feature amounts are similar to a given image can be searched by learning using learning data in which a learning image is associated with a plurality of classification codes.

  Note that index learning is performed, for example, as follows. First, the feature amount of the learning image is extracted from the learning image included in the learning data by the same method as the feature extraction unit 112. Then, an index for searching for a learning image whose feature quantity is similar to that of the divided image is learned by, for example, a k-Nest (K Nearest Neighbors) method. In this case, the index storage unit 151 can be configured to store, for example, an index represented by a kd (k-dimensional) tree that can be used in the KNN method.

  Note that the index learning method and data structure are not limited to the KNN method and the kd tree. Any learning method and corresponding data structure can be applied as long as a learning image having a similar feature amount can be searched.

  The correspondence storage unit 152 stores a learning image stored in the index storage unit 151 in association with a plurality of classification codes assigned in advance to the learning image. The correspondence storage unit 152 is referred to when acquiring a classification code corresponding to the searched learning image.

  The rule storage unit 153 stores a rule for classifying the divided image into one of a plurality of classes corresponding to each classification code in accordance with the feature amount of the divided image. For example, the rule storage unit 153 stores a rule learned by the random forest method using learning data in which a learning image is associated with one classification code for the learning image.

  The rule learning is executed as follows, for example. First, the feature amount of the learning image is extracted from the learning image included in the learning data by the same method as the feature extraction unit 112. Then, a rule for searching for a learning image whose feature quantity is similar to that of the divided image is learned by a random forest method. In this case, the rule storage unit 153 stores rules represented by a plurality of decision trees that determine classes to be classified by branching according to feature amounts.

  The rule learning method is not limited to the random forest method. If the random forest method is used, the so-called dimensional curse problem caused by using a high-dimensional feature vector obtained by vectorizing a large number of features can be solved. That is, in the random forest method, a part of the feature amounts included in the feature vector is selected at random, and the divided images are classified according to the selected feature amount. Further, such classification is performed on each of the plurality of decision trees, and finally a class to be classified is determined by voting (majority decision) on the result of each decision tree. Therefore, a class (classification code) to be classified more objectively can be determined.

  In the rule storage unit 153, one classification code is associated with one class. Therefore, for the divided images classified by the classification unit 116, one classification code corresponding to the classified class is obtained. On the other hand, in the above-described correspondence storage unit 152, a plurality of classification codes are associated with one learning image. In addition, the search unit 113 searches for a plurality of similar learning images with respect to the divided image. Therefore, a plurality of classification codes are obtained as classification codes to be assigned to the divided images searched by the search unit 113.

  The communication unit 122 transmits / receives various information to / from an external device such as the user terminal 200. For example, the communication unit 122 receives the target image or information for identifying the target image from the user terminal 200. Further, the communication unit 122 transmits the classification code determined for the target image to the user terminal 200.

  The image receiving unit 101 receives the designation of the target image. For example, the image receiving unit 101 receives a target image received from the user terminal 200 via the communication unit 122. Note that the method for receiving the target image is not limited to this. For example, information for identifying the target image may be received, and the target image may be acquired from the inside of the annotation giving apparatus 100 or an external device based on this information.

  The dividing unit 111 divides the target image into a plurality of divided images. For example, the dividing unit 111 recognizes a graphic area and a character area included in the target image, and divides them into a plurality of divided images corresponding to the respective areas. The dividing method is not limited to this, and any dividing method can be applied.

  The feature extraction unit 112 analyzes each divided image and extracts a feature amount of the divided image. As the feature amount, any conventionally used index such as a color histogram, color scheme, edge, texture, composition, etc. can be applied.

  The search unit 113 searches the index storage unit 151 for a plurality of learning images corresponding to feature amounts similar to the extracted feature amounts for each divided image. The search unit 113 searches the index storage unit 151 for k learning images similar to the feature amount extracted from the divided image, for example, by the KNN method. The applicable search method is not limited to the KNN method, and for example, an approximate nearest neighbor (ANN) method may be applied.

  The calculation unit 114 calculates the score of the classification code for each classification code associated with the learning image searched by the search unit 113. Specifically, the calculation unit 114 has a low appearance frequency of the classification code in the learning data used for learning the index of the index storage unit 151, has a high similarity between the divided image and the learning image, and associates them. The score which becomes a large value is calculated so that the feature-value of the obtained some learning image is mutually similar. Details of the score calculation method will be described later.

  The first determination unit 115 prioritizes a classification code having a large calculated score and determines it as a classification code to be assigned to the target image. For example, the first determination unit 115 determines a predetermined number of classification codes having higher scores as classification codes to be assigned to the target image. As will be described later, the second determination unit 117 and the third determination unit 119 also determine the classification code to be assigned. Then, the output unit 121 determines and outputs a classification code having a high trust value (described later) as a classification code to be finally added to the target image from the determined classification code.

  The classification unit 116 classifies each divided image divided by the dividing unit 111 into any class using the feature amount extracted by the feature extraction unit 112 and the rule stored in the rule storage unit 153. For example, the classification unit 116 classifies the divided images into classes according to the feature amounts using the learned rules by the random forest method.

  The second determination unit 117 determines the classification code associated with the classified class as the classification code to be assigned to the target image.

  The pattern detection unit 118 detects a predetermined image pattern from the target image. As the image pattern, for example, an easily identifiable geometric pattern such as a circle, an ellipse (26.1 in the Wien classification), and a quadrangle (26.4 in the Wien classification) can be applied.

  The third determination unit 119 determines a classification code associated with the detected image pattern in advance as a classification code to be assigned to the target image. For example, when a square is detected, the third determination unit 119 determines a classification code to which “26.4.1” that is a classification code corresponding to the “square” of the Vienna code is assigned.

  The output unit 121 outputs the determined classification code. In the present embodiment, the output unit 121 integrates the classification codes determined by the first determination unit 115, the second determination unit 117, and the third determination unit 119, and the credit value is obtained from all the classification codes. A high predetermined number of classification codes are selected and output as final classification codes. The trust value is a value representing the certainty of the classification code. The credit value is calculated in advance, for example, when learning is performed using learning data, and is stored in a predetermined storage unit (not shown) as a table (hereinafter referred to as a credit table) in which classification codes and credit values are associated with each other. Keep it. Details of the trust table will be described later.

  As described above, in this embodiment, A. A function for determining a classification code from a plurality of images searched by the KNN method or the like (hereinafter referred to as function A); A function for determining a classification code corresponding to a class classified by the random forest method (hereinafter referred to as function B), C.I. The classification code is determined by three functions: a function for determining a classification code corresponding to an image pattern detected from the target image (hereinafter referred to as function C).

  The annotation giving apparatus 100 of the present embodiment only needs to have at least the function A. With function A, it is possible to determine a predetermined number of classification codes whose scores of classification codes calculated according to the appearance frequency of classification codes in the learning data are higher as classification codes to be given. That is, it is possible to appropriately determine a classification code having a strong relationship with the image feature amount. If the function B is further provided, the problem of the curse of dimension can be solved as described above, and the classification code can be determined more objectively.

  Furthermore, if the function C is provided, the problem that the classification codes are unbalanced can be solved. The problem that the classification codes are unbalanced is that, for example, a simple image pattern such as a circle or a rectangle is likely to be included in a lot of learning data. Therefore, there is a classification code corresponding to such an image pattern. It is a problem that it will be included in many learning results.

  According to the function C, it is possible to easily determine a classification code corresponding to an image pattern that can be included in many images. Further, if the classification code excluding the classification code that can be determined by the function C is determined by the functions A and B, it is possible to avoid that some classification codes are included in many learning results.

  When both functions A and B are provided, the classification codes determined by each may be divided so as not to overlap. For example, the function B may be configured to select and learn a classification code that can be learned from learning data with a small amount of data, and to determine other classification codes using the function A.

  Next, an outline of function A will be described. FIG. 2 is a diagram for explaining an outline of a function for determining a classification code from a plurality of images searched by the KNN method or the like.

  As shown in FIG. 2, the input target image 21 is divided into a plurality of divided images 22a to 22c. Note that the number of divisions is not limited to three. Although omitted in FIG. 2, feature amounts are extracted by the feature extraction unit 112 for each divided image.

The search unit 113 searches the divided images for learning images with similar feature amounts extracted by the KNN method with reference to the index storage unit 151. Thereby, a search result 23 for each divided image is obtained. The search result 23 includes k / n (n is the number of divisions) similar learning images for each divided image. Each of the obtained k / n learning images is associated with a similarity to the divided image. As a result, k learning images are obtained for the target image. FIG. 2 shows an example in which the obtained k learning images are arranged like Img ₁ to Img _{k in} descending order of similarity. In FIG. 2, the image group 24 represents the obtained k learning images.

  The calculation unit 114 acquires the classification code associated with the learning image for each divided image obtained in this manner from the correspondence storage unit 152. Then, the calculation unit 114 calculates the appearance frequency, the similarity, the rank representing the order of similarity, and the mutual similarity (described later) of the feature quantities of the associated learning images with respect to the obtained classification code. Use to calculate the score of the classification code. Thereby, the scored classification code list 25 is obtained.

  Next, the annotation determination process by the annotation assigning apparatus 100 according to the present embodiment configured as described above will be described. FIG. 3 is a flowchart showing the overall flow of the annotation determination process in the present embodiment.

  First, the image receiving unit 101 receives a designated target image (step S301). Thereafter, function A (steps S302 to S307), function B (steps S302 to S309), and function C (steps S310 and S311) are respectively executed. Each function can be executed in any order. Moreover, you may comprise so that each function may be performed in parallel.

  As processing common to function A and function B, first, the dividing unit 111 divides the target image into a plurality of divided images (step S302). Further, the feature extraction unit 112 extracts the feature amount of the divided image from each divided image (step S303).

  Next, as processing unique to function A, the search unit 113 searches the index storage unit 151 for k learning images corresponding to feature quantities similar to the extracted feature quantities (step S304). The search unit 113 acquires the classification code corresponding to the searched learning image from the correspondence storage unit 152 (step S305).

  Next, the calculation unit 114 executes a score calculation process for calculating the score of the acquired classification code (step S306). Details of the score calculation process will be described later. Next, the first determination unit 115 determines a predetermined number of classification codes having higher calculated scores as classification codes to be assigned to the target image (step S307).

  As processing unique to the function B, the classification unit 116 refers to the rule storage unit 153 and classifies each divided image into any class by the random forest method (step S308). Next, the second determination unit 117 determines a classification code corresponding to the classified class as a classification code to be given to the target image (step S309).

  As processing unique to the function C, the pattern detection unit 118 detects a predetermined image pattern from the target image (step S310). Details of the image pattern detection process will be described later. Next, the 3rd determination part 119 determines the classification code predetermined with respect to the detected image pattern as a classification code provided to a target image (step S311).

  After the classification code is determined by each function (step S307, step S309, step S311), the output unit 121 selects an optimal classification code from the determined classification codes and finally outputs the classification code (Step S312). The output unit 121 outputs the determined classification code, for example, to the user terminal 200 via the communication unit 122 (step S312), and ends the annotation determination process.

  Next, details of the score calculation process in step S306 will be described. FIG. 4 is a flowchart showing the overall flow of the score calculation process.

  First, the calculation unit 114 acquires an unprocessed classification code from the classification code acquired in step S305 in FIG. 3 (step S401). Note that the calculation unit 114 acquires one unprocessed classification code from the classification code group including all the classification codes acquired for each divided image.

Next, the calculation unit 114 executes element a calculation processing for calculating an element a (factor_a) that is a value (first element value) used for calculating a score (step S402). The element a is a value that increases as the degree of similarity with respect to the divided images increases and the order decreases when the similarities are arranged in descending order. Details of the element a calculation process will be described later. The calculation formula for the element a is expressed by the following formula (1).

Here, code _r represents the r-th (1 ≦ r ≦ number of classification codes) classification code among the acquired classification codes. factor_a (code _r ) represents the value of element a for calculating a score for code _r . SimScore (Img _i ) represents the similarity to the divided image obtained for Img _i at the time of search. Img _i represents the learning image code _r is included in the IM (code _r) is the set of correspondence is learning image. base is a predetermined radix. rank (Img _i ) represents the order (rank) of the magnitude of similarity of the learning image Img _{i in} the k search results. The rank is an integer value of 1 to k, and the value decreases as the similarity degree increases.

The base is set to a value less than 1 (for example, 0.95). Thus, power of base to rank the (Img _i) and index, can be made smaller as the rank (Img _i) is large. It may be configured so as to index the rank _(Img i) -1.

After calculating the element a, the calculation unit 114 executes element b calculation processing for calculating the element b (factor_b), which is another value (second element value) used for calculating the score (step S403). . The element b is a value such that when a plurality of learning images are associated with the classification code, the feature amount of each learning image increases as they become similar to each other. When three or more learning images are associated with the classification code, a mutual similarity indicating a degree of similarity between the feature quantities is calculated for all two learning image combinations, and element b Is added. Details of the element b calculation process will be described later. The calculation formula for the element b is expressed by the following formulas (2) to (5).

factor_b (code _r ) represents the value of element b for calculating the score for code _r . In the formula (4), w (Img _s , Img _t ) represents the mutual similarity between Img _s and Img _t . _{(4) dist (Img s,} Img t) contained in the formula represents the Euclidean distance between the feature vector comprising a feature value of a feature vector and Img _t including the feature quantity of Img _s in feature vector space . Further, C included in the equation (4) is a predetermined constant, and for example, C = 100 is used.

After calculating the element b, the calculation unit 114 calculates an IDF (Inverse Document Frequency) of the classification code (code _r ) (step S404). IDF of code _r is calculated by the following equation (6).
idf (code _r ) = log (N / df (code _r )) (6)

N represents the total number of learning images stored in the index storage unit 151. df (code _r ) represents the number of learning images associated with code _r among learning images stored in the index storage unit 151.

Next, the calculation unit 114 calculates score (code _r ) representing the score of the classification code (code _r ) by using the calculated IDF, element a, and element b according to the following equation (7) (step S405).
score (code _r ) = α × idf (code _r ) × factor_a (code _r )
+ Β × factor_b (code _r ) (7)

  α and β represent predetermined constants. As described above, the calculation unit 114 calculates the linear sum of the product of the IDF and the element a and the element b as a score.

  Next, the calculation unit 114 determines whether or not all the classification codes have been processed (step S406), and if not (step S406: No), obtains the next unprocessed classification code. The process is repeated (step S401). When all the classification codes have been processed (step S406: Yes), the score calculation process ends.

  Next, details of the element a calculation process in step S402 will be described. FIG. 5 is a flowchart showing the overall flow of the element a calculation process.

First, the calculation unit 114 initializes the value of the element a (factor_a (code _r )) corresponding to the currently processed classification code (code _r ) to 0 (step S501). Next, the calculation unit 114 acquires an unprocessed learning image (hereinafter referred to as Img _i ) among the searched learning images (step S502). Next, the calculation unit 114 determines whether or not the classification code code _r is associated with the learning image Img _i (step S503). The calculation unit 114 determines whether a desired classification code is associated with the learning image Img _i by referring to the correspondence storage unit 152, for example.

When the classification code code _r is associated with the learning image Img _i (step S503: Yes), the calculation unit 114 calculates the base rank (Img _i ) power, and the obtained value and the learning image Img _i The product of the similarity (SimScore (Img _i )) obtained with respect to is added to factor_a (code _r ) (step S504).

When the classification code code _r is not associated with the learning image Img _i (step S503: No), and after step S504, the calculation unit 114 determines whether or not all learning images have been processed ( Step S505).

  When all the learning images have not been processed (step S505: No), the calculation unit 114 acquires the next unprocessed learning image and repeats the processing (step S502). When all the learning images have been processed (step S505: Yes), the element a calculation process ends.

  Next, details of the element b calculation process in step S403 will be described. FIG. 6 is a flowchart showing the overall flow of the element b calculation process.

First, calculation unit 114, the value of the classification is currently processing code (code _r) to the corresponding element _{b (factor_b (code r))} , and the counter i is initialized to 0 (step S601). Also, the calculation unit 114 initializes the counter j to i + 1 (step S602).

Next, the calculation unit 114 acquires a learning image (referred to as Im _i ) in which the rank is i-th (step S603). In addition, the calculation unit 114 acquires a learning image (Im _j ) in which the rank is jth (step S604). Next, the calculation unit 114 determines whether or not the classification code code _r is associated with both the acquired Im _i and Im _j (step S605).

If both are associated (step S605: Yes), calculation unit 114 calculates the mutual similarity between Im _i and Im _j, is added to FACTOR_B (code _r) (step S606). The mutual similarity is calculated so as to increase as the Euclidean distance in the feature vector space decreases as in the above equation (4).

When the classification code code _r is not associated with both Im _i and Im _j (step S605: No), and after step S606, the calculation unit 114 adds 1 to the value of j (step S607). . Next, the calculation unit 114 determines whether j is smaller than k (step S608). If j is smaller (step S608: Yes), the process returns to step S604 and repeats the process. When j is greater than or equal to k (step S608: No), the calculation unit 114 adds 1 to the value of i (step S609). Next, the calculation unit 114 determines whether i is smaller than k−1 (step S610). If i is smaller (step S610: Yes), the process returns to step S603 and repeats the process. If i is greater than or equal to k−1 (step S610: No), the element b calculation process ends.

  Next, details of the image pattern detection processing in step S310 will be described. FIG. 7 is an explanatory diagram showing an example of image pattern detection processing. FIG. 7 shows an example in which an image pattern including one or more quadrangles is detected.

  The pattern detection unit 118 removes noise from the target image (step S701) and detects a contour (step S702). Next, the pattern detection unit 118 calculates vertices of each detected contour (step S703), and acquires a contour having four vertices (step S704). Next, the pattern detection unit 118 calculates the length of each side of the contour and the size of the corner (step S705). The pattern detection unit 118 determines that the outline is rectangular (26.4.2 of the Wien classification), square (26.4.1 of the Wien classification), or other than the calculation result of the length of the side and the size of the corner. Whether it is a rectangle can be determined. That is, the pattern detection unit 118 can detect image patterns that are rectangular and square.

  If a plurality of contours are detected, the pattern detection unit 118 further detects the positional relationship between the contours (step S706). For example, the pattern detection unit 118 detects that two rectangles overlap each other by comparing the magnitude relationship between the coordinates of four vertices of each rectangle. In this case, for example, an image pattern which is “a plurality of juxtaposed / combined or intersecting quadrangles (Wien classification 26.4.9)” is detected. Similarly, image patterns corresponding to other classification codes of Wien classification (26.4.4, 26.4.7, and 26.4.8 in FIG. 7) can be detected.

  Next, the trust table referred to when the output unit 121 determines the optimum classification code will be described. FIG. 8 is a diagram illustrating an example of the data structure of the trust table. As shown in FIG. 8, the credit table stores the credit value of the classification code when detected by each of the function A (KNN), the function B (random forest), and the function C (pattern detection) for each classification code. To do.

  The trust value is calculated in advance when, for example, a classification code is detected for each function in the learning data, and stored in the trust table. As the credit value, for example, an F value that is a harmonic average of the recall rate and the matching rate can be used. The credit value is not limited to this, and any index can be applied as long as it represents the certainty of the classification code. For example, either the recall rate or the matching rate may be used as the credit value.

  The output unit 121 refers to such a credit table, and obtains a credit value corresponding to each classification code determined by the first determination unit 115, the second determination unit 117, and the third determination unit 119. Then, a predetermined number of classification codes having high credit values are selected and output as final classification codes. In addition, the output unit 121 determines the credit from the column of the credit table corresponding to the determination unit that has determined the classification code among the determination units (first determination unit 115, second determination unit 117, and third determination unit 119). Get the value. For example, when the classification code “26.1.1” is obtained by the first determination unit 115, the credit value “0.41” corresponding to the column “KNN” is acquired from the credit table. Further, when the same classification code is determined by a plurality of determination units, the maximum credit value is acquired from the corresponding credit values.

Next, a specific example of processing for calculating the score of the classification code by the function A will be described. Hereinafter, a case where five learning images Img _{1 to} Img ₅ are obtained as learning images similar to the divided image obtained by dividing the target image will be described as an example. FIG. 9 is a diagram illustrating an example of ranks, similarities, and associated classification codes (related classification codes) obtained for each learning image.

  As shown in FIG. 9, in this example, {1.1, 2.3, 4.2, 3.5, 5.1} is obtained as the classification code corresponding to each learning image. Therefore, the score calculation process in step S306 is executed for these five classification codes.

The classification code may be associated with a plurality of learning images. In the example of FIG. 9, for example, the classification code “1.1” is associated with two learning images of Img ₁ and Img ₃ . Therefore, for example, a set IM (1.1) of learning images associated with the classification code “1.1” is IM (1.1) = {Img ₁ , Img ₃ }. Similarly, IM (2.3) = {Img ₁ , Img ₂ }, IM (4.2) = {Img ₁ , Img ₄ }, IM (3.5) = {Img ₂ , Img ₄ , Img ₅ } , And IM (5.1) = {Img ₂ , Img ₃ , Img ₅ }.

The element a (factor_a (1.1)) of the classification code “1.1” is calculated as follows.
factor_a (1.1) = 0.57 × power (0.95, 0)
+ 0.48 × power (0.95, 2) = 1.0032

power (a, b) represents a function for calculating a power with a as a radix and b as an exponent. Here, rank (Img _i ) -1 is used as an index.

Similarly, the element a of the other classification code is calculated as follows.
factor_a (2.3) = 0.57 × power (0.95, 0)
+ 0.52 × power (0.95, 1) = 1.064
factor_a (4.2) = 0.57 × power (0.95, 0)
+ 0.46 × power (0.95, 3) ≈0.964
factor_a (3.5) = 0.52 × power (0.95, 1)
+ 0.46 × power (0.95, 3)
+ 0.32 × power (0.95, 4) ≈1.149
factor_a (5.1) = 0.52 × power (0.95, 1)
+ 0.48 × power (0.95, 2)
+ 0.32 × power (0.95, 4) ≈1.188

  In this example, the value of the element a for the classification code “5.1” is the largest. Therefore, the score for the classification code “5.1” is expected to be a larger value.

FIG. 10 is a diagram illustrating a calculation example of the mutual similarity between learning images. FIG. 10 shows an example of values calculated by the above equation (4). Using the values in FIG. 10, the element b of the classification code is calculated as follows according to the above equation (2).
factor_b (1.1) = 0.41
factor_b (2.3) = 0.43
factor_b (4.2) = 0.35
factor_b (3.5) = 0.40 + 0.38 + 0.15 = 0.93
factor_b (5.1) = 0.53 + 0.38 + 0.56 = 1.47

  In this example, the value of the element b for the classification code “5.1” is the largest. Therefore, the score for the classification code “5.1” is expected to be a larger value. The actually calculated score varies depending on the IDF, α, and β values of each classification code, as shown in the above equation (2). However, from the calculation results of the element a and the element b in the above example, it can be said that there is a high possibility that the score for the classification code “5.1” is maximized.

As shown in FIG. 9, the classification code “5.1” is associated with three images (Img ₂ , Img ₃ , Img ₅ ) among the searched five images. As shown in FIG. 10, the classification code “5.1” is included in both of the two images Img ₃ and Img ₅ that have the highest mutual similarity. As described above, according to the present embodiment, a score having a larger value is associated with a classification code that is associated with a larger number of images and has a large mutual similarity between a plurality of associated images. Can be calculated. Therefore, it is possible to more appropriately determine the annotation to be added to the image.

  Next, a hardware configuration of the annotation assigning apparatus according to the present embodiment will be described with reference to FIG. FIG. 11 is a diagram illustrating an example of a hardware configuration of the annotation assigning apparatus according to the present embodiment.

  The annotation assigning apparatus according to the present embodiment communicates with a control device such as a CPU 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM 53, and an external storage device such as an HDD and a CD drive device by connecting to a network. The communication I / F 54 that performs the above, a display device such as a display device, an input device such as a keyboard and a mouse, and a bus 61 that connects each unit are provided, and has a hardware configuration using a normal computer.

  The program executed by the annotation assigning apparatus according to the present embodiment is an installable or executable file, and is a computer such as a CD-ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk). Recorded on a readable recording medium and provided as a computer program product.

  Further, the program executed by the annotation assigning apparatus of the present embodiment may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. In addition, the program executed by the annotation assigning apparatus according to the present embodiment may be provided or distributed via a network such as the Internet.

  Further, the program of the present embodiment may be configured to be provided by being incorporated in advance in the ROM 52 or the like.

  The program executed by the annotation assigning apparatus according to the present embodiment includes the above-described units (image receiving unit, dividing unit, feature extracting unit, search unit, calculating unit, first determining unit, classification unit, second determining unit, pattern The module configuration includes a detection unit, a third determination unit, an output unit, and a communication unit). As actual hardware, the CPU (processor) 51 reads a program from the storage medium and executes the program, so that each unit is It is loaded on the main storage device, and the above-described units are generated on the main storage device.

DESCRIPTION OF SYMBOLS 100 Annotation giving apparatus 101 Image reception part 111 Division | segmentation part 112 Feature extraction part 113 Search part 114 Calculation part 115 1st determination part 116 Classification | category part 117 2nd determination part 118 Pattern detection part 119 3rd determination part 121 Output part 122 Communication part 151 Index storage unit 152 Corresponding storage unit 153 Rule storage unit 200a, 200b User terminal 300 Network

Jia-Yu Pan et al., "Cross-Modal Correlation Mining Using Graph Algorithms" (Knowledge Discovery and Data Mining: Challenges and Realities with Real-word Data, Chapter IV), Information Science Reference, June 2006.

Claims (7)

An index storage unit that stores an index in which a plurality of predetermined images and feature amounts of the images are associated;
A correspondence storage unit that associates and stores the image and a plurality of predetermined annotations;
A dividing unit that divides the target image to be annotated into a plurality of divided images;
A feature extraction unit that analyzes the divided image and extracts a feature amount of the divided image;
A search unit that searches the index storage unit for a predetermined number of images having a high degree of similarity between the corresponding feature value and the extracted feature value;
For each of the annotations associated with the searched image, the annotation is greater as the appearance frequency of the annotation in the correspondence storage unit is smaller, and is larger as the similarity of the corresponding image is larger. A calculation unit that calculates a score of the annotation that becomes a larger value as the feature quantities of the plurality of images obtained are similar to each other;
A first determination unit that determines the annotation to be given to the target image in preference to the annotation with a large score;
An annotation giving apparatus comprising: The calculation unit is similar in that the reverse appearance frequency that is greater as the appearance frequency is smaller, the first element value that is greater as the similarity of the corresponding image is larger, and the feature quantities of the plurality of associated images are similar to each other. Calculating a second element value that is larger, and calculating the score that is a linear sum of the product of the first element value and the reverse appearance frequency and the second element value;
The annotation giving apparatus according to claim 1. The calculation unit further calculates the score that is larger as the rank representing the order of the similarity of the corresponding images is larger,
The annotation giving apparatus according to claim 1. A rule storage unit for storing a rule for classifying an image into one of a plurality of classes associated with annotations according to the feature amount of the image;
A classification unit that applies the feature amount extracted by the feature extraction unit to the rule to classify the divided image into one of the classes;
A second determination unit that determines the annotation associated with the classified class as an annotation to be added to the target image;
The annotation giving apparatus according to claim 1. A pattern detection unit for detecting a predetermined image pattern from the target image;
A third determination unit that determines an annotation previously associated with the image pattern as an annotation to be added to the target image;
The annotation giving apparatus according to claim 1. A dividing step in which the dividing unit divides the target image to be annotated into a plurality of divided images;
A feature extraction step of analyzing the divided image and extracting a feature amount of the divided image;
A search unit stores an index in which a plurality of predetermined images and feature amounts of the images are associated with each other, and the similarity between the corresponding feature amount and the extracted feature amount is large in advance. A search step for searching a predetermined number of the images;
Appearance of the annotation in the correspondence storage unit for each of the annotations associated with the image searched in the correspondence storage unit in which the calculation unit associates and stores the image and a plurality of predetermined annotations The annotation score that is larger as the frequency is smaller and larger as the degree of similarity of the corresponding image is larger and becomes larger as the feature quantities of the plurality of associated images are similar to each other is calculated. A calculating step to
A first determination unit that determines the annotation to be given to the target image in preference to the annotation with a large score;
Annotation method characterized by comprising: An index storage unit that stores an index that associates a plurality of predetermined images with the feature amount of the image, and a correspondence storage unit that stores the image and a plurality of predetermined annotations in association with each other. A computer with
A dividing unit that divides the target image to be annotated into a plurality of divided images;
A feature extraction unit that analyzes the divided image and extracts a feature amount of the divided image;
A search unit that searches the index storage unit for a predetermined number of images having a high degree of similarity between the corresponding feature value and the extracted feature value;
For each of the annotations associated with the searched image, the annotation is greater as the appearance frequency of the annotation in the correspondence storage unit is smaller, and is larger as the similarity of the corresponding image is larger. A calculation unit that calculates a score of the annotation that becomes a larger value as the feature quantities of the plurality of images obtained are similar to each other;
A first determination unit that determines the annotation to be given to the target image in preference to the annotation with a large score;
Annotation program to make it function.

Images