JP2012064050A - Image retrieval device, method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image retrieval device capable of extracting a character string even when characters are arranged in various directions.SOLUTION: An image retrieval device comprises an image storage part for storing an image, a character recognition part for executing a character recognition processing to an image and extracting area information of each character, a metadata storage part for storing area information of each character as metadata of the image, an input part for inputting a character string for searching an image, a retrieval part for retrieving area information of each character stored in the metadata storage part based on each character of the character string, a polygonal line generation part for determining each of central coordinates corresponding to each character of the character string from the retrieved area information to generate a polygonal line using a plurality of central coordinates and a score calculating part for determining the linking score of the polygonal line.

Description

  Embodiments described herein relate generally to an image search apparatus.

  Searching for an image by recognizing characters included in a bitmap image taken by a user with a camera has been considered.

  Unlike documents with a well-defined format, bitmap images often do not work well for character recognition, and the accuracy of character string extraction is low. Therefore, there is a problem that reproducibility when a user searches for a bitmap image by a character string is lowered.

  For example, letters in hanging advertisements such as trains and buses and signs are not always lined up vertically and horizontally. Furthermore, since characters may be arranged in various directions even in one image, it is difficult to correctly analyze the layout.

  Also, in bitmap images, the change in brightness is large and the spacing between characters is often larger than the size of the characters, so the area that should originally be one is divided into a plurality of areas. It is difficult to extract a lump as a word.

Tomohiko Tsuji and two others, “Camera-based information acquisition system using real-time word recognition technology”, February 18, 2009, IEICE Technical Report

  Provided is an image search apparatus capable of extracting a character string even when characters are arranged in various directions.

  The image search apparatus according to the present embodiment includes an image storage unit that stores an image, a character recognition unit that performs character recognition processing on the image, extracts region information for each character, and stores the region information for each character as a meta data for the image. A metadata storage unit for storing data, an input unit for inputting a character string for searching for an image, and the region information for each character stored in the metadata storage unit based on each character of the character string A search unit for searching for, a center coordinate corresponding to each character of the character string from the searched area information, and a polygonal line generation unit for generating a polygonal line using the plurality of center coordinates, and a connection of the polygonal line A score calculation unit for obtaining a score.

The figure which shows the structure of the image search device 100 which concerns on this embodiment. The figure which shows the flow of a process in the image area division part 120 and the character recognition part 130. FIG. The figure which shows the processing result by the character recognition part. The figure which shows the example of the area | region information for every character stored in the metadata storage part 150. FIG. Explanatory drawing when the query containing ambiguity is input into the query input part 160. FIG. The figure which shows the flow of a process in the broken line production | generation part 180. FIG. The figure which shows the example of the broken line produced | generated by the broken line production | generation part 180. FIG. The figure which shows the example (1) of the connection scoring strategy in the score calculation part 190. FIG. The figure which shows the example (2) of the connection scoring strategy in the score calculation part 190. FIG. The figure which shows the example (3) of the connection scoring strategy in the score calculation part 190. FIG. The figure which shows the example (4) of the connection scoring strategy in the score calculation part 190. FIG. The figure which shows the example of the connection score with respect to the broken line of FIG. The figure which shows the example of a display of the search result in the search result presentation part 200. FIG.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a diagram illustrating a configuration of an image search apparatus 100 according to the present embodiment. The image search apparatus 100 includes an image input unit 110 that receives an input of an image from a user, an image region dividing unit 120 that divides the input image into regions of a piecewise size, and character recognition processing for each divided region. And a character recognition unit 130 that extracts region information for each character, an image storage unit 140 that stores input image data itself, and metadata that stores the extracted region information for each character as metadata of the image. A storage unit 150 is provided.

  Each time the user gives a query character string to the query input unit 160 to search for an image, the area information for each character corresponding to each character included in the query character string is read from the metadata storage unit 150. An area information search unit 170 to search, a center coordinate corresponding to each character of the query character string based on the searched area information, respectively, and a polygonal line generation unit 180 to generate a polygonal line using these center coordinates, The score calculation unit 190 for obtaining a connection score of each broken line, ranking of each broken line is performed based on the calculated connection score, and the image stored in the image storage unit 140 and the character area on the broken line are used as search results. A search result presenting unit 200 that presents them in a superimposed manner is provided.

  The image region dividing unit 120 divides the image received from the user by the image input unit 110 into regions having a piecewise size. This is because existing character recognition often does not work well in a bitmap image mainly targeted by the present embodiment, and the extraction accuracy as a character string is low. In other words, the recognition accuracy for each character is relatively high, but the vertical and horizontal connections are complicated, so that it cannot be recognized as a correct word. For example, it is difficult to correctly analyze the layout because characters included in suspended advertisements such as trains and buses and billboards are not always arranged vertically and horizontally.

  By the way, if an area of an image can be analyzed correctly with an existing layout analysis device, such as a magazine or exhibition panel, the image that has been correctly analyzed can be imaged with existing technology using the character string of the recognition result. You can search. Therefore, the image area dividing unit 120 may divide only an area that has not been correctly analyzed by the existing technology into an area having a piecewise size.

  FIG. 2 is a flowchart showing the flow of processing in the image area dividing unit 120 and the character recognition unit 130. In the image area dividing process and the character recognition process, first, the minimum area size S is determined (S200). The minimum area size S may be set in advance by the number of pixels regardless of the size of the image, or may be determined by a ratio to the vertical or horizontal length of the image to be processed. If the same minimum area size is used for all images, the amount of processing may become very large when the image size is large, and the image size is small when determining the ratio with respect to the vertical or horizontal length of the image. Then, it may be too coarse, so S may be determined by combining both. After determining S, the size of the target area for character recognition is gradually increased based on S, and the expanded target area is shifted little by little along the X and Y axes, and what are the characters included in the target area? It will be repeated to extract.

  Specifically, first, the number of sections in the X-axis direction CX = the horizontal length of the image / S and the number of sections in the Y-axis direction CY = the vertical length of the image / S are obtained (S210). In the example of the image shown in FIG. 3, CX = 21 and CY = 15. Subsequently, N = 1 is set. This N is a numerical value indicating how many times the minimum area size has been extended to increase the size of the target area for character recognition (S220).

  While N is incremented by 1, the extraction of what characters are included in the target area is repeated as follows until N × S becomes larger than the threshold (S230, S310).

  First, assuming that the position of the target area in the X-axis direction is I and the position J in the Y-axis direction, the size of the target area is N, so 0 ≦ I ≦ CX−N and 0 ≦ J ≦ CY−N. Therefore, I = 0 and J = 0 are set (S240, S260), and character recognition is performed while incrementing each by 1 (S320, S300) (S280). If a character is recognized, the recognition result is stored in the DB ( S290).

  For example, when N = 1, character recognition is performed for all the 21 × 15 smallest cells shown in FIG. In the case of FIG. 3, since there is no character corresponding to this size cell, it is not extracted. Subsequently, when N = 2, the character recognition process is executed for all 20 × 14 squares in which the smallest squares shown in FIG. In the case of FIG. 3, R3, R4, and R5 are characters corresponding to this size cell, so “T”, “N”, and “K” are extracted. Further, when N = 3, the character recognition process is executed for all 19 × 13 squares in which the smallest squares shown in FIG. In the case of FIG. 3, “R”, “R2”, and “R6” are characters corresponding to this size cell, so “K”, “M”, and “L” are extracted.

  FIG. 4 shows an example in which the metadata storage unit 150 stores area information for each character extracted from the image shown in FIG. This figure shows that six regions having region IDs R1 to R6 are extracted from the image having the image ID P1 stored in the image storage unit 140. For example, the center coordinate of the region R1 is (X1, Y1), the character is “K”, the size is 32 pixels, the font is Gothic, and the color is black. Similarly, the center coordinates of the region R3 are (X3, Y3), indicating that the character is “na”, the size is 20 pixels, the font is Gothic, and the color is red. As shown in FIG. 4, in this embodiment, the area information is stored in the DB so that the character recognition result is unique for the sake of simplicity. However, if the character recognition result is ambiguous, However, the characters may not be unique and may be stored as separate characters with certainty. In the case of FIG. 4, none of the extracted regions overlap, but in the case of kanji, the extracted regions may overlap because there is a possibility that the constituent element may be recognized as another character. Alternatively, in addition to the size, font, and color, if there is information that can be extracted using the character recognition device, it may be stored in the metadata storage unit 150 as area information for each character.

  Next, a process for searching for an image using region information for each character as metadata of the image extracted and stored in this manner will be described. The query input unit 160 receives a query character string for image search from the user and divides it for each character. For example, when a query character string “camera” is given, it is divided into “f”, “me”, and “la”. In this embodiment, for the sake of simplicity, it is assumed that a query character string in which each character is fixed is received as an input, but a character string including ambiguity such as a result obtained by subjecting an image to character recognition processing may be input. .

  FIG. 5 is a diagram for explaining a case where a query including ambiguity is input to the query input unit 160. For example, when an image as shown in FIG. 5 is subjected to character recognition processing, the first and third characters are determined to be “K” and “La”, but whether the second character is “Me” or “N”. It is ambiguous. However, in this case as well, multiple query strings obtained by multiplying the confidence score of each character by the concatenation score, such as 0.7 for the concatenation score of the query string “Camera” and 0.3 for the concatenation score of the query string “Canara” Therefore, in the following, description of the character string including ambiguity is omitted.

  The divided query character string is sent to the area information search unit 170. The area information search unit 170 searches the metadata of the image stored in the metadata storage unit 150, that is, the area information for each character of the image, and determines an image ID including all characters of the query character string. List all. At this time, in the case of a query character string in which the same character appears multiple times, such as “Albania”, if the image does not include two or more “a”, the final search result is not suitable, but will be described later. Since it is possible to apply processing that does not match the search result by connecting scoring of broken lines, at this point, if the search is performed only based on whether all the characters are included, ignoring the number included in the image Good.

  Next, a process in which the broken line generation unit 180 generates a broken line from an image including all the characters of the query character string obtained as described above will be described. FIG. 6 is a flowchart showing the flow of processing in the polygonal line generation unit 180. This figure is difficult to understand because it includes recursive processing. Therefore, an outline will be described below using pseudo code. Of the following two functions, GetSs is the main function of the polygonal line generation process, and GetSsSub is a subfunction responsible for recursive processing.

GetSs (query string Q, image group Ps containing all characters of query string) {
Polyline set Ss = {} // S600
foreach (Image P: Ps) {// S610
I = 0
S = []
GetSsSub (P, Q, I, S, Ss) // S620
}
return Ss // S680
}

GetSsSub (Image P, Query string Q, Index I, Line buffer S, Line set Ss) {
if (I is equal to the length of the query string Q) {// S630
Add polyline buffer S to polyline set Ss // S660
return
}
foreach (region R: region information of the I-th character Q [I] of the query string Q included in the image P) {// S640, S650
GetSsSub (P, Q, I + 1, S + R center coordinates, Ss) // S670
}
}

The main GetSs function initializes a set of broken lines Ss to be generated (S600), and then calls the GetSsSub function for all images (S610). When calling the GetSsSub function, the polygonal line buffer S is initialized and I = 0 is set (S620). The GetSsSub function increments I by 1 (S670), while I is smaller than the length of the query string (S630), enumerates all areas corresponding to each character of the query string (S640), Generation is performed (S650). Therefore, a comprehensive broken line is generated as a whole. Incidentally, in the flowchart, the process of incrementing I by 1 (S670) corresponds to calling GetSsSub and reducing recursion by one step, and the process of decrementing I by 1 (S710) corresponds to exiting GetSsSub and increasing recursion by one step. Since the polygonal line buffer S only stores data being processed, it is necessary to delete the last center coordinate from S when the recursion is increased by one stage (S680). Therefore, the polygonal line generated for the query character string needs to be added to the polygonal line set Ss (S660).

  FIG. 7 is a diagram illustrating an example of a polygonal line generated by the polygonal line generation unit 180. Since the image P1 includes two “f” s, 2 × 1 × 1 = 2 two line sets are generated for the query character string “camera”. Since the image P2 includes both “f” and “me”, 2 × 2 × 1 = 4 broken line sets are generated. Since only one image P3 is included in the image P3, only one broken line is generated. Here, “×” means repeated calculation of the GetSsSub function.

  Next, the process in which the score calculation part 190 calculates a connection score with respect to the produced | generated broken line set is demonstrated. There are various methods for calculating the connection score. The simplest connection scoring strategy is to give a connection score of 1 or 0 depending on whether or not all the line segments constituting the polygonal line set exist on a preset figure. Even if it is a bitmap image, if it is a character string that can be read by humans, it is considered that the flow of the line of sight does not change abruptly.

  However, the connection score of a broken line that is slightly deviated only by whether or not it is on a straight line or an arc will be 0. Therefore, as shown in the example (1) of the connection scoring strategy in FIG. 8, two parallel lines and concentric circles including all line segments constituting the polygonal line are obtained, and the interval D is determined from a true straight line or arc. You may make it calculate a connection score considering it as an error.

  Conversely, a strategy of lowering the connection score for a broken line with an unnatural line of sight is conceivable. For example, line segments intersecting or looping means that the line of sight is moving back and forth between characters, and is not a natural line of sight flow. Therefore, as shown in the example (2) of the connection scoring strategy in FIG. 9, a broken line in which line segments intersect or loop may be given −∞ as a connection score and excluded from the search result.

  Alternatively, a connection score function is set in advance according to the direction of the line segment, a link score is obtained for each line segment constituting the polygonal line, and the total or maximum value of the link scores of all the line segments is used. A concatenated scoring strategy is also possible. The example (3) of the connection scoring strategy shown in FIG. 10 is an example of a function that defines the correspondence between the direction (angle) of the line segment and the connection score. From the viewpoint of the flow of human eyes, the character string typically goes from left to right (0 degrees) or from top to bottom (270 degrees), so it takes a large value at these angles, and right to left (180 degrees). ) Since it is almost impossible to go from bottom to top (90 degrees), it is a function that takes a small value at these angles.

  In addition, the function of the connection score is set in advance according to the angle difference with the previous line segment, the difference in length, the ratio of the length, etc., and the connection score of the line segments constituting the broken line is obtained, A concatenation scoring strategy that uses the sum or maximum of the concatenation scores for all line segments is also conceivable. The example (4) of the connection scoring strategy shown in FIG. 11 is an example of a function that defines the correspondence between the angle difference, the length difference, the length ratio, and the connection score. From the viewpoint of the line of sight, it is considered that the character strings are typically arranged at equal intervals. Therefore, the difference in angle and the difference in length are closer to 0, and the length ratio is closer to 1. It is a function that takes a value. However, a projection distortion often occurs in a bitmap image, and in such a case, the length is not equal, but decreases at a constant rate. Therefore, the length ratio may be a function that takes a large value with a value smaller than 1.

  In the above description, it has been assumed that there is no other character area on or around the line segment constituting the polygonal line, but in general, there may be other character areas. However, the fact that the line segment crosses other character areas in the surrounding area means that the line of sight crosses other character areas, and is not a natural flow of line of sight. Therefore, in such a case, the score may be lowered.

  Although several linked scoring strategies have been described, the score calculation unit 190 in this embodiment may select any one of these to calculate a linked score, or may be calculated by a plurality of scoring strategies. The total or maximum value of the connected scores may be used. In addition, regarding the area information corresponding to the vertices of the line segments constituting the broken line, the connection score may be adjusted according to the similarity of the character attributes (size, font, color, etc.).

  FIG. 12 is a diagram illustrating an example of the connection score for the example of the broken line illustrated in FIG. 7. Here, the linked scoring strategy of the score calculation unit 190 may be changed each time a query character string is given, but is usually fixed. Therefore, the connection score for the same broken line is fixed. In other words, if the connection score calculated for the broken line is saved once and the saved query value is read when the same query string is given after the next time, the line will be generated for the next and subsequent searches. Processing can be omitted and the processing speed can be increased.

  Finally, the search result presentation unit 200 ranks broken lines based on the connection score calculated by the score calculation unit 190, acquires images corresponding to higher ranks from the image storage unit 140, and retrieves characters corresponding to the broken lines. The region is displayed superimposed on the image.

  FIG. 13 is a diagram illustrating a display example of search results in the search result presentation unit 200. Various methods can be considered as the superimposed display method. In the present embodiment, an area in which the width of the character size is given to the broken line is surrounded by a frame line and displayed. At this time, in the example of FIG. 12, the highest ranking polygonal lines S2-1 and S2-3 are related to one image P2. However, when multiple polygonal lines are displayed in a superimposed manner, it is difficult to see a single image. Only broken lines with high scores are displayed. By explicitly superimposing the corresponding area on the image, it is possible to check the order in which the query character string appears, so that the search results can be presented easily for the user. Become.

  Note that the present embodiment is not limited to the above-described embodiment as it is, and can be embodied by modifying the components without departing from the scope of the invention in the implementation stage. Moreover, various inventions can be formed by appropriately combining a plurality of constituent elements disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, constituent elements over different embodiments may be appropriately combined.

DESCRIPTION OF SYMBOLS 100 Image search device 110 Image input part 120 Image area division part 130 Character recognition part 140 Image storage part 150 Metadata storage part 160 Query input part 170 Area information search part 180 Polygon line generation part 190 Score calculation part 200 Search result presentation part

Claims (4)

An image storage unit for storing images;
A character recognition unit that performs character recognition processing on the image and extracts area information for each character;
A metadata storage unit that stores the area information for each character as metadata of the image;
An input unit for inputting a character string for searching for an image;
A search unit that searches the region information for each character stored in the metadata storage unit based on each character of the character string;
A center line corresponding to each character of the character string is obtained from the searched area information, and a polygonal line generation unit that generates a polygonal line using the plurality of center coordinates,
An image search apparatus comprising: a score calculation unit that calculates a connection score of the broken lines.   The image search apparatus according to claim 1, further comprising a search result presentation unit that ranks the polygonal lines based on the connection score, and superimposes and presents character regions on the polygonal lines on the image. Performing character recognition processing on the image stored in the image storage unit and extracting region information for each character;
Storing the area information for each character in the metadata storage unit as metadata of the image;
Entering a string to search for images;
Searching the region information for each character stored in the metadata storage unit based on each character of the character string;
Obtaining center coordinates corresponding to each character of the character string from the searched area information, and generating a polygonal line using a plurality of the center coordinates;
An image search method comprising: calculating a connection score of the broken line. Computer
Means for storing images;
Means for character recognition processing the image and extracting region information for each character;
Metadata storage means for storing the region information for each character as metadata of the image;
An input unit for inputting a character string for searching for an image;
Means for retrieving the region information for each character stored in the metadata storage means based on each character of the character string;
A center line corresponding to each character of the character string is obtained from the searched area information, and a polygonal line generation unit that generates a polygonal line using the plurality of center coordinates,
Means for determining a connection score of the broken lines;
Image search program to be executed as.

Images