JP2018097707A - Information processor, character recognition method, computer program, and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor for performing character recognition of a character string included in a plurality of photographed images with partially deteriorated quality.SOLUTION: An information processor 100 generates a first recognition graph from a first character pixel block extracted from a first image obtained by photographing a character string and a first character element classified from the first character pixel block. The information processor 100 generates a second recognition graph from a second character pixel block extracted from a second image obtained by photographing a character string and a second character element classified from the second character pixel block. On the basis of determination on whether or not the first character pixel block and the second character pixel block correspond to the same character string, the information processor 100 generates an integrated recognition graph obtained by integrating the first recognition graph and the second recognition graph and recognizes the character string on the basis of the integrated recognition graph.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an information processing apparatus having a character recognition function for recognizing characters from a plurality of images obtained from a moving image or the like.

  An information processing device with a camera function such as a smartphone or a tablet terminal can photograph a subject such as a paper surface or a signboard and recognize a character string printed on the subject by character string recognition processing from the photographed image. . In photographing by an information processing apparatus, a reduction in quality of a photographed image due to camera shake or a shift in autofocus affects the accuracy of character recognition. In Patent Literature 1, an image processing apparatus that captures a moving image of a subject on which characters are printed, performs character recognition on a plurality of captured images of the same character string, and integrates the results to obtain a character string recognition result. Is disclosed. This image processing apparatus employs the largest number of recognition results for each target character, thereby excluding recognition results from defective captured images and realizing highly accurate character recognition using the effect of majority vote. Patent Document 2 discloses a character recognition method using context information of recognition target characters, such as N-gram, which is the frequency of occurrence of concatenated characters, and a dictionary that defines character string patterns to be recognized. This method performs highly accurate character recognition by using context information when selecting a character string recognition result from a plurality of classification candidates generated as a result of classifying an image of a recognition target area as a character. In this method, a recognition result as a character string can also be obtained by integrating a set of recognition target areas extracted from an input image by a plurality of extraction methods and using context information.

JP 2010-218061 A US Publication No. 2015/0356365

  When extracting an area of a character to be recognized from a low-quality captured image, the information processing apparatus may extract a plurality of characters as one area or may divide and extract one character into a plurality of areas. . In this case, in the image processing apparatus of Patent Document 1, even if an attempt is made to collect recognition results for each character from a plurality of captured images obtained from a moving image, the extraction of the character region is inappropriate. The association is difficult. For this reason, it is not possible to obtain a number of recognition results sufficient to obtain the effect of majority voting, and character recognition with high accuracy cannot be performed. Alternatively, it is necessary for the user to perform shooting for a long time until a high-quality shot image is obtained. The character recognition method using the context information in Patent Document 2 is effective even when it is difficult to extract and extract a character area from a captured image in units of one character. In this case, one captured image is a processing target. It is. For this reason, when there is a set of character areas including cut-out errors from a plurality of captured images constituting a moving image, it is difficult to obtain a character string recognition result as a character string using context information from all of them.

  The present invention has been made in view of the above problems, and mainly provides an information processing apparatus capable of performing character recognition of a character string included in a plurality of captured images partially having low quality with high accuracy. Objective.

  The information processing apparatus according to the present invention includes an acquisition unit that acquires a captured image obtained by capturing a character string printed on a subject, an extraction unit that extracts a character pixel block from a character string portion of the captured image, and the character pixel block. Classifying means for classifying character elements based on the above, generating means for generating a recognition graph from the character pixel block and the classification result of the character elements, recognition means for recognizing the character string based on the recognition graph, And the generation means includes a first character pixel block extracted from a first image of the plurality of photographed images and a first character element classified from the first character pixel block. From the second character pixel block extracted from the second image different from the first image among the plurality of captured images, and the second character pixel block, while generating a first recognition graph The second character element to be classified, A second recognition graph is generated, and the recognizing unit determines whether the first character pixel block and the second character pixel block correspond to the same character string. An integrated recognition graph in which the recognition graph is integrated with the second recognition graph is generated, and the character string is recognized based on the integrated recognition graph.

  According to the present invention, by recognizing a character string based on an integrated recognition graph, it is possible to perform highly accurate character recognition of character strings included in a plurality of captured images that are partially low in quality.

(A), (b) is a figure which shows the external appearance example of information processing apparatus. The functional block diagram of information processing apparatus. The flowchart showing the flow of a character string recognition process. The flowchart showing the flow of the process of S308. Explanatory drawing of the extraction process of a character pixel block, and the production | generation process of a character element. FIG. The illustration figure of the recognition graph to which context information is applied. The flowchart showing the flow of the update process of a recognition graph. Explanatory drawing of the specific example which performs a character string recognition process. (A)-(f) is explanatory drawing of the specific example of a recognition graph. Explanatory drawing of the modification of a character string recognition process.

  Hereinafter, embodiments will be described in detail with reference to the drawings.

(Configuration of information processing device)
FIG. 1 is a diagram illustrating an appearance example of the information processing apparatus according to the present embodiment. The information processing apparatus 100 has a plate-like appearance, and includes an image pickup device 101 on one surface thereof (FIG. 1A), and a display device 102 and operation buttons 103 on the other surface opposite to the one surface. (FIG. 1B). The information processing apparatus 100 is a computer that includes a communication unit 104 and a control unit 105 therein. The information processing apparatus 100 is realized by, for example, a smartphone, a tablet terminal, or a high function camera. The control unit 105 includes a CPU (Central Processing Unit), a memory, and the like. The control unit 105 controls the operation of the information processing apparatus 100.

  The imaging device 101 captures a real world scene and acquires image data representing a captured image. The image pickup device 101 includes, for example, a lens and an image pickup device. The display device 102 is an output interface that allows the user to visually recognize the image data acquired by the image pickup device 101. The display device 102 is, for example, a liquid crystal display. The operation button 103 is an input interface for the user to instruct the information processing apparatus 100 to start shooting, end shooting, and the like. The operation button 103 is a mechanical or pressure-sensitive physical button. When the display device 102 is configured by a touch panel, the display device 102 may have the function of the operation button 103. The communication unit 104 is a communication interface that performs wireless communication. The communication unit 104 controls communication between the information processing apparatus 100 and an apparatus such as an external server via a communication network.

  The image pickup device 101 can shoot a moving image by continuously shooting a subject a plurality of times. The captured moving image is immediately displayed on the display 102. Therefore, the user of the information processing apparatus 100 can view the subject that the information processing apparatus 100 is photographing at any time. The information processing apparatus 100 has a function of performing character recognition of a character string included in a captured image. The information processing apparatus 100 may display the character string recognition result acquired as a result of character recognition on the display unit 102 in association with the captured image. The information processing apparatus 100 may transmit the acquired character string recognition result to an external apparatus via the communication unit 104.

  FIG. 2 is a functional block diagram of the information processing apparatus 100. In the information processing apparatus 100, for example, a CPU provided in the control unit 105 realizes each function by executing a predetermined computer program. The information processing apparatus 100 functions as a captured image acquisition unit 201, a captured image processing unit 202, a display control unit 203, a character string detection unit 204, a character string recognition unit 205, and a character string information storage unit 206.

  The captured image acquisition unit 201 acquires image data of a captured image from the imaging device 101 at a predetermined capturing interval. The shooting interval is, for example, 30 times or 60 times per second. The image data is sent to the captured image processing unit 202 and the display control unit 203.

The captured image processing unit 202 performs image processing on the image data of the captured image acquired from the captured image acquisition unit 201 at a predetermined capturing interval. The captured image processing unit 202 of the present embodiment performs at least the following three processes.
(Process 1) The image data is corrected to a format suitable for processing by the display control unit 203, the character string detection unit 204, and the character string recognition unit 205.
(Process 2) An image suitable for processing by the display control unit 203, the character string detection unit 204, and the character string recognition unit 205 and an inappropriate image are selected from the image data.
(Process 3) If there is a movement of the same subject in the selected plurality of images, the movement amount is calculated.

  The display control unit 203 generates a display image to be displayed on the display device 102. The display control unit 203 causes the display 102 to display the generated display image. The display image is a captured image represented by image data acquired from the captured image acquisition unit 201, for example. The display control unit 203 displays the display image on the display unit 102 at the same time interval as the shooting interval, thereby allowing the user to check the shooting content and shooting state. Note that the display image may be image-processed by the captured image processing unit 202 so that the user can easily recognize it. The display image may have a character string recognition result added and superimposed.

  The character string detection unit 204 detects a region of a character string image that is a target of character recognition in the captured image based on the image data that has been subjected to image processing by the captured image processing unit 202. The character string recognition unit 205 performs a known character string recognition process within the region of the character string image detected by the character string detection unit 204, and generates a character string recognition result including a sequence of character codes. The character string recognition unit 205 performs character string recognition processing in the region of the character string image of the photographed image based on the image data image-processed by the photographed image processing unit 202.

  The character string information storage unit 206 stores the area of the character string image detected by the character string detection unit 204 as character string information. The area of the character string image is represented by a rectangle, for example, and the character string information storage unit 206 stores the coordinates of the rectangle as the area of the character string image. When the character string detection unit 204 detects a plurality of character string image regions, the character string information storage unit 206 stores the coordinates of each region. The character string information storage unit 206 stores the character string recognition result generated by the character string recognition unit 205 for the character string in addition to the character string information. The content stored in the character string information storage unit 206 may be transmitted from the communication unit 104 to an external device as necessary.

(Character string recognition processing)
FIG. 3 is a flowchart showing the flow of character string recognition processing using the information processing apparatus 100 having such a configuration. Hereinafter, the flowchart is realized by the CPU executing the control program. When the information processing apparatus 100 is instructed to start the character string recognition process, the information processing apparatus 100 activates the image pickup device 101 and starts shooting at a predetermined shooting interval. As a result, the image pickup device 101 captures a moving image composed of continuous captured images.

  The captured image acquisition unit 201 continuously acquires image data representing captured images captured by the image pickup device 101 at a predetermined capturing interval (S301). That is, the character string recognition process of FIG. 3 is continuously performed on captured images that are continuous at a predetermined capturing interval. Hereinafter, image data representing a photographed image photographed at time t is represented as an image t. In the character string recognition process, character recognition is performed on the image t or an image obtained by processing the image t by the captured image processing unit 202.

  The captured image processing unit 202 performs the above (Process 1) on the image t. Specifically, the captured image processing unit 202 performs known filter processing such as blur correction and image processing such as resolution conversion processing on the image t (S302). Through these processes, the image t becomes image data suitable for processes such as display on the display 102, character string detection, and character string recognition.

The captured image processing unit 202 performs the above (Process 2) on the image t after image processing. Thereby, the captured image processing unit 202 determines whether or not the image quality of the image t after the image processing is appropriate for the processing performed by the character string detection unit 204 and the character string recognition unit 205 (S303). This determination is performed using an evaluation value such as the total amount or density of edges included in the image t, for example. If the evaluation value is lower than the predetermined threshold, the image t is determined to be a captured image with low quality in which the blur has occurred and an inappropriate image for which the accuracy of character string detection cannot be expected.
In addition, the sum or average of the pixel differences from the immediately preceding captured image (images t-1, t-2, etc.) may be used as the evaluation value. When such an evaluation value is lower than the threshold value, it is determined that the image t is a captured image with little blur and suitable for character recognition. Further, when the attitude of the information processing apparatus 100 can be detected by an acceleration sensor or the like, the detection result may be used as an evaluation value. In this case, it is determined that the image t when the information processing apparatus 100 is stationary is appropriate for character recognition. Note that either of the processing of S302 and S303 by the captured image processing unit 202 may be performed first.

  When the image quality of the image t is appropriate for the processing performed by the character string detection unit 204 and the character string recognition unit 205 (S303: Y), the captured image processing unit 202 performs the above (processing 3) on the image t. Do. When the same character string to be character-recognized moves between two images of the image t and an image t-x (x is an integer of 1 or more) captured before that, the captured image processing unit 202 The movement amount is calculated. Specifically, the character string subject to character recognition is a character string stored in the character string information storage unit 206. This character string is a character string detected in the past by a character string recognition process described later with respect to a captured image before the image t. The captured image processing unit 202 updates the character string information representing the region of the character string image based on the calculated movement amount (S304). The image t-x used for calculating the movement amount is always the previous photographed image, for example, x = 1. In that case, in the process of S304, the coordinates of the character string are updated each time according to the movement amount from the image t-1 to the image t, and thereafter, these can be handled as the position of the character string in the image t. . Alternatively, the movement amount may be calculated between an image t ′ subjected to a character string detection process described later and the image t. At this time, the character string coordinates at the time of detection from the image t ′ are held in the character string information as the original character string coordinates, and the coordinates updated by the movement amount are set as the character string coordinates in the image t. Furthermore, the character string coordinates may be updated by calculating the amount of movement for the detection images that differ for each character string.

  For the calculation of the movement amount, a known method based on optical flow detection can be used. For example, the movement amount is calculated using local feature point detection such as a well-known SIFT (Scale-Invariant Feature Transform) or ORB (Oriented FAST and Rotated BRIEF). A pair of feature points that are common to the two images is acquired by local feature point detection, and the distance between the pairs that match or are close to the target subject in the feature point pairs is set as the amount of movement of the subject. Alternatively, corresponding point matching is performed between feature points obtained from the entire two images, a homography transformation matrix between the images is estimated, and the movement amount is calculated by performing homography transformation on the coordinates of the target character string.

  The captured image processing unit 202 determines whether or not the image t is an execution target of the character string detection process (S305). Whether or not the image t is an execution target of the character string detection process is detected by, for example, the image n, the image 2n, and the image 3n when the character string detection unit 204 captures n times within the time of one detection process. It is determined that the process is to be executed, and other captured images are determined not to be the detection process. Alternatively, the processing state of the character string detection unit 204 is monitored in real time, and only when the previously acquired image is not being processed, the current image t is determined to be a detection process execution target. Further, the evaluation value obtained by the process of S303 may be added to the determination condition for the execution target of the detection process.

  When the image t is an execution target of the character string detection process (S305: Y), the character string detection unit 204 detects a character string existing in the image t (S306). The detection result is stored in the character string information storage unit 206 as character string coordinate information. The character string coordinate information is, for example, a list of coordinates of four corners of a rectangle that includes a character string region. The character string coordinate information may be associated with information that is detected from the image t. When there is no free space in the character string information storage unit 206 and character string information detected from a past captured image is stored, only the character string information detected from the image t is stored after erasing the past information. You may do it. Alternatively, the correspondence between the character string information detected from the image t and the past character string information is checked, and if there is a match, the coordinate information of the character string information is updated. You may make it preserve | save.

The character string detector 204 detects a character string in the image t by a known technique. For example, the character string detection unit 204 binarizes the image t, connects the black pixels to be connected by a labeling process, and extracts a connected pixel block (Connected Component). A “connected pixel block” refers to a region of a pixel group in which pixels are connected. The character string detection unit 204 combines a character pixel block that is estimated to be a character from the size of the circumscribed rectangle among the extracted connected pixel blocks with other character pixel blocks existing in the vicinity, thereby To extract. Each extracted character string is given position information for character string recognition processing to be described later.
Note that the character string detection unit 204 may extract a connected pixel block by connecting pixels having similar luminance and color, for example, instead of binarizing the image t. Alternatively, the character string detection unit 204 may perform edge extraction and extract a connected pixel block from connected edge pixels. In order to speed up the character string detection process, the character string detection unit 204 may detect a character string by extracting a connected pixel block from the image t subjected to the reduction process.

  After the character string is detected or when the image t is not an execution target of the character string detection process (S305: N), the character string recognition unit 205 determines whether the image t is an execution target of the character string recognition process. (S307). Whether or not the character string recognition process is to be executed is determined based on one recognition processing time of the character string recognition unit 205 and the number of captured images acquired during that time, as in the process of S305. The character string recognition unit 205 may add an evaluation value as to whether or not the image t used in the process of S303 is appropriate for the character string recognition process to the determination condition as to whether or not to execute. In step S <b> 305, the character string recognition unit 205 performs n times (n> 1) of character string recognition processing per character string detection processing in synchronization with the number of times the character string detection processing is actually performed. In conjunction with the determination in this process, it may be determined whether or not the character string recognition process is an object. When there is no information on the detected character string in the character string information storage unit 206, the character string recognition unit 205 determines that the image t is not an execution target of the character string recognition process.

  When the image t is an execution target of the character string recognition process (S307: Y), the character string recognition unit 205 executes the character string recognition process for the image t (S308). The processing target is character string information stored in the character string information storage unit 206, and the processing result is added to the corresponding character string information. Details of the character string recognition process will be described later.

After the character string recognition, the display on the display 102 is updated (S309). When the image quality of the image t is not appropriate for the processing performed by the character string detection unit 204 and the character string recognition unit 205 (S303: Y), or when the image t is not an execution target of the character string recognition processing (S307: N) also updates the display on the display 102 (S309). The display control unit 203 updates the display on the display unit 102 with the captured image being processed.
At this time, the display control unit 203 may add the character string information stored in the character string information storage unit 206 to the display. For example, the display control unit 203 may display the frame so that the frame is superimposed on the shadow image so that the user can recognize the detected character string position based on the detection result of each character string obtained in the process of S306. Alternatively, the display control unit 203 may display the character string recognition result added in the process of S308 with a frame or at a specific position on the display screen. The information processing apparatus 100 may use the contents of the character string information storage unit 206 for purposes other than display at the timing of display update. For example, the information processing apparatus 100 may be used by another application that is operating. Alternatively, the information processing apparatus 100 may provide a character string recognition result to a program operating on another apparatus via the communication unit 104.

  FIG. 4 is a flowchart showing the flow of character string recognition processing in S308. Here, a character string recognition process performed by the character string recognition unit 205 using the image t for one character string i stored in the character string information storage unit 206 will be described. Actually, the character string recognition unit 205 repeatedly executes the character string recognition process for all the character strings stored in the character string information storage unit 206.

  The character string recognition unit 205 acquires a character string image to be recognized as the character string i from the image t (S401). For example, the character string recognition unit 205 derives a rectangular range circumscribing the four points from the coordinate information of the four corners of the rectangle that includes the character string, and acquires the coordinates of the range. Note that the coordinate information of the character string i is updated to the coordinates in the image t by the process of S304.

  The character string recognition unit 205 extracts a character pixel block constituting at least a part of the character from the acquired character string image (S402). The character string recognizing unit 205 can extract a character pixel block by the same method as the process of S306. The character string recognition unit 205 generates a character element from the set of extracted character pixel blocks (S403). A “character element” is a group of one or a plurality of adjacent character pixel blocks.

  FIG. 5 is an explanatory diagram of character pixel block extraction processing and character element generation processing. FIG. 5 illustrates a state in which character pixel blocks 501 to 506 are extracted from the character string image 500 and character elements 511 to 521 are generated from the character pixel blocks 501 to 506. The character string image 500 is printed on a subject such as paper, and is acquired from a photographed image obtained by photographing the subject. In the example of FIG. 5, the character string image 500 is acquired from a photographed image of three characters “cultural” printed on the subject. Based on the character string image 500, the character string recognition unit 205 extracts six character pixel blocks 501 to 506 as black pixel blocks to be connected. The character string recognition unit 205 generates character elements 511 to 516 as character elements corresponding to the character pixel blocks 501 to 506, respectively. In each character pixel block 501 to 506, the character string recognizing unit 205 groups one or more character pixel blocks adjacent to each other with a distance equal to or less than the threshold value in the right direction to generate character elements 511 to 521. The widths of the character elements 511 to 521 are limited so as not to become too large. For example, the character string recognition unit 205 does not generate a character element whose grouped width exceeds 1.2 times the character string height, for example. As the “character string height”, for example, a value estimated from the maximum height of the character pixel block is used.

  In the example of FIG. 5, a character element 517 in which the character pixel block 502 and the character pixel block 503 are grouped is generated. A character element 518 in which the character pixel block 503 and the character pixel block 504 are grouped is generated. A character element 519 in which the character pixel block 504 and the character pixel block 505 are grouped is generated. A character element 520 in which the character pixel block 504, the character pixel block 505, and the character pixel block 506 are grouped is generated. A character element 521 in which the character pixel block 505 and the character pixel block 506 are grouped is generated.

  The character string recognition unit 205 may exclude a character element that is unlikely to correspond to one character among the character elements 511 to 521 at this stage. For example, when another character pixel block is completely or mostly included in a predetermined character pixel block, it is unlikely that each character pixel block corresponds to another character. In addition, when it is known that the character string image is horizontally written Japanese, there is a low possibility that the character pixel block overlapping in the vertical direction becomes a separate character. The character string recognizing unit 205 may generate a character element after first integrating two or more character pixel blocks having such a relationship as one character pixel block.

  The character string recognition unit 205 classifies the generated character elements after generating the character elements (S404). The character string recognition unit 205 performs a classification process on each of the generated character elements by matching a pixel pattern within the range of the character elements in the character string image with a character pattern dictionary learned in advance. The character string recognition unit 205 stores the classification result in association with the character element as a character code corresponding to the category of the character pattern dictionary and a numerical value representing the reliability of the classification. The classification result to be stored may be one character code for one character element, or may be ranked by a plurality of reliability levels and may be a plurality of character codes.

  The character string recognizing unit 205 performs a classification process by a known character recognition prescription, for example. For example, the character string recognition unit 205 normalizes the size of the pixel pattern to be classified, divides the pixel pattern into blocks (7 × 7), and totals the amount of black pixel edges in each block for each direction. To generate a feature vector. When totaling in the four directions of vertical, horizontal, and diagonal, the edge amount of (4 × 7 × 7) is the feature vector to be classified. Similarly, the character string recognition unit 205 stores a character pattern dictionary including feature vectors obtained from the learning data of each character. The character string recognizing unit 205 calculates the inter-vector distance between the target feature vector and the feature vector of each character in the dictionary at the time of collation, and sets one character having the closest distance as the classification result. Alternatively, the character string recognizing unit 205 sets a classification result having a possibility of classification into a plurality of characters whose distance is equal to or less than a threshold value. The classification result is given a numerical reliability based on the same distance. Note that the character string recognition unit 205 may classify the character elements by another feature vector extraction method or a collation method. For example, the character string recognition unit 205 may perform the classification process using a pattern learning device using a neural network or the like.

  The character string recognition unit 205 determines whether or not the current process is the first character string recognition process for the character string i stored in the character string information storage unit 206 (S405). The character string recognizing unit 205 makes this determination based on, for example, whether or not information of a recognition graph described later is added to the target character string information. If not added, the character string recognition unit 205 determines that this is the first character string recognition process.

  In the case of the first character string recognition process (S405: Y), the character string recognition unit 205 creates a new recognition graph for the character string i (S406). The recognition graph is created as a directed graph in which character pixel blocks extracted from the character string image are sections of the graph, and classification results for the character elements are weighted edges. The recognition graph is added to the information of the character string i in the character string information storage unit 206 and stored.

  The character string recognition unit 205 creates a recognition graph as follows, for example. First, the character string recognition unit 205 sets a set {C} = {C1,. . . , Cn}. Here, the character string i is substantially horizontal, and C1, C2,... Are arranged so that the left end coordinates of the character pixel block are in ascending order. The character string recognition unit 205 creates n nodes N1 to Nn of the graph and a terminal node Ne corresponding to a virtual terminal character pixel cluster so as to correspond to the character pixel blocks C1 to Cn.

  Next, the character string recognizing unit 205 groups the character pixel blocks included in the set {C} to set a character element set {G} = {G1,. . . , Gu}. For the character element Gt in the set {G}, the character pixel block that is included in Gt and has the smallest left end coordinate is Ci, and is located to the right of Ci, not included in the character element Gt, and has the smallest left end coordinate. Let Cj be the character pixel block. In this case, the character string recognition unit 205 generates a weighted side Eij from the node Ni to the node Nj corresponding to the character pixel block Ci and Cj based on the classification result performed on the character element Gt. When the character element Gt has a plurality of classification possibilities, the character string recognition unit 205 generates a plurality of sides Eij corresponding to each.

  The character string recognition unit 205 sets a weight based on the classification result of the character elements corresponding to the sides for each side of the recognition graph. The case where the character element Gt has three classification possibilities of (character code A, reliability a), (character code B, reliability b), and (character code C, reliability c) will be described. In this case, the character string recognition unit 205 uses three different sides represented as Eij (A, a), Eij (B, b), and Eij (C, c) as the side Eij from the node Ni to the node Nj. Generate.

  FIG. 6 is an exemplary diagram of a recognition graph. This recognition graph is created with the character pixel blocks 501 to 506 in FIG. 5 as nodes and the character elements 511 to 521 as edges. The recognition graph has a total of seven nodes, nodes N1 to N6 and a terminal node Ne corresponding to the character pixel blocks 501 to 506. As a result of the classification of the character element 511, there is a possibility of classification into three characters, “sentence”, “community”, and “father”, and the reliability is “0.6”, “0.3”, “0.2”. Is obtained. As a result, three sides E12 (sentence, 0.6), E12 (intersection, 0.3), and E12 (father, 0.2) are generated as sides from the node N1 to the node N2. Similarly, an edge based on the classification result of each character element is generated to the right of the node N2. In FIG. 6, only one edge is shown as each edge other than E12, but this is for simplifying the figure, and there are actually edges corresponding to a plurality of classification possibilities. Sometimes.

  The character string recognition unit 205 that created the recognition graph applies the context information to the recognition graph (S407). “Applying context information” means changing the weight of each side of the recognition graph based on the context information. “Context information” is information relating to the relative relationship between the character of interest in the character string and the characters before and after it. The context information includes, for example, N-gram that defines the probability of occurrence of N characters that are continuously present in a large amount of text collected from books and Internet documents. In the present embodiment, an example in which a two-character frequency Biggram, which is a type of N-gram and is commonly used, is applied to a recognition graph as context information will be described below.

There are sides Eik (A, w1) and Ekj (B, w2) connected through one node Nk, and the consecutive occurrence probability of the characters “A” and “B” is defined as P in the Bigram information. Then, the weights w1 and w2 on both sides are changed to w1 ′ and w2 ′, respectively, by (Equation 1).
w ′ = w + d (P ≧ th) (Formula 1)
“D” is an added value to the weight. “Th” is a constant serving as a threshold relating to the occurrence probability.

  The character string recognizing unit 205 obtains, from Bigram information, the continuous occurrence probability of two characters corresponding to the classification result for all combinations of the two sides on the left and right sides in all sections other than both ends in the recognition graph. Then, the weight is changed by applying (Equation 1). When there is a combination of one side and a plurality of sides through a predetermined clause, the character string recognition unit 205 determines only a combination of two sides corresponding to two characters having the highest occurrence probability in the Bigram information. Select and change its weight. In this case, the character string recognition unit 205 may change the weights for all combinations. Alternatively, the character string recognizing unit 205 may change the value after distributing the addition value for the weight based on the relative ratio of the occurrence probabilities.

An example of changing edge weights using Bigram information for the recognition graph of FIG. 6 will be described below. The consecutive occurrence probability of two characters defined in the Bigram information is defined as follows, and the threshold value th = 0.001 and the correction value d = 0.2 in (Expression 1).
"Culture" = 0.01
"Sentence A" = 0.0001
"Correct" = 0.00005
"Intercourse" = 0.0001
“Fathering” = 0.003
"Father Lee" = 0.0002

  At this time, the two characters having a probability equal to or higher than the threshold th and to be changed by (Equation 1) are “culture” and “paternalization”. The edges corresponding to these are a combination of E12 and E24, and E12 has edges for both “sentence” and “father” classification possibilities. Here, the character string recognizing unit 205 changes only the two sides corresponding to “culture” having the highest probability, and changes only the combination of E12 (sentence, 0.6) and E24 (formation, 0.5) to E12. (Sentence, 0.8) and E24 (Chemical, 0.7).

  FIG. 7 is an exemplary diagram of a recognition graph to which context information is applied. In FIG. 7, in addition to the correction related to “culture”, the occurrence probability of “chemical” in the Bigram information is equal to or higher than the threshold value, and therefore similar correction values are applied to E24 and E4e. Note that (Equation 1) is an example. For example, the weight may be reduced when the probability is less than or equal to the threshold, and a function value using the probability as an argument, for example, log (P) or exp (P) is added or multiplied. Then, the context information may be applied.

  After applying the context information to the recognition graph, the character string recognition unit 205 selects an optimum path in the recognition graph after the application processing (S408). The “optimal path” is a path that connects the characters of the classification result of the edges that passed from the first node to the terminal node of the recognition graph to obtain a plausible recognition result for the entire character string i. In order to obtain the optimum path, the character string recognition unit 205 sets, for example, a plurality of paths covering all combinations of sides, and the value obtained by dividing the total number of sides by the number of sides passes through the maximum value. Select the path. Alternatively, the character string recognizing unit 205 defines a value based on the reciprocal of the edge weight or the like as an edge penalty, and selects an optimum path by selecting a path that minimizes the total path penalty.

  The character string recognition unit 205 outputs, as a character string recognition result for the character string i included in the photographed image, a character code string obtained by connecting the classification results of the selected sides based on the selected optimal path (S409). With the output of the character string recognition result, the character string recognition process ends. The output character string recognition result is added to the information of the character string i in the character string information storage unit 206.

  When the current process is not the first character string recognition process for the character string i stored in the character string information storage unit 206 (S405: N), the character string recognition unit 205 updates the recognition graph as follows. Processing is performed (S410). In this case, the recognition graph generated in the previous character string recognition process is added to the character string i to be processed in the character string information storage unit 206. Therefore, the character string recognizing unit 205 updates the recognition graph based on the character pixel block, the character element, and the classification result obtained from the character string image input in the current processing of S401. The update performed here includes at least one of adding a new clause or a new side and changing the weight of an existing side.

  FIG. 8 is a flowchart showing the flow of the recognition graph update process. Here, it is assumed that the recognition graph S ′ has already been generated for the character string i as a result of the processing of S406 (see FIG. 4) from the character string image acquired from the image t ′ in the past. At that time, a set of character pixel blocks {C ′} is generated from the character string image by the process of S402, and a set of character elements {G ′} is generated by the process of S403. Assume that the recognition graph S ′ includes a set of nodes {N ′} and a set of edges {E ′} corresponding to each. In this process, for the same character string image i, a character pixel block set {C} and a character element set {G} are already generated from the character string image obtained from the image t, and the character string recognition unit In step S <b> 410, the process of S <b> 410 is performed using these as inputs.

  Note that the weight of each side of the recognition graph S ′ may have been updated by the context information application process of S407. In the subsequent processing, it is desirable that the edge weight of the recognition graph S ′ is an original value based on the classification result. The character string recognizing unit 205 stores these original values as part of the recognition information of the character string i, and resets them to the original values in the process of FIG. Alternatively, the character string recognizing unit 205 does not actually change the weight of each side in the process of S407, and when selecting the optimum path in the process of S408, the context information is added while summing the weights of the edges on the route. You may make it add a weight and a penalty in consideration.

  From the character pixel block included in the character image block set {C} and the character elements included in the character element set {G}, the character string recognition unit 205 uses the node set {N} and the edge set {E}. } Is created (S801). The character string recognition unit 205 creates a recognition graph by the same process as the process of S406.

  The character string recognition unit 205 initializes the pair clause list Lp prepared as information to be used in the recognition graph update processing to be empty (S802). The “pair clause list Lp” is a list having as an element a pair (Nx, N′x) corresponding to the clause Nx of the recognition graph S and the clause N′x of the recognition graph S ′ stored in the character string information. is there.

  The character string recognizing unit 205 performs loop processing on the nodes Nx (= N1, N2,...) Included in the node set {N} of the recognition graph S one by one in order (S803, S804). First, the character string recognizing unit 205 determines whether or not a node N′y corresponding to the node Nx of the recognition graph S exists in the recognition graph S ′ (S803). If it exists (S803: Y), the character string recognition unit 205 adds (Nx, N'y) to the pair clause list Lp, assuming that the clause Nx and the clause N'y are a pair (S804). If it does not exist (S803: N), the character string recognition unit 205 repeats this loop process for the next Nx.

Here, an example of a method for determining whether or not the node Nx corresponds to the node N′y will be described. The node Nx of the recognition graph S is generated based on the character pixel block Cx extracted from the image t currently being processed. The node N′y of the recognition graph S ′ is generated based on the character pixel block C′y extracted by the previous character string recognition process, that is, extracted from the past image t ′. Here, when the character pixel block Cx extracted from the image t currently being processed and the character pixel block C′y extracted from the image t ′ in the past satisfy the following conditional expression, the nodes Nx and N It is determined that 'y corresponds.
| Left end coordinate of Cx− (Left end coordinate of C′y + ΔC′y) | <D
ΔC′y is a movement amount of the character pixel block C′y between the image t ′ and the image t. The amount of movement is obtained, for example, by the captured image processing unit 202 as the amount of movement of the target character pixel block C′y as in the process of S304 in FIG. D is a positive constant and indicates an allowable amount of displacement of the character pixel block in consideration of the movement amount. The constant D is set to a value related to the actual size of the character string i, for example, 1/10 of the height of the character string i.

  When the character string is printed from left to right in the horizontal direction, the determination method based on the above formula can be used. If the character string is printed from top to bottom, the determination may be made by comparing the upper end coordinates. Further, the shape characteristics of both character pixel blocks may be considered. For example, it may be determined that the node Nx and the node N′y correspond only when part or all of the shapes match. In particular, when the shapes of both character pixel blocks partially match, only the case where the left side of the character pixel block Cx matches the left side of the character pixel block C′y or vice versa is supported. May be determined.

  The loop processing is completed by performing the processing of S803 and S804 for all the nodes Nx included in the set {N} of nodes of the recognition graph. When the loop processing in S803 and S804 is completed, the character string recognition unit 205 creates a side list Le having all sides of the recognition graph S as initial values (S805). When there are a plurality of sides between a set of clauses Ni and Nj due to a plurality of classification possibilities of character elements, the character string recognition unit 205 has different sides Eij1, Eij2,. . . To the side list Le.

  The character string recognizing unit 205 performs loop processing on the edges Eij included in the edge list Le one by one in order (S806 to S811).

  The character string recognizing unit 205 checks whether or not there are nodes corresponding to the recognition graph S ′ for the first and last nodes of the edge Eij of the recognition graph S, that is, the nodes Ni and Nj (S806). Specifically, the character string recognition unit 205 checks whether the pair clause list Lp includes an element that forms a pair with the target clause. The character string recognizing unit 205 performs a process that branches when the corresponding number of pairs is (0, 1, 2). When the number of pairs is “0” (S806: 0), that is, when there are no clauses corresponding to the clause Ni and the clause Nj in the recognition graph S ′, the character string recognition unit 205 does not perform processing for the current side Eij, This loop process is repeated for the next side in the side list Le.

  When the number of pairs is “1” (S806: 1), that is, when only one of the node Ni or the node Nj has a node that is a pair in the recognition graph S ′, the character string recognition unit 205 recognizes the recognition graph S ′. The node having no node corresponding to is added as a new node to the recognition graph S ′ (S807). For example, when there is no clause corresponding to the clause Nj in the recognition graph S ′, the character string recognition unit 205 adds the clause N′j ′ to the recognition graph S ′. “J ′” represents a subscript that is unique in the recognition graph S ′ after the addition. Further, the character string recognition unit 205 adds a pair (Nj, N′j ′) including a clause, Nj, and a newly added clause N′j ′ to the pair clause list Lp. Thereafter, the edge Eij has a node corresponding to the recognition graph S ′ in both nodes.

  The character string recognition unit 205 adds a new side having the same weight as the side Eij between the two clauses of the recognition graph S ′ respectively corresponding to the two clauses at both ends of the side Eij (S808). For example, in the character string recognition unit 205, the nodes Ni and Nj of the recognition graph S correspond to the nodes N′k and N′l of the recognition graph S ′, respectively, the edge Eij is the classification result A, and the classification reliability w. , Add edge E′kl (A, w) to recognition graph S ′. When there are a plurality of sides between the node Ni and the node Nj, the character string recognition unit 205 adds all the sides. At this time, the association of the clauses between the recognition graph S and the recognition graph S ′ can be easily performed by using the pair clause list Lp.

  The character string recognizing unit 205 removes the edge Eij from the edge list Lp so that it will not be added to the recognition graph S ′ thereafter (S809). The character string recognition unit 205 repeats this loop process for the next side of the side list Lp.

  When the number of pairs is “2” (S806: 2), that is, when both the nodes Ni and Nj have a pair of nodes in the recognition graph S ′, the nodes corresponding to both the first and last of the edge Eij are recognized. In the graph S ′. In this case, the character string recognizing unit 205 sets the nodes of the recognition graph S ′ corresponding to the nodes Ni and Nj as the nodes N′k and N′l, and the edge E′kl having the same classification result as the edge Eij is obtained. It is checked whether it exists (S810). If it does not exist (S810: N), the character string recognition unit 205 adds a new edge having the same weight as the edge Eij to the recognition graph S '(S808).

  If it exists (S810: Y), the character string recognition unit 205 updates the weight of the side E′kl having the same classification result as the side Eij (S811). For example, the character string recognition unit 205 compares the classification reliability of the two and sets the higher value as a new weight. This is an example, and the weight may be updated to the average value of both. When there are a plurality of sides Eij, the character string recognition unit 205 repeats this loop process for each side.

  When the loop processing of S806 to S811 for all the edges included in the edge list Le is completed, the character string recognition unit 205 determines whether or not edges that can be added to the recognition graph S ′ remain in the list Le ( S812). If it remains (S812: Y), the character string recognition unit 205 executes the loop processing of S806 to S811 again. If no character string remains (S812: N), the character string recognizing unit 205 ends the recognition graph update process for the character string i.

  For example, (1) the edge list is empty, or (2) the edge list Le in the loop processing of S806 to S811 immediately before is determined as to whether the edge that can be added to the recognition graph S ′ remains in the list Le. This is performed under the two conditions of no change in the number of elements. When any one of the conditions is satisfied, the character string recognition unit 205 determines that there are remaining edges that can be added to the recognition graph S ′ in the list Le.

(Specific example of character string recognition)
FIG. 9 is an explanatory diagram of a specific example in which the character string recognition process shown in FIGS. 4 and 8 is performed. An image 901 is a captured image t ′ at time t ′, and an image 902 is a captured image t at time t after time t ′. Images 901 and 902 include character string images 910 and 920, respectively. The character string images 910 and 920 correspond to the same character string i in the same subject. However, since the angle at the time of shooting is different, the position, angle, size, etc. of the character string are different. FIG. 10 is an explanatory diagram of a specific example of the recognition graph.

  The character string recognition unit 205 receives the image 901 and performs the character string recognition process of FIG. 4 for the character string image 910. Through the processing of S401 and S402, the character string recognition unit 205 extracts character pixel blocks 911 to 917 from the character string image 910. The character string recognizing unit 205 assumes that the recognition process for the character string i is the first time, and generates the recognition graph 1000 of FIG. 10A as the recognition graph S ′ through the processing of S403 to S406. The recognition graph 1000 includes nodes 1001 to 1007 and a terminal node 1008 corresponding to the character pixel blocks 911 to 917.

  Subsequently, the character string recognition unit 205 performs the character string recognition process of FIG. 4 on the character string image 920 with the image 902 as an input. The character string recognizing unit 205 extracts character pixel blocks 921 to 927 from the character string image 920 through the processes of S401 and S402. Since the character string recognition unit 205 has already performed the recognition process on the character string i, the process from S403 to S405 to the process of S410, that is, the process of FIG. 8 is performed. The character string recognizing unit 205 generates the recognition graph 1010 of FIG. 10B as the recognition graph S by the processing of S801. The recognition graph 1010 includes nodes 1011 to 1017 and a terminal node 1018 corresponding to the character pixel blocks 921 to 927.

  The character string recognition unit 205 generates a pair clause list Lp between the node of the recognition graph 1000 in FIG. 10A and the node of the recognition graph 1010 in FIG. The determination condition described in the processing of S803 is used to determine whether the clause is a pair. Here, (1001, 1011), (1002, 1012), (1003, 1013), (1005, 1014), (1006, 1015), (1007, 1016) form a pair.

  In the processing from S805 to S812, the character string recognition unit 205 performs processing to add or update an edge from the recognition graph 1010 in FIG. 10B to the recognition graph 1000 in FIG. FIG. 10C shows the recognition graph 1020 after the recognition graph 1010 is updated.

  The addition of the side is performed according to the number of pairs of nodes at both ends of the side as described in the processing of S806. The side having the number of pairs “2”, for example, the side 1091 between the node 1011 and the node 1012 does not have an edge of the same classification between the node 1001 and the node 1002 forming a pair. Therefore, the character string recognition unit 205 adds a new side 1093 having the same classification and weight as the side 1091 in the process of S808. When the number of pairs is “1”, for example, the edge 1092 between the nodes 1016 and 1017, the character string recognition unit 205 first starts a new recognition graph as a node corresponding to the node 1017 having no pair. Add node 1021 to 1000. Then, the character string recognition unit 205 adds a side 1094 having the same classification and weight as the side 1092 between the node 1007 and the node 1021. Similar processing is performed for the other sides.

When the recognition graph update processing ends, the character string recognition unit 205 applies the context information to the recognition graph 1020 in FIG. Here, the weight of the target side is updated by (Equation 1) described in the procedure of S407. Among the Bigram information to be used, two characters whose continuous occurrence probability P is equal to or greater than the threshold th in (Equation 1) are listed below.
“Chloride” “Chemical” “Hika” “Kari” “Cutting” “Riu” “Um”

  FIG. 10F shows a recognition graph 1020 after using these to correct the weight of each side according to (Equation 1). Note that the correction value d in Equation 1 is 0.1. As an example, the weight before the update of the edge 1094 was “0.7”, but “0.1” is added by each of the “Liu” and “Um” of the Bigram information, and the weight becomes “0.9”. It was.

  The character string recognizing unit 205 selects an optimum path from the recognition graph to which the context information is applied in the process of S408. Here, the character string recognizing unit 205 determines the optimum path by adding the weight of each side and dividing the divided value by the number of sides. In the recognition graph 1020 of FIG. 10F, “potassium chloride = 0.90” is obtained as the optimum path. This is a correct character string recognition result.

  The result of applying the same context information to the recognition graph 1000 in FIG. 10A and the recognition graph 1010 in FIG. 10B without performing the update processing of the recognition graph shown in FIG. ) And FIG. 10 (e). When the optimum paths are obtained in the recognition graph 1000 in FIG. 10D and the recognition graph 1010 in FIG. 10E, “Tetsuru Karihiro = 0.65” and “salt-cut-offum = 0.70” are obtained.

  This indicates that even if each of the character string images 910 and 920 of the images 901 and 902 in FIG. 9 is recognized and the context information is applied, a correct character string recognition result cannot be obtained. In addition, even when trying to obtain the correct character string by fusing the two character string recognition results “Tetsuka Karihiro” and “Shio-Iri-um”, the correspondence between each character cannot be uniquely determined. Accurate fusion is difficult. However, the character string recognizing unit 205 performs an update process such that the recognition graph obtained from the character string image 920 is integrated into the recognition graph obtained from the character string image 910, whereby the two character string images 910 and 920 are updated. An integrated recognition graph including interpretation can be obtained. In the integrated graph, since a combination of edges to which context information can be applied is obtained simultaneously from the two character string images 910 and 920, there is a high possibility that a correct character string recognition result is obtained by selecting the optimum path.

  According to the present embodiment, the information processing apparatus 100 receives the first image obtained by capturing the target character string as an input, sets the extracted character pixel block as a node, and sets the character element classification result obtained by combining the character pixel blocks as a side. A first recognition graph is generated. The method of combining character pixel blocks corresponds to a method of cutting out individual characters from a character string image. Next, the information processing apparatus 100 receives the second image obtained by photographing the same target character string, generates a second recognition graph using the character pixel block and the character element extracted in the same manner, and changes the contents according to the contents. Update the recognition graph of 1.

  The information processing apparatus 100 sets the position of the character pixel block corresponding to each of the nodes of the first recognition graph and the nodes of the second recognition graph, and each of the positions between the first image and the second image. Considering the amount of movement of the character pixel block, they are associated as a pair. When the node corresponding to the node of the second recognition graph does not exist in the first recognition graph, the information processing apparatus 100 adds a new compatible node to the first recognition graph as a pair. The information processing apparatus 100 copies an edge between two adjacent nodes in the second recognition graph as an edge between the two nodes corresponding to the two nodes in the first recognition graph. The information processing apparatus 100 obtains a character string recognition result by applying the context information to the updated first recognition graph, updating the edge weights, and selecting the optimum path.

  Through such processing, the information processing apparatus 100 applies the context information to the entire character pixel block and character element classification results extracted from both the first image and the second image, and the likely character string. The recognition result can be selected. As a result, it is possible to perform a character string recognition process with higher accuracy than when the first image and the second image are individually recognized.

  In the above example, the first image and the second image obtained by photographing the target character string are targets of processing, but there may be more images obtained by photographing the target character string. The information processing apparatus 100 applies the context information after integrating the recognition graph obtained from the third, fourth,... Images obtained by capturing the target character string in addition to the first image and the second image. Also good.

  The information processing apparatus 100 generates a recognition graph including the entire character pixel block and character element classification results extracted from the first image and one or more other images for the same character string. The information processing apparatus 100 applies context information to the recognition graph and selects a plausible character string recognition result. As a result, it is possible to perform character string recognition processing with higher accuracy than recognizing each image individually. Further, when acquiring the first and one or more other images in time series, the information processing apparatus 100 can perform a character string recognition process that gradually improves the recognition accuracy.

(Modification)
FIG. 11 is an explanatory diagram of a modification of the character string recognition process. The image 1100, the image 1110, and the image 1120 are captured images obtained by capturing the character string “26A489J” of identification information printed on a non-rigid cloth or the like. Since the cloth is deformed depending on the photographing timing, each image 1100, 1110, 1120 is deformed to a state where some characters cannot be recognized. Such a photographed image is likely to occur, for example, in an image photographed for recognizing a marathon runner number number. The information processing apparatus 100 performs character string recognition processing illustrated in FIGS. 3, 4, and 8 on the images 1100, 1110, and 1120.

  The information processing apparatus 100 extracts five character pixel blocks from the image 1100 acquired first. The information processing apparatus 100 classifies the character pixel block as a character element and generates a recognition graph 1101. In the recognition graph of FIG. 11, the reliability value of the character classification on each side is omitted.

  Next, the information processing apparatus 100 acquires the image 1110, extracts six character pixel blocks, classifies them into character elements, and generates a recognition graph 1111. In order to update the recognition graph 1101, the information processing apparatus 100 updates the paired clauses corresponding to the clauses a to e excluding the terminal node f of the recognition graph 1101 and the nodes g to l excluding the terminal node of the recognition graph 1111. Generate a list.

  In this example, the information processing apparatus 100 compares the image shapes of character pixel blocks corresponding to a node, and generates a pair node from a pair of character pixel blocks determined to match or approximate the shape. The determination of whether or not the shapes match can use SIFT or the like that is resistant to image rotation and deformation as a known image feature amount. In this example, (g, a) and (h, b) are determined as paired clauses, and the nodes corresponding to the other nodes i, j, k, and l in the recognition graph 1111 are added to the recognition graph 1101. Each edge of the recognition graph 1111 is copied as an edge between corresponding nodes in the recognition graph 1101.

  Subsequently, the information processing apparatus 100 acquires the image 1120, extracts five character pixel blocks, classifies them into character elements, and generates a recognition graph 1121. In order to update the recognition graph 1101, the information processing apparatus 100 updates a pair graph corresponding to the nodes a to e and the added nodes i to l of the recognition graph 1101 and the nodes n to r of the recognition graph 1121. Is generated. In this example, the information processing apparatus 100 determines that (o, j), (p, k), (q, d), and (r, e) are paired clauses, and a node corresponding to the node n of the recognition graph 1121. Is added to the recognition graph 1101. Each edge of the recognition graph 1121 is copied as an edge between corresponding nodes in the recognition graph 1101.

  The recognition graph 1101 updated by the above processing becomes a recognition graph 1131. In the recognition graph 1131, a virtual start node “0” is arranged first so that a different node can be used as the start node when detecting the optimum path.

  The information processing apparatus 100 applies context information to the recognition graph 1131 and selects an optimal path. In this example, the context information uses a format definition predetermined as a character string of identification information to be recognized as a character string pattern dictionary. The “format definition” specifically defines the character type, which is the number of characters and each character position. For example, the format definition is a definition in which the character string of the identification information is always composed of seven characters, the third and seventh characters are uppercase alphabets, and the others are numbers. In the recognition graph 1131, there is only a single path “26A489J” for outputting a character string so as to satisfy this definition. Therefore, the information processing apparatus 100 outputs the character string “26A489J” as a recognition result, and ends the character string recognition process.

  In this example as well, two or four or more captured images used for character recognition may be used.

  In the character string recognition process as described above, the information processing apparatus 100 includes character pixel blocks extracted from two or more photographed images for the same character string and character element classification results that are combinations of character pixel blocks. Generate recognition graphs and integrate them. The information processing apparatus 100 applies the context information to the integrated recognition graph and selects a likely character string recognition result. As a result, it is possible to acquire a character string recognition result with higher accuracy than character recognition of two or more captured images individually. Further, when acquiring a plurality of captured images in time series, the information processing apparatus 100 can perform a character string recognition process in which recognition accuracy is gradually improved.

  The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Claims (11)

Acquisition means for acquiring a photographed image obtained by photographing a character string printed on a subject;
Extracting means for extracting a character pixel block from a character string portion of the photographed image;
Classification means for classifying the character elements based on the character pixel block;
Generating means for generating a recognition graph from the character pixel block and the classification result of the character elements;
Recognizing means for recognizing the character string based on the recognition graph,
The generating means includes a first character pixel block extracted from a first image of the plurality of photographed images and a first character element classified from the first character pixel block. A recognition graph is generated and classified from a second character pixel block extracted from a second image different from the first image among the plurality of captured images and the second character pixel block. Generating a second recognition graph from the second character element;
The recognizing unit determines whether the first character pixel block and the second character pixel block correspond to the same character string based on the determination as to whether or not the first character pixel block and the second character pixel block correspond to the same character string. Generating an integrated recognition graph integrated with the graph, and recognizing the character string based on the integrated recognition graph,
Information processing device. The recognition means corresponds to the same character string in the first character pixel block and the second character pixel block based on an optical flow between the first image and the second image. It is characterized by determining whether or not,
The information processing apparatus according to claim 1. The recognizing unit determines that the first character pixel block and the second character pixel block are the same character string based on a result of homography conversion between the first image and the second image. It is determined whether or not it corresponds to,
The information processing apparatus according to claim 1. The recognition unit has nodes based on the character pixel block and sides based on the classification result of the character elements, and the weight of each side is set based on the reliability of the classification included in the classification result Produces
The recognition means obtains the recognition result of the character string by selecting a path from which the total sum of the weights of the edges is minimized or maximized from the integrated recognition graph.
The information processing apparatus according to claim 1. The recognition means obtains a pair between the node of the first recognition graph and the node of the second recognition graph, and based on the pair, the edge of the second recognition graph is determined as the first recognition graph. The integrated recognition graph is generated by copying to a graph,
The information processing apparatus according to claim 4. The recognition means is based on the node based on the first character pixel block and the second character pixel block when the first character pixel block and the second character pixel block correspond to the same character string. It is determined that there is a pair between the clauses, and when it is determined that they do not correspond, a clause based on the second character pixel block is added to the first recognition graph to form a pair.
The information processing apparatus according to claim 5. The recognition means updates the weight of each side of the integrated recognition graph using context information, selects the path from the updated integrated recognition graph, and obtains a recognition result of the character string. ,
The information processing apparatus according to claim 4. The recognition means uses either N-gram or a character string pattern dictionary for the context information.
The information processing apparatus according to claim 7. A method executed by an information processing apparatus including predetermined imaging means,
A plurality of captured images obtained by capturing a character string printed on a predetermined subject is acquired from the imaging unit, a character pixel block is extracted from a character string portion of the captured image, and the extracted character pixel block is extracted. Classify text elements based on
A first recognition graph is generated from a first character pixel block extracted from a first image of the plurality of photographed images and a first character element classified from the first character pixel block. And a second character pixel block extracted from a second image different from the first image of the plurality of photographed images, and a second character element classified from the second character pixel block And generate a second recognition graph from
The first recognition graph and the second recognition graph are integrated based on whether or not the first character pixel block and the second character pixel block correspond to the same character string. An integrated recognition graph is generated, and the character string is recognized based on the integrated recognition graph.
Character recognition method. In a computer provided with predetermined imaging means,
A plurality of captured images obtained by capturing a character string printed on a predetermined subject is acquired from the imaging unit, a character pixel block is extracted from a character string portion of the captured image, and the extracted character pixel block is extracted. Processing to classify character elements based on,
A first recognition graph is generated from a first character pixel block extracted from a first image of the plurality of photographed images and a first character element classified from the first character pixel block. And a second character pixel block extracted from a second image different from the first image of the plurality of photographed images, and a second character element classified from the second character pixel block Generating a second recognition graph from
The first recognition graph and the second recognition graph are integrated based on whether or not the first character pixel block and the second character pixel block correspond to the same character string. A process of generating an integrated recognition graph and recognizing the character string based on the integrated recognition graph;
A computer program for running.   A computer-readable storage medium storing the computer program according to claim 10.